Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/7342#discussion_r34711152
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
---
@@ -761,4 +761,43 @@ class DataFrameSuite extends QueryTest {
assert(f.getMessage.contains("column3"))
assert(!f.getMessage.contains("column2"))
}
+
+ test("SPARK-6941: Better error message for inserting into RDD-based
Table") {
+ val df = Seq(Tuple1(1)).toDF()
+ val insertion = Seq(Tuple1(2)).toDF("col")
+
+ // pass case: parquet table (HadoopFsRelation)
+ df.write.mode(SaveMode.Overwrite).parquet("tmp_parquet")
+ val pdf = ctx.read.parquet("tmp_parquet")
+ pdf.registerTempTable("parquet_base")
+ insertion.write.insertInto("parquet_base")
+
+ // pass case: json table (InsertableRelation)
+ df.write.mode(SaveMode.Overwrite).json("tmp_json")
+ val jdf = ctx.read.json("tmp_json")
+ jdf.registerTempTable("json_base")
+ insertion.write.mode(SaveMode.Overwrite).insertInto("json_base")
+
+ // error cases: insert into a RDD
+ df.registerTempTable("rdd_base")
+ val e1 = intercept[AnalysisException] {
+ insertion.write.insertInto("rdd_base")
+ }
+ assert(e1.getMessage.contains("Inserting into an RDD-based table is
not allowed."))
+
+ // error case: insert into a RDD based on data source
+ val indirectDS = pdf.select("_1").filter($"_1" > 5)
+ indirectDS.registerTempTable("indirect_ds")
+ val e2 = intercept[AnalysisException] {
+ insertion.write.insertInto("indirect_ds")
+ }
+ assert(e2.getMessage.contains("Inserting into an RDD-based table is
not allowed."))
+
+ // error case: insert into a OneRowRelation
+ new DataFrame(ctx, OneRowRelation).registerTempTable("one_row")
+ val e3 = intercept[AnalysisException] {
+ insertion.write.insertInto("one_row")
+ }
+ assert(e3.getMessage.contains("Inserting into an RDD-based table is
not allowed."))
+ }
--- End diff --
For this test suite, can you change `class DataFrameSuite extends QueryTest
{` to `class DataFrameSuite extends QueryTest with SQLTestUtils {`? Then, you
can use `withTempPath` to create a temp dir and save data in this temp dir.
`withTempPath` will automatically clean up all of your temp data/dirs.
Basically, you can do
```
test("SPARK-6941: Better error message for inserting into RDD-based Table")
{
withTempPath { dir =>
// You can create some dirs inside the given temp dir.
val tempParquet = new File(dir, "tmp_parquet")
val tempJson = new File(dir, "tmp_json")
...
// Your test code, which uses created temp dirs.
...
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
