Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20705#discussion_r173358166
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
---
@@ -852,52 +846,52 @@ class MetastoreDataSourcesSuite extends QueryTest
with SQLTestUtils with TestHiv
(from to to).map(i => i -> s"str$i").toDF("c1", "c2")
}
- withTable("insertParquet") {
- createDF(0, 9).write.format("parquet").saveAsTable("insertParquet")
+ withTable("t") {
+ createDF(0, 9).write.saveAsTable("t")
checkAnswer(
- sql("SELECT p.c1, p.c2 FROM insertParquet p WHERE p.c1 > 5"),
+ sql("SELECT p.c1, p.c2 FROM t p WHERE p.c1 > 5"),
(6 to 9).map(i => Row(i, s"str$i")))
intercept[AnalysisException] {
- createDF(10,
19).write.format("parquet").saveAsTable("insertParquet")
+ createDF(10, 19).write.saveAsTable("t")
}
- createDF(10,
19).write.mode(SaveMode.Append).format("parquet").saveAsTable("insertParquet")
+ createDF(10, 19).write.mode(SaveMode.Append).saveAsTable("t")
checkAnswer(
- sql("SELECT p.c1, p.c2 FROM insertParquet p WHERE p.c1 > 5"),
+ sql("SELECT p.c1, p.c2 FROM t p WHERE p.c1 > 5"),
(6 to 19).map(i => Row(i, s"str$i")))
- createDF(20,
29).write.mode(SaveMode.Append).format("parquet").saveAsTable("insertParquet")
+ createDF(20, 29).write.mode(SaveMode.Append).saveAsTable("t")
checkAnswer(
- sql("SELECT p.c1, c2 FROM insertParquet p WHERE p.c1 > 5 AND p.c1
< 25"),
+ sql("SELECT p.c1, c2 FROM t p WHERE p.c1 > 5 AND p.c1 < 25"),
(6 to 24).map(i => Row(i, s"str$i")))
intercept[AnalysisException] {
- createDF(30, 39).write.saveAsTable("insertParquet")
+ createDF(30, 39).write.saveAsTable("t")
}
- createDF(30,
39).write.mode(SaveMode.Append).saveAsTable("insertParquet")
+ createDF(30, 39).write.mode(SaveMode.Append).saveAsTable("t")
checkAnswer(
- sql("SELECT p.c1, c2 FROM insertParquet p WHERE p.c1 > 5 AND p.c1
< 35"),
+ sql("SELECT p.c1, c2 FROM t p WHERE p.c1 > 5 AND p.c1 < 35"),
(6 to 34).map(i => Row(i, s"str$i")))
- createDF(40,
49).write.mode(SaveMode.Append).insertInto("insertParquet")
+ createDF(40, 49).write.mode(SaveMode.Append).insertInto("t")
checkAnswer(
- sql("SELECT p.c1, c2 FROM insertParquet p WHERE p.c1 > 5 AND p.c1
< 45"),
+ sql("SELECT p.c1, c2 FROM t p WHERE p.c1 > 5 AND p.c1 < 45"),
(6 to 44).map(i => Row(i, s"str$i")))
- createDF(50,
59).write.mode(SaveMode.Overwrite).saveAsTable("insertParquet")
+ createDF(50, 59).write.mode(SaveMode.Overwrite).saveAsTable("t")
checkAnswer(
- sql("SELECT p.c1, c2 FROM insertParquet p WHERE p.c1 > 51 AND p.c1
< 55"),
+ sql("SELECT p.c1, c2 FROM t p WHERE p.c1 > 51 AND p.c1 < 55"),
(52 to 54).map(i => Row(i, s"str$i")))
- createDF(60,
69).write.mode(SaveMode.Ignore).saveAsTable("insertParquet")
+ createDF(60, 69).write.mode(SaveMode.Ignore).saveAsTable("t")
checkAnswer(
- sql("SELECT p.c1, c2 FROM insertParquet p"),
+ sql("SELECT p.c1, c2 FROM t p"),
(50 to 59).map(i => Row(i, s"str$i")))
- createDF(70,
79).write.mode(SaveMode.Overwrite).insertInto("insertParquet")
+ createDF(70, 79).write.mode(SaveMode.Overwrite).insertInto("t")
checkAnswer(
- sql("SELECT p.c1, c2 FROM insertParquet p"),
+ sql("SELECT p.c1, c2 FROM t p"),
(70 to 79).map(i => Row(i, s"str$i")))
--- End diff --
That is because this PR minimally changed only the test case causing
failures. We cannot generalize all test cases at an one-shot huge PR for all
modules. That will make it difficult to backport the other commits. The main
goal of this PR is improving test- ability for new data sources.
For example, `SPARK-8156:create table to specific database by 'use dbname'`
writes to parquet, but reads with SQL, not by `read.parquet`. So, it doesn't
fail.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
