Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/5801#issuecomment-99222228
Seems our test cases are not sufficient to catch the problem. Can you also
add the following test cases.
In `InsertSuite`, let's change the `val rdd` defined in `beforeAll` to `val
rdd = sparkContext.parallelize((1 to 10).map(i => s"""{"a":$i,
"b":"str${i}"}"""), 5)`. Then, let's change the test of [INSERT OVERWRITE a
JSONRelation multiple
times](https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala#L100-120)
to
```
test("INSERT OVERWRITE a JSONRelation multiple times") {
sql(
s"""
|INSERT OVERWRITE TABLE jsonTable SELECT a, b FROM jt
""".stripMargin)
checkAnswer(
sql("SELECT a, b FROM jsonTable"),
(1 to 10).map(i => Row(i, s"str$i"))
)
// Writing the table to less part files.
val rdd1 = sparkContext.parallelize((1 to 10).map(i => s"""{"a":$i,
"b":"str${i}"}"""), 5)
jsonRDD(rdd1).registerTempTable("jt1")
sql(
s"""
|INSERT OVERWRITE TABLE jsonTable SELECT a, b FROM jt1
""".stripMargin)
checkAnswer(
sql("SELECT a, b FROM jsonTable"),
(1 to 10).map(i => Row(i, s"str$i"))
)
// Writing the table to more part files.
val rdd2 = sparkContext.parallelize((1 to 10).map(i => s"""{"a":$i,
"b":"str${i}"}"""), 10)
jsonRDD(rdd2).registerTempTable("jt2")
sql(
s"""
|INSERT OVERWRITE TABLE jsonTable SELECT a, b FROM jt2
""".stripMargin)
checkAnswer(
sql("SELECT a, b FROM jsonTable"),
(1 to 10).map(i => Row(i, s"str$i"))
)
sql(
s"""
|INSERT OVERWRITE TABLE jsonTable SELECT a * 10, b FROM jt1
""".stripMargin)
checkAnswer(
sql("SELECT a, b FROM jsonTable"),
(1 to 10).map(i => Row(i * 10, s"str$i"))
)
dropTempTable("jt1")
dropTempTable("jt2")
}
```
Also, add the following in the `InsertSuite`.
```
test("save directly to the path of a JSON table") {
table("jt").selectExpr("a * 5 as a", "b").save(path.toString, "json",
SaveMode.Overwrite)
checkAnswer(
sql("SELECT a, b FROM jsonTable"),
(1 to 10).map(i => Row(i * 5, s"str$i"))
)
table("jt").save(path.toString, "json", SaveMode.Overwrite)
checkAnswer(
sql("SELECT a, b FROM jsonTable"),
(1 to 10).map(i => Row(i, s"str$i"))
)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
