Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/23130#discussion_r236719477
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/sources/SaveLoadSuite.scala ---
@@ -142,4 +143,15 @@ class SaveLoadSuite extends DataSourceTest with
SharedSQLContext with BeforeAndA
assert(e.contains(s"Partition column `$unknown` not found in schema
$schemaCatalog"))
}
}
+
+ test("skip empty files in load") {
+ withTempDir { dir =>
+ val path = dir.getCanonicalPath
+ Files.write(Paths.get(path, "empty"), Array.empty[Byte])
+ Files.write(Paths.get(path, "notEmpty"), "a".getBytes)
+ val readback = spark.read.option("wholetext", true).text(path)
+
+ assert(readback.rdd.getNumPartitions == 1)
--- End diff --
It seems expected value should be on right. I changed the order and got the
following:
```scala
assert(123 === readback.rdd.getNumPartitions)
```
```
123 did not equal 1
ScalaTestFailureLocation: org.apache.spark.sql.sources.SaveLoadSuite at
(SaveLoadSuite.scala:155)
Expected :1
Actual :123
```
Current assert triggers correct message:
```scala
assert(readback.rdd.getNumPartitions == 123)
```
```
1 did not equal 123
ScalaTestFailureLocation: org.apache.spark.sql.sources.SaveLoadSuite at
(SaveLoadSuite.scala:155)
Expected :123
Actual :1
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
