This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 33bdbf8 [SPARK-34231][AVRO][TEST] Make proper use of resource file
within AvroSuite test case
33bdbf8 is described below
commit 33bdbf80dad4138272bafff5cf5867a2f0a2ddb2
Author: Erik Krogen <[email protected]>
AuthorDate: Wed Jan 27 16:16:33 2021 +0900
[SPARK-34231][AVRO][TEST] Make proper use of resource file within AvroSuite
test case
Change `AvroSuite."Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES"`
to use `episodesAvro`, which is loaded as a resource using the classloader,
instead of trying to read `episodes.avro` directly from a relative file path.
This is the proper way to read resource files, and currently this test will
fail when called from my IntelliJ IDE, though it will succeed when called from
Maven/sbt, presumably due to different working directory handling.
No, unit test only.
Previous failure from IntelliJ:
```
Source 'src/test/resources/episodes.avro' does not exist
java.io.FileNotFoundException: Source 'src/test/resources/episodes.avro'
does not exist
at
org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1405)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1072)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1040)
at
org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34(AvroSuite.scala:397)
at
org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34$adapted(AvroSuite.scala:388)
```
Now it succeeds.
Closes #31332 from xkrogen/xkrogen-SPARK-34231-avrosuite-testfix.
Authored-by: Erik Krogen <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
(cherry picked from commit b2c104bd87361e5d85f7c227c60419af16b718f2)
Signed-off-by: HyukjinKwon <[email protected]>
---
.../avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index 34f01de..e6063d6 100644
--- a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -388,11 +388,11 @@ class AvroSuite extends QueryTest with SharedSQLContext
with SQLTestUtils {
}.getMessage
assert(message.contains("No Avro files found."))
- val srcFile = new File("src/test/resources/episodes.avro")
- val destFile = new File(dir, "episodes.avro")
- FileUtils.copyFile(srcFile, destFile)
+ Files.copy(
+ Paths.get(new URL(episodesAvro).toURI),
+ Paths.get(dir.getCanonicalPath, "episodes.avro"))
- val result =
spark.read.format("avro").load(srcFile.getAbsolutePath).collect()
+ val result = spark.read.format("avro").load(episodesAvro).collect()
checkAnswer(spark.read.format("avro").load(dir.getAbsolutePath),
result)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
