This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 374aeef [SPARK-34231][AVRO][TEST] Make proper use of resource file
within AvroSuite test case
374aeef is described below
commit 374aeef0ef89d5ad42d0e590175be53fd4ac220f
Author: Erik Krogen <[email protected]>
AuthorDate: Wed Jan 27 16:16:33 2021 +0900
[SPARK-34231][AVRO][TEST] Make proper use of resource file within AvroSuite
test case
Change `AvroSuite."Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES"`
to use `episodesAvro`, which is loaded as a resource using the classloader,
instead of trying to read `episodes.avro` directly from a relative file path.
This is the proper way to read resource files, and currently this test will
fail when called from my IntelliJ IDE, though it will succeed when called from
Maven/sbt, presumably due to different working directory handling.
No, unit test only.
Previous failure from IntelliJ:
```
Source 'src/test/resources/episodes.avro' does not exist
java.io.FileNotFoundException: Source 'src/test/resources/episodes.avro'
does not exist
at
org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1405)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1072)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1040)
at
org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34(AvroSuite.scala:397)
at
org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34$adapted(AvroSuite.scala:388)
```
Now it succeeds.
Closes #31332 from xkrogen/xkrogen-SPARK-34231-avrosuite-testfix.
Authored-by: Erik Krogen <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
(cherry picked from commit b2c104bd87361e5d85f7c227c60419af16b718f2)
Signed-off-by: HyukjinKwon <[email protected]>
---
.../avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index 1ac0fb2..b192b3d 100644
--- a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -392,11 +392,11 @@ abstract class AvroSuite
}.getMessage
assert(message.contains("No Avro files found."))
- val srcFile = new File("src/test/resources/episodes.avro")
- val destFile = new File(dir, "episodes.avro")
- FileUtils.copyFile(srcFile, destFile)
+ Files.copy(
+ Paths.get(new URL(episodesAvro).toURI),
+ Paths.get(dir.getCanonicalPath, "episodes.avro"))
- val result =
spark.read.format("avro").load(srcFile.getAbsolutePath).collect()
+ val result = spark.read.format("avro").load(episodesAvro).collect()
checkAnswer(spark.read.format("avro").load(dir.getAbsolutePath),
result)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
