This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 0eff303 [SPARK-34231][AVRO][TEST] Make proper use of resource file
within AvroSuite test case
0eff303 is described below
commit 0eff303b4d03764db335e7d48dbce53278d02b5b
Author: Erik Krogen <[email protected]>
AuthorDate: Wed Jan 27 16:16:33 2021 +0900
[SPARK-34231][AVRO][TEST] Make proper use of resource file within AvroSuite
test case
Change `AvroSuite."Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES"`
to use `episodesAvro`, which is loaded as a resource using the classloader,
instead of trying to read `episodes.avro` directly from a relative file path.
This is the proper way to read resource files, and currently this test will
fail when called from my IntelliJ IDE, though it will succeed when called from
Maven/sbt, presumably due to different working directory handling.
No, unit test only.
Previous failure from IntelliJ:
```
Source 'src/test/resources/episodes.avro' does not exist
java.io.FileNotFoundException: Source 'src/test/resources/episodes.avro'
does not exist
at
org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1405)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1072)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1040)
at
org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34(AvroSuite.scala:397)
at
org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34$adapted(AvroSuite.scala:388)
```
Now it succeeds.
Closes #31332 from xkrogen/xkrogen-SPARK-34231-avrosuite-testfix.
Authored-by: Erik Krogen <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
(cherry picked from commit b2c104bd87361e5d85f7c227c60419af16b718f2)
Signed-off-by: HyukjinKwon <[email protected]>
---
.../avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index e576c16..15905a7 100644
--- a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -379,11 +379,11 @@ abstract class AvroSuite extends QueryTest with
SharedSparkSession {
}.getMessage
assert(message.contains("No Avro files found."))
- val srcFile = new File("src/test/resources/episodes.avro")
- val destFile = new File(dir, "episodes.avro")
- FileUtils.copyFile(srcFile, destFile)
+ Files.copy(
+ Paths.get(new URL(episodesAvro).toURI),
+ Paths.get(dir.getCanonicalPath, "episodes.avro"))
- val result =
spark.read.format("avro").load(srcFile.getAbsolutePath).collect()
+ val result = spark.read.format("avro").load(episodesAvro).collect()
checkAnswer(spark.read.format("avro").load(dir.getAbsolutePath),
result)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
