This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 4e13ba9 [SPARK-31238][SPARK-31284][TEST][FOLLOWUP] Fix
readResourceOrcFile to create a local file from resource
4e13ba9 is described below
commit 4e13ba90446745fc5a9f46ed1f80c6eefb738795
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Mar 27 18:44:53 2020 -0700
[SPARK-31238][SPARK-31284][TEST][FOLLOWUP] Fix readResourceOrcFile to
create a local file from resource
### What changes were proposed in this pull request?
This PR aims to copy a test resource file to a local file in `OrcTest`
suite before reading it.
### Why are the changes needed?
SPARK-31238 and SPARK-31284 added test cases to access the resouce file in
`sql/core` module from `sql/hive` module. In **Maven** test environment, this
causes a failure.
```
- SPARK-31238: compatibility with Spark 2.4 in reading dates *** FAILED ***
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
path in absolute URI:
jar:file:/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2-hive-2.3-jdk-11/sql/core/target/spark-sql_2.12-3.1.0-SNAPSHOT-tests.jar!/test-data/before_1582_date_v2_4.snappy.orc
```
```
- SPARK-31284: compatibility with Spark 2.4 in reading timestamps ***
FAILED ***
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
path in absolute URI:
jar:file:/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2-hive-2.3/sql/core/target/spark-sql_2.12-3.1.0-SNAPSHOT-tests.jar!/test-data/before_1582_ts_v2_4.snappy.orc
```
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Pass the Jenkins with Maven.
Closes #28059 from dongjoon-hyun/SPARK-31238.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit d025ddbaa7e7b9746d8e47aeed61ed39d2f09f0e)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/sql/execution/datasources/orc/OrcTest.scala | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala
index 16772fe..e929f90 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala
@@ -22,6 +22,7 @@ import java.io.File
import scala.reflect.ClassTag
import scala.reflect.runtime.universe.TypeTag
+import org.apache.commons.io.FileUtils
import org.scalatest.BeforeAndAfterAll
import org.apache.spark.sql._
@@ -136,6 +137,10 @@ abstract class OrcTest extends QueryTest with
FileBasedDataSourceTest with Befor
protected def readResourceOrcFile(name: String): DataFrame = {
val url = Thread.currentThread().getContextClassLoader.getResource(name)
- spark.read.orc(url.toString)
+ // Copy to avoid URISyntaxException when `sql/hive` accesses the resources
in `sql/core`
+ val file = File.createTempFile("orc-test", ".orc")
+ file.deleteOnExit();
+ FileUtils.copyURLToFile(url, file)
+ spark.read.orc(file.getAbsolutePath)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
