GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/16147
[SPARK-18718][TESTS] Skip some test failures due to path length limitation
and fix tests to pass on Windows
## What changes were proposed in this pull request?
There are some tests failed on Windows due to the wrong format of path and
the limitation of path length as below:
This PR proposes both to fix the failed tests by fixing the path for the
tests below:
- `InsertSuite`
```
Exception encountered when attempting to run a suite with class name:
org.apache.spark.sql.sources.InsertSuite *** ABORTED *** (12 seconds, 547
milliseconds)
org.apache.spark.sql.AnalysisException: Path does not exist:
file:/C:projectsspark arget mpspark-177945ef-9128-42b4-8c07-de31f78bbbd6;
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
```
- `PathOptionSuite`
```
- path option also exist for write path *** FAILED *** (1 second, 93
milliseconds)
"C:[projectsspark arget mp]spark-5ab34a58-df8d-..." did not
equal "C:[\projects\spark\target\tmp\]spark-5ab34a58-df8d-..."
(PathOptionSuite.scala:93)
org.scalatest.exceptions.TestFailedException:
at
org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)
at
org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)
...
```
- `UDFSuite`
```
- SPARK-8005 input_file_name *** FAILED *** (2 seconds, 234 milliseconds)
"file:///C:/projects/spark/target/tmp/spark-e4e5720a-2006-48f9-8b11-797bf59794bf/part-00001-26fb05e4-603d-471d-ae9d-b9549e0c7765.snappy.parquet"
did not contain
"C:\projects\spark\target\tmp\spark-e4e5720a-2006-48f9-8b11-797bf59794bf"
(UDFSuite.scala:67)
org.scalatest.exceptions.TestFailedException:
at
org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)
at
org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)
...
```
and to skip the tests belows which are being failed on Windows due to path
length limitation.
- `SparkLauncherSuite`
```
Test org.apache.spark.launcher.SparkLauncherSuite.testChildProcLauncher
failed: java.lang.AssertionError: expected:<0> but was:<1>, took 0.062 sec
at
org.apache.spark.launcher.SparkLauncherSuite.testChildProcLauncher(SparkLauncherSuite.java:177)
...
```
The stderr from the process is `The filename or extension is too long`
which is equivalent to the one below.
- `BroadcastJoinSuite`
```
04:09:40.882 ERROR org.apache.spark.deploy.worker.ExecutorRunner: Error
running executor
java.io.IOException: Cannot run program
"C:\Progra~1\Java\jdk1.8.0\bin\java" (in directory
"C:\projects\spark\work\app-20161205040542-0000\51658"): CreateProcess
error=206, The filename or extension is too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at
org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:167)
at
org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)
Caused by: java.io.IOException: CreateProcess error=206, The filename or
extension is too long
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(ProcessImpl.java:386)
at java.lang.ProcessImpl.start(ProcessImpl.java:137)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 2 more
04:09:40.929 ERROR org.apache.spark.deploy.worker.ExecutorRunner: Error
running executor
(appearently infinite same error messages)
...
```
## How was this patch tested?
Manually tested via AppVeyor.
**Before**
`InsertSuite`:
https://ci.appveyor.com/project/spark-test/spark/build/148-InsertSuite-pr
`PathOptionSuite`:
https://ci.appveyor.com/project/spark-test/spark/build/139-PathOptionSuite-pr
`UDFSuite`:
https://ci.appveyor.com/project/spark-test/spark/build/143-UDFSuite-pr
`SparkLauncherSuite`:
https://ci.appveyor.com/project/spark-test/spark/build/141-SparkLauncherSuite-pr
`BroadcastJoinSuite`:
https://ci.appveyor.com/project/spark-test/spark/build/145-BroadcastJoinSuite-pr
**After**
`PathOptionSuite`:
https://ci.appveyor.com/project/spark-test/spark/build/140-PathOptionSuite-pr
`SparkLauncherSuite`:
https://ci.appveyor.com/project/spark-test/spark/build/142-SparkLauncherSuite-pr
`UDFSuite`:
https://ci.appveyor.com/project/spark-test/spark/build/144-UDFSuite-pr
`InsertSuite`:
https://ci.appveyor.com/project/spark-test/spark/build/147-InsertSuite-pr
`BroadcastJoinSuite`:
https://ci.appveyor.com/project/spark-test/spark/build/149-BroadcastJoinSuite-pr
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark fix-tests
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16147.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16147
----
commit d391f8cfe27563a6f9dbee7e3aec54f1a996de5e
Author: hyukjinkwon <[email protected]>
Date: 2016-12-05T09:36:08Z
Fix tests in SparkLauncherSuite, UDFSuite, BroadcastJoinSuite, InsertSuite
and PathOptionSuite
commit b0074b51d13f430080f221000793b1c838082339
Author: hyukjinkwon <[email protected]>
Date: 2016-12-05T14:00:46Z
Fix Java style
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
