GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/17987
[WIP][SPARK-19707][SPARK-18922][TESTS][SQL][CORE] Fix test failures/the
invalid path check for sc.addJar on Windows
## What changes were proposed in this pull request?
This PR proposes two things:
- A follow up for SPARK-19707 (Improving the invalid path check for
sc.addJar on Windows as well).
```
org.apache.spark.SparkContextSuite:
- add jar with invalid path *** FAILED *** (32 milliseconds)
2 was not equal to 1 (SparkContextSuite.scala:309)
...
```
- Fix path vs URI related test failures on Windows.
```
org.apache.spark.storage.LocalDirsSuite:
- SPARK_LOCAL_DIRS override also affects driver *** FAILED *** (0
milliseconds)
new java.io.File("/NONEXISTENT_PATH").exists() was true
(LocalDirsSuite.scala:50)
...
- Utils.getLocalDir() throws an exception if any temporary directory
cannot be retrieved *** FAILED *** (15 milliseconds)
Expected exception java.io.IOException to be thrown, but no exception
was thrown. (LocalDirsSuite.scala:64)
...
```
```
org.apache.spark.sql.hive.HiveSchemaInferenceSuite:
- orc: schema should be inferred and saved when INFER_AND_SAVE is
specified *** FAILED *** (203 milliseconds)
java.net.URISyntaxException: Illegal character in opaque part at index
2: C:\projects\spark\target\tmp\spark-dae61ab3-a851-4dd3-bf4e-be97c501f254
...
- parquet: schema should be inferred and saved when INFER_AND_SAVE is
specified *** FAILED *** (203 milliseconds)
java.net.URISyntaxException: Illegal character in opaque part at index
2: C:\projects\spark\target\tmp\spark-fa3aff89-a66e-4376-9a37-2a9b87596939
...
- orc: schema should be inferred but not stored when INFER_ONLY is
specified *** FAILED *** (141 milliseconds)
java.net.URISyntaxException: Illegal character in opaque part at index
2: C:\projects\spark\target\tmp\spark-fb464e59-b049-481b-9c75-f53295c9fc2c
...
- parquet: schema should be inferred but not stored when INFER_ONLY is
specified *** FAILED *** (125 milliseconds)
java.net.URISyntaxException: Illegal character in opaque part at index
2: C:\projects\spark\target\tmp\spark-9487568e-80a4-42b3-b0a5-d95314c4ccbc
...
- orc: schema should not be inferred when NEVER_INFER is specified ***
FAILED *** (156 milliseconds)
java.net.URISyntaxException: Illegal character in opaque part at index
2: C:\projects\spark\target\tmp\spark-0d2dfa45-1b0f-4958-a8be-1074ed0135a
...
- parquet: schema should not be inferred when NEVER_INFER is specified ***
FAILED *** (547 milliseconds)
java.net.URISyntaxException: Illegal character in opaque part at index
2: C:\projects\spark\target\tmp\spark-6d95d64e-613e-4a59-a0f6-d198c5aa51ee
...
```
```
org.apache.spark.sql.execution.command.DDLSuite:
- create temporary view using *** FAILED *** (15 milliseconds)
org.apache.spark.sql.AnalysisException: Path does not exist:
file:/C:projectsspark arget
mpspark-3881d9ca-561b-488d-90b9-97587472b853 mp;
...
- insert data to a data source table which has a non-existing location
should succeed *** FAILED *** (109 milliseconds)
file:/C:projectsspark%09arget%09mpspark-4cad3d19-6085-4b75-b407-fe5e9d21df54
did not equal
file:///C:/projects/spark/target/tmp/spark-4cad3d19-6085-4b75-b407-fe5e9d21df54
(DDLSuite.scala:1869)
...
- insert into a data source table with a non-existing partition location
should succeed *** FAILED *** (94 milliseconds)
file:/C:projectsspark%09arget%09mpspark-4b52e7de-e3aa-42fd-95d4-6d4d58d1d95d
did not equal
file:///C:/projects/spark/target/tmp/spark-4b52e7de-e3aa-42fd-95d4-6d4d58d1d95d
(DDLSuite.scala:1910)
...
- read data from a data source table which has a non-existing location
should succeed *** FAILED *** (93 milliseconds)
file:/C:projectsspark%09arget%09mpspark-f8c281e2-08c2-4f73-abbf-f3865b702c34
did not equal
file:///C:/projects/spark/target/tmp/spark-f8c281e2-08c2-4f73-abbf-f3865b702c34
(DDLSuite.scala:1937)
...
- read data from a data source table with non-existing partition location
should succeed *** FAILED *** (110 milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
- create datasource table with a non-existing location *** FAILED *** (94
milliseconds)
file:/C:projectsspark%09arget%09mpspark-387316ae-070c-4e78-9b78-19ebf7b29ec8
did not equal
file:///C:/projects/spark/target/tmp/spark-387316ae-070c-4e78-9b78-19ebf7b29ec8
(DDLSuite.scala:1982)
...
- CTAS for external data source table with a non-existing location ***
FAILED *** (16 milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
- CTAS for external data source table with a existed location *** FAILED
*** (15 milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
- data source table:partition column name containing a b *** FAILED ***
(125 milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
- data source table:partition column name containing a:b *** FAILED ***
(143 milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
- data source table:partition column name containing a%b *** FAILED ***
(109 milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
- data source table:partition column name containing a,b *** FAILED ***
(109 milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
- location uri contains a b for datasource table *** FAILED *** (94
milliseconds)
file:/C:projectsspark%09arget%09mpspark-5739cda9-b702-4e14-932c-42e8c4174480a%20b
did not equal
file:///C:/projects/spark/target/tmp/spark-5739cda9-b702-4e14-932c-42e8c4174480/a%20b
(DDLSuite.scala:2084)
...
- location uri contains a:b for datasource table *** FAILED *** (78
milliseconds)
file:/C:projectsspark%09arget%09mpspark-9bdd227c-840f-4f08-b7c5-4036638f098da:b
did not equal
file:///C:/projects/spark/target/tmp/spark-9bdd227c-840f-4f08-b7c5-4036638f098d/a:b
(DDLSuite.scala:2084)
...
- location uri contains a%b for datasource table *** FAILED *** (78
milliseconds)
file:/C:projectsspark%09arget%09mpspark-62bb5f1d-fa20-460a-b534-cb2e172a3640a%25b
did not equal
file:///C:/projects/spark/target/tmp/spark-62bb5f1d-fa20-460a-b534-cb2e172a3640/a%25b
(DDLSuite.scala:2084)
...
- location uri contains a b for database *** FAILED *** (16 milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
- location uri contains a:b for database *** FAILED *** (15 milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
- location uri contains a%b for database *** FAILED *** (0 milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
```
```
org.apache.spark.sql.hive.execution.HiveDDLSuite:
- create hive table with a non-existing location *** FAILED *** (16
milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
- CTAS for external hive table with a non-existing location *** FAILED ***
(16 milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
- CTAS for external hive table with a existed location *** FAILED *** (16
milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
- partition column name of parquet table containing a b *** FAILED ***
(156 milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
- partition column name of parquet table containing a:b *** FAILED *** (94
milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
- partition column name of parquet table containing a%b *** FAILED ***
(125 milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
- partition column name of parquet table containing a,b *** FAILED ***
(110 milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
- partition column name of hive table containing a b *** FAILED *** (15
milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
- partition column name of hive table containing a:b *** FAILED *** (16
milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
- partition column name of hive table containing a%b *** FAILED *** (16
milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
- partition column name of hive table containing a,b *** FAILED *** (0
milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
- hive table: location uri contains a b *** FAILED *** (0 milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
- hive table: location uri contains a:b *** FAILED *** (0 milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
- hive table: location uri contains a%b *** FAILED *** (0 milliseconds)
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:java.lang.IllegalArgumentException: Can not create a Path
from an empty string);
...
```
```
org.apache.spark.sql.sources.PathOptionSuite:
- path option also exist for write path *** FAILED *** (94 milliseconds)
file:/C:projectsspark%09arget%09mpspark-2870b281-7ac0-43d6-b6b6-134e01ab6fdc
did not equal
file:///C:/projects/spark/target/tmp/spark-2870b281-7ac0-43d6-b6b6-134e01ab6fdc
(PathOptionSuite.scala:98)
...
```
```
org.apache.spark.sql.CachedTableSuite:
- SPARK-19765: UNCACHE TABLE should un-cache all cached plans that refer
to this table *** FAILED *** (110 milliseconds)
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
...
```
```
org.apache.spark.sql.execution.DataSourceScanExecRedactionSuite:
- treeString is redacted *** FAILED *** (250 milliseconds)
"file:/C:/projects/spark/target/tmp/spark-3ecc1fa4-3e76-489c-95f4-f0b0500eae28"
did not contain
"C:\projects\spark\target\tmp\spark-3ecc1fa4-3e76-489c-95f4-f0b0500eae28"
(DataSourceScanExecRedactionSuite.scala:46)
...
```
## How was this patch tested?
Tested via AppVeyor for each and checked it passed once each. These should
be retested via AppVeyor in this PR.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark windows-20170515
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17987.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17987
----
commit ea113211b09d0b9b7876472d8bcb47476b6f598d
Author: hyukjinkwon <[email protected]>
Date: 2017-05-15T10:00:31Z
Fix test failures on Windows
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
