Repository: spark
Updated Branches:
refs/heads/branch-2.3 a9d0784e6 -> 72c13ed84
[SPARK-23695][PYTHON] Fix the error message for Kinesis streaming tests
## What changes were proposed in this pull request?
This PR proposes to fix the error message for Kinesis in PySpark when its jar
is missing but explicitly enabled.
```bash
ENABLE_KINESIS_TESTS=1 SPARK_TESTING=1 bin/pyspark pyspark.streaming.tests
```
Before:
```
Skipped test_flume_stream (enable by setting environment variable
ENABLE_FLUME_TESTS=1Skipped test_kafka_stream (enable by setting environment
variable ENABLE_KAFKA_0_8_TESTS=1Traceback (most recent call last):
File
"/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File
"/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 72, in _run_code
exec code in run_globals
File "/.../spark/python/pyspark/streaming/tests.py", line 1572, in <module>
% kinesis_asl_assembly_dir) +
NameError: name 'kinesis_asl_assembly_dir' is not defined
```
After:
```
Skipped test_flume_stream (enable by setting environment variable
ENABLE_FLUME_TESTS=1Skipped test_kafka_stream (enable by setting environment
variable ENABLE_KAFKA_0_8_TESTS=1Traceback (most recent call last):
File
"/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File
"/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 72, in _run_code
exec code in run_globals
File "/.../spark/python/pyspark/streaming/tests.py", line 1576, in <module>
"You need to build Spark with 'build/sbt -Pkinesis-asl "
Exception: Failed to find Spark Streaming Kinesis assembly jar in
/.../spark/external/kinesis-asl-assembly. You need to build Spark with
'build/sbt -Pkinesis-asl assembly/package
streaming-kinesis-asl-assembly/assembly'or 'build/mvn -Pkinesis-asl package'
before running this test.
```
## How was this patch tested?
Manually tested.
Author: hyukjinkwon <[email protected]>
Closes #20834 from HyukjinKwon/minor-variable.
(cherry picked from commit 56e8f48a43eb51e8582db2461a585b13a771a00a)
Signed-off-by: Takuya UESHIN <[email protected]>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/72c13ed8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/72c13ed8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/72c13ed8
Branch: refs/heads/branch-2.3
Commit: 72c13ed844d6be6510ce2c5e3526c234d1d5e10f
Parents: a9d0784
Author: hyukjinkwon <[email protected]>
Authored: Thu Mar 15 10:55:33 2018 -0700
Committer: Takuya UESHIN <[email protected]>
Committed: Thu Mar 15 10:55:45 2018 -0700
----------------------------------------------------------------------
python/pyspark/streaming/tests.py | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/72c13ed8/python/pyspark/streaming/tests.py
----------------------------------------------------------------------
diff --git a/python/pyspark/streaming/tests.py
b/python/pyspark/streaming/tests.py
index 5b86c1c..ca28c9b 100644
--- a/python/pyspark/streaming/tests.py
+++ b/python/pyspark/streaming/tests.py
@@ -1503,10 +1503,13 @@ def search_flume_assembly_jar():
return jars[0]
-def search_kinesis_asl_assembly_jar():
+def _kinesis_asl_assembly_dir():
SPARK_HOME = os.environ["SPARK_HOME"]
- kinesis_asl_assembly_dir = os.path.join(SPARK_HOME,
"external/kinesis-asl-assembly")
- jars = search_jar(kinesis_asl_assembly_dir,
"spark-streaming-kinesis-asl-assembly")
+ return os.path.join(SPARK_HOME, "external/kinesis-asl-assembly")
+
+
+def search_kinesis_asl_assembly_jar():
+ jars = search_jar(_kinesis_asl_assembly_dir(),
"spark-streaming-kinesis-asl-assembly")
if not jars:
return None
elif len(jars) > 1:
@@ -1569,7 +1572,7 @@ if __name__ == "__main__":
else:
raise Exception(
("Failed to find Spark Streaming Kinesis assembly jar in %s. "
- % kinesis_asl_assembly_dir) +
+ % _kinesis_asl_assembly_dir()) +
"You need to build Spark with 'build/sbt -Pkinesis-asl "
"assembly/package streaming-kinesis-asl-assembly/assembly'"
"or 'build/mvn -Pkinesis-asl package' before running this test.")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
