Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21198#discussion_r185158778
--- Diff: python/pyspark/streaming/tests.py ---
@@ -1549,7 +1549,9 @@ def search_kinesis_asl_assembly_jar():
kinesis_jar_present = True
jars = "%s,%s,%s" % (kafka_assembly_jar, flume_assembly_jar,
kinesis_asl_assembly_jar)
- os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars %s pyspark-shell" % jars
+ existing_args = os.environ.get("PYSPARK_SUBMIT_ARGS")
--- End diff --
I think there could be ... at least I know one informal way like
`SPARK_TESTING=1 pyspark pyspark.sql.tests` for a quicker test. It wouldn't
hard to have one if statement.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
