Marcelo Vanzin created SPARK-26933: -------------------------------------- Summary: spark-submit does not make zip files provided with --py-files visible to pyspark Key: SPARK-26933 URL: https://issues.apache.org/jira/browse/SPARK-26933 Project: Spark Issue Type: Bug Components: PySpark, Spark Core Affects Versions: 2.4.0 Reporter: Marcelo Vanzin
I ran into this while testing the changes I made for SPARK-24736. If you provide, e.g., {{--py-files dependency.zip}} to spark-submit, that archive is not added to the pyspark app's PYTHONPATH. That makes the dependency not visible to the pyspark code. {{.py}} files added using {{--py-files}} work fine (if they're imported after a SparkContext is instantiated). -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org