Marcelo Vanzin created SPARK-26934: -------------------------------------- Summary: python dependencies with "local:" URIs are not visible to executors Key: SPARK-26934 URL: https://issues.apache.org/jira/browse/SPARK-26934 Project: Spark Issue Type: Bug Components: PySpark, Spark Core Affects Versions: 2.4.0 Reporter: Marcelo Vanzin
Noticed this when running tests for SPARK-24736. If you provide a file with a "local:" URI to {{--py-files}}, spark-submit makes it visible to the driver, but not executors. That makes code that runs in executors and references those files fail. The underlying issue is that when providing dependency information to executors, when running tasks, {{SparkContext}} erases any references to files with a "local:" scheme, so executors don't know they exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org