shanyu zhao created SPARK-30845: ----------------------------------- Summary: spark-submit pyspark app on yarn uploads local pyspark archives Key: SPARK-30845 URL: https://issues.apache.org/jira/browse/SPARK-30845 Project: Spark Issue Type: Bug Components: Spark Submit Affects Versions: 2.4.5, 2.4.4, 2.4.3, 2.4.2, 2.4.1, 2.4.0 Reporter: shanyu zhao
Use spark-submit to submit a pyspark app on Yarn, and set this in spark-env.sh: {code:bash} export PYSPARK_ARCHIVES_PATH=local:/opt/spark/python/lib/pyspark.zip,local:/opt/spark/python/lib/py4j-0.10.7-src.zip {code} You can see that these local archives are still uploaded to Yarn distributed cache. yarn.Client: Uploading resource file:/opt/spark/python/lib/pyspark.zip -> hdfs://myhdfs/user/test1/.sparkStaging/application_1581024490249_0001/pyspark.zip -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org