pralabhkumar opened a new pull request, #37417: URL: https://github.com/apache/spark/pull/37417
### What changes were proposed in this pull request? This PR will place spark.files , spark.jars and spark.pyfiles to the current working directory on the driver in K8s cluster mode ### Why are the changes needed? This mimics the behaviour of Yarn and also helps user to access files from PWD . Also as mentioned in the jira By doing this, users can, for example, leverage PEX to manage Python dependences in Apache Spark: ``` pex pyspark==3.0.1 pyarrow==0.15.1 pandas==0.25.3 -o myarchive.pex PYSPARK_PYTHON=./myarchive.pex spark-submit --files myarchive.pex ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tested via unit test cases and also ran on local K8s cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
