HyukjinKwon opened a new pull request, #41215: URL: https://github.com/apache/spark/pull/41215
### What changes were proposed in this pull request? This PR proposes a new configuration `spark.sql.execution.pyspark.python` that sets the Python executable on worker nodes. Note that, if the Python executable is different from the one previously ran, it will creates a new Python worker processes because we reuse Python workers but they are unique by both executable path and env variables as a key: https://github.com/apache/spark/blob/d7a8b852eaa6cc04df1eea0018a9b9de29b1c4fe/core/src/main/scala/org/apache/spark/SparkEnv.scala#L123-L124 This PR is also a basework for Spark Connect to support a different set of dependencies. ### Why are the changes needed? This can be useful especially when you want to run your Python with a different set of dependencies during runtime (see also https://www.databricks.com/blog/2020/12/22/how-to-manage-python-dependencies-in-pyspark.html) ### Does this PR introduce _any_ user-facing change? No, this PR adds a configuration but that's internal for now. ### How was this patch tested? Manually tested as below: ```python import sys from pyspark.sql.functions import udf spark.range(1).select(udf(lambda x: sys.executable)("id")).show(truncate=False) spark.conf.set("spark.sql.execution.pyspark.python", "/Users/hyukjin.kwon/miniconda3/envs/another-python/bin/python") spark.range(1).select(udf(lambda x: sys.executable)("id")).show(truncate=False) ``` ``` +---------------------------------------------------------+ |<lambda>(id) | +---------------------------------------------------------+ |/.../miniconda3/envs/python3.9/bin/python3| +---------------------------------------------------------+ +-------------------------------------------------------------+ |<lambda>(id) | +-------------------------------------------------------------+ |/.../miniconda3/envs/another-python/bin/python| +-------------------------------------------------------------+ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
