Kaicheng Zhou created SPARK-59233:
-------------------------------------
Summary: Pass the session's Python worker environment to every
Python function family
Key: SPARK-59233
URL: https://issues.apache.org/jira/browse/SPARK-59233
Project: Spark
Issue Type: Task
Components: PySpark, Spark Core
Affects Versions: 5.0.0, 4.4.0
Reporter: Kaicheng Zhou
SPARK-58752 added the {{spark.pythonWorkerEnv.*}} session configurations, which
set environment variables for the Python workers that run a session's Python
functions. It
installed them for the regular scalar Python UDF only, so the same logic
written as a pandas UDF, a UDTF or {{applyInPandas}} received nothing — a
difference users cannot
predict from anything in the API.
Extend the environment to every Python worker a session launches for a Python
function that session supplied: scalar UDFs in all serialization modes,
{{mapInPandas}} and
{{{}mapInArrow{}}}, grouped-map, cogrouped-map, grouped-aggregate and window
functions, row and Arrow UDTFs, {{applyInPandasWithState}} and
{{{}transformWithState{}}}, Python data sources including the planning and
streaming-reader workers, and the {{foreachBatch}} worker.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
