[
https://issues.apache.org/jira/browse/SPARK-59233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng reassigned SPARK-59233:
-------------------------------------
Assignee: Kaicheng Zhou
> Pass the session's Python worker environment to every Python function family
> ----------------------------------------------------------------------------
>
> Key: SPARK-59233
> URL: https://issues.apache.org/jira/browse/SPARK-59233
> Project: Spark
> Issue Type: Task
> Components: PySpark, Spark Core
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Kaicheng Zhou
> Assignee: Kaicheng Zhou
> Priority: Major
> Labels: pull-request-available
>
> SPARK-58752 added the {{spark.pythonWorkerEnv.*}} session configurations,
> which set environment variables for the Python workers that run a session's
> Python functions. It
> installed them for the regular scalar Python UDF only, so the same logic
> written as a pandas UDF, a UDTF or {{applyInPandas}} received nothing — a
> difference users cannot
> predict from anything in the API.
> Extend the environment to every Python worker a session launches for a Python
> function that session supplied: scalar UDFs in all serialization modes,
> {{mapInPandas}} and
> {{{}mapInArrow{}}}, grouped-map, cogrouped-map, grouped-aggregate and window
> functions, row and Arrow UDTFs, {{applyInPandasWithState}} and
> {{{}transformWithState{}}}, Python data sources including the planning and
> streaming-reader workers, and the {{foreachBatch}} worker.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
