[
https://issues.apache.org/jira/browse/SPARK-61220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61220:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> inheritable_thread_target does not carry the thread-local user context
> extensions to the threads that run the function
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61220
> URL: https://issues.apache.org/jira/browse/SPARK-61220
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The decorator copies and shares only the tags of the client's thread-local
> state, not the user context extensions that the decorating thread added
> (per-thread in the client since SPARK-61186).
> Official PySpark deep-copies every attribute of session.client.thread_local
> and sets them in each thread that runs the function, so an extension added
> before decorating (such as the pipelines add_pipeline_analysis_context one)
> is missing from those threads' requests.
> Repro: add a thread-local extension, decorate a function with
> inheritable_thread_target(spark), and count the extensions inside it: ours
> sees 0, official sees 1.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
