Hyukjin Kwon created SPARK-61220:
------------------------------------
Summary: inheritable_thread_target does not carry the thread-local
user context extensions to the threads that run the function
Key: SPARK-61220
URL: https://issues.apache.org/jira/browse/SPARK-61220
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The decorator copies and shares only the tags of the client's thread-local
state, not the user context extensions that the decorating thread added
(per-thread in the client since SPARK-61186).
Official PySpark deep-copies every attribute of session.client.thread_local and
sets them in each thread that runs the function, so an extension added before
decorating (such as the pipelines add_pipeline_analysis_context one) is missing
from those threads' requests.
Repro: add a thread-local extension, decorate a function with
inheritable_thread_target(spark), and count the extensions inside it: ours sees
0, official sees 1.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
