[
https://issues.apache.org/jira/browse/SPARK-61166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61166.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 357
[https://github.com/apache/spark-connect-rust/pull/357]
> Make the threads of inheritable_thread_target share the copy of the tags of
> the session, in the Rust and Python clients, as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61166
> URL: https://issues.apache.org/jira/browse/SPARK-61166
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's inheritable_thread_target(session) copies the thread_local
> of the client, of the tags of the thread, when it decorates a function, and
> gives every thread that runs the function that one copy: the threads share
> it. So a thread that the function starts from another thread of the function
> sees the tags that that thread added, and the tags of the decorating thread
> stay its own.
> The Python client copies the tags into each thread instead, so each thread
> starts from the tags of the decorating thread and the threads do not share
> the tags that they add. Apache Spark's test_inheritable_tags_with_deco of
> test_parity_job_cancellation.py fails on this client (the nested thread lacks
> the tag "b" of the thread that started it).
> The thread-local user context extensions, which PySpark's decorator copies in
> the same thread_local, are not propagated by the client's decorator at all;
> that is not changed here.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
