Tian Gao created SPARK-54158:
--------------------------------
Summary: lock in ProfilerCollector is not really doing anything
Key: SPARK-54158
URL: https://issues.apache.org/jira/browse/SPARK-54158
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 4.1.0
Reporter: Tian Gao
We had a lock in ProfilerCollector, but it's not really doing anything. It
protects plenty of atomic operations like "stats = self._perf_profile_results",
which is a shallow copy anyway. You'll still read and modify later.
The whole code is still not thread-safe (the accumulator can just modify the
result anytime without any locking mechanism).
We need to decide whether we want it to be thread-safe (I assume we want for
continuous profiling). If we do, we need to do it properly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
