Hongze Zhang created SPARK-49133:
------------------------------------
Summary: Make MemoryConsumer thread-safe
Key: SPARK-49133
URL: https://issues.apache.org/jira/browse/SPARK-49133
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.5.1
Reporter: Hongze Zhang
TaskMemoryManager is thread-safe however MemoryConsumer is not.
Thinking of multiple threads are allocating memory in the same task (although
it's known not to be ideal regarding Spark's one-thread-per-task model), it
leads to issues that developer should add a lock in the caller code of
MemoryConsumer. However since it's common that one memory consumer spills
another, using lock in user code could easily causes ABBA dead lock, e.g.,
consumer A locks it self to acquire memory from task memory manager, while
consumer B locks task memory manager then spills A.
It's easy to make MemoryConsumer thread-safe. AFAICS the only change needed so
far is to make accesses of field `used` atomic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
