[
https://issues.apache.org/jira/browse/SPARK-49133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongze Zhang updated SPARK-49133:
---------------------------------
Description:
TaskMemoryManager is thread-safe however MemoryConsumer is not.
Thinking of multiple threads are allocating memory in the same task (although
it's known not to be ideal regarding Spark's one-thread-per-task model), it
leads to practices that developer should add a lock in the caller code of
MemoryConsumer. However since it's common that one memory consumer spills
another, using lock in user code could easily causes ABBA dead lock, e.g.,
consumer A locks it self to acquire memory from task memory manager, while
consumer B locks task memory manager then spills A.
It's easy to make MemoryConsumer thread-safe. AFAICS the only change needed so
far is to make accesses of field `used` atomic.
was:
TaskMemoryManager is thread-safe however MemoryConsumer is not.
Thinking of multiple threads are allocating memory in the same task (although
it's known not to be ideal regarding Spark's one-thread-per-task model), it
leads to issues that developer should add a lock in the caller code of
MemoryConsumer. However since it's common that one memory consumer spills
another, using lock in user code could easily causes ABBA dead lock, e.g.,
consumer A locks it self to acquire memory from task memory manager, while
consumer B locks task memory manager then spills A.
It's easy to make MemoryConsumer thread-safe. AFAICS the only change needed so
far is to make accesses of field `used` atomic.
> Make MemoryConsumer thread-safe
> -------------------------------
>
> Key: SPARK-49133
> URL: https://issues.apache.org/jira/browse/SPARK-49133
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.5.1
> Reporter: Hongze Zhang
> Priority: Major
>
> TaskMemoryManager is thread-safe however MemoryConsumer is not.
> Thinking of multiple threads are allocating memory in the same task (although
> it's known not to be ideal regarding Spark's one-thread-per-task model), it
> leads to practices that developer should add a lock in the caller code of
> MemoryConsumer. However since it's common that one memory consumer spills
> another, using lock in user code could easily causes ABBA dead lock, e.g.,
> consumer A locks it self to acquire memory from task memory manager, while
> consumer B locks task memory manager then spills A.
> It's easy to make MemoryConsumer thread-safe. AFAICS the only change needed
> so far is to make accesses of field `used` atomic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
