Huanli Wang created SPARK-45942:
-----------------------------------
Summary: Only do the thread interruption check for putIterator on
executors
Key: SPARK-45942
URL: https://issues.apache.org/jira/browse/SPARK-45942
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Huanli Wang
https://issues.apache.org/jira/browse/SPARK-45025
introduces a peaceful thread interruption handling. However, there is an edge
case: when a streaming query is stopped on the driver, it interrupts the stream
execution thread. If the streaming query is doing memory store operations on
driver and performs {{doPutIterator}} at the same time, the [unroll process
will be
broken|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L224]
and [returns used
memory|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L245-L247].
This can result in {{closeChannelException}} as it falls into this [case
clause|https://github.com/apache/spark/blob/aa646d3050028272f7333deaef52f20e6975e0ed/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1614-L1622]
which opens an I/O channel and persists the data into the disk. However,
because the thread is interrupted, the channel will be closed at the begin:
[https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/nio/channels/spi/AbstractInterruptibleChannel.java#L172]
and throws out {{closeChannelException}}
On executors, [the task will be killed if the thread is
interrupted|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L374],
however, we don't do it on the driver.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
