[
https://issues.apache.org/jira/browse/SPARK-45942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-45942:
-----------------------------------
Labels: pull-request-available (was: )
> Only do the thread interruption check for putIterator on executors
> ------------------------------------------------------------------
>
> Key: SPARK-45942
> URL: https://issues.apache.org/jira/browse/SPARK-45942
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Huanli Wang
> Priority: Major
> Labels: pull-request-available
>
> https://issues.apache.org/jira/browse/SPARK-45025
> introduces a peaceful thread interruption handling. However, there is an edge
> case: when a streaming query is stopped on the driver, it interrupts the
> stream execution thread. If the streaming query is doing memory store
> operations on driver and performs {{doPutIterator}} at the same time, the
> [unroll process will be
> broken|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L224]
> and [returns used
> memory|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L245-L247].
> This can result in {{closeChannelException}} as it falls into this [case
> clause|https://github.com/apache/spark/blob/aa646d3050028272f7333deaef52f20e6975e0ed/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1614-L1622]
> which opens an I/O channel and persists the data into the disk. However,
> because the thread is interrupted, the channel will be closed at the begin:
> [https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/nio/channels/spi/AbstractInterruptibleChannel.java#L172]
> and throws out {{closeChannelException}}
> On executors, [the task will be killed if the thread is
> interrupted|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L374],
> however, we don't do it on the driver.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
