[ 
https://issues.apache.org/jira/browse/SPARK-60733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146915#comment-18146915
 ] 

Gengliang Wang commented on SPARK-60733:
----------------------------------------

SPARK-60726 already tells a library to stop a read once its task was killed, 
failed or completed: {{is_interrupted}} of the task context of a read then 
returns true.

https://github.com/apache/spark/pull/60640 proposed to also return true once 
the task starts to complete, before its pre-completion listeners run, and once 
Spark closes the reader while another thread is in {{get_next}}. I closed that 
PR: its main use was as the base of https://github.com/apache/spark/pull/60641 
(SPARK-61195), which is closed in favor of the smaller SPARK-62031, which does 
not need it.

The two remaining gaps are narrow: a thread that a pre-completion listener 
waits for while it is in {{get_next}}, which today is only the writer of 
pipelined Python UDFs ({{spark.python.udf.pipelined.enabled}}, off by default), 
and a reader that is closed outside of a task. The PR can be reopened if they 
matter.

> Tell a native data source library to stop a read once Spark closes its reader
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-60733
>                 URL: https://issues.apache.org/jira/browse/SPARK-60733
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: Kousuke Saruta
>            Priority: Major
>              Labels: pull-request-available
>




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to