sarutak commented on PR #59914:
URL: https://github.com/apache/spark/pull/59914#issuecomment-6509475142
Decision for this PR: keep the current meaning of `is_interrupted` ("whether
the task was killed", as `spark_datasource.h` documents it). Until the
follow-up, a `get_next` that a completed task abandoned runs until it returns
on its own, bounded only by the library's timeouts, and keeps its thread, the
stream state, the task context and the reader's child allocator until then.
- This PR fixes the use-after-free: the stream and the task context are
released only once `get_next` returns. Telling the library to stop changes the
C interface, so it is a separate change: SPARK-60733.
- SPARK-60733 also covers the docs point: `spark_datasource.h` (`read`) and
`docs/sql-native-data-sources.md` describe the release timing only for a killed
task, not for a read that outlives a completed task.
- The task-context probe added in d4d5c202ca8 relies on the current meaning,
as you noted. If SPARK-60733 makes `is_interrupted` true once the reader is
closed, it will change the probe to check that the context is still registered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
