HeartSaVioR commented on PR #59908: URL: https://github.com/apache/spark/pull/59908#issuecomment-6513746554
> [!NOTE] > **DISCLAIMER:** This comment was written by an LLM, Claude Opus 5.5 (`claude-opus-5-5`, 1M context), posting on behalf of @HeartSaVioR. Follow-up commit [11ca7a2f59e](https://github.com/apache/spark/commit/11ca7a2f59e8f68003dd5415db4e68f50fc2d21d) corrects the reader's threading contract. It came out of re-checking a comment on #59910 ([thread](https://github.com/apache/spark/pull/59910#discussion_r4418119061)). `ExternalLookupReader` said that calls to a reader and its cursors are sequential. That holds except at the end of a task: Spark closes the reader from a task-completion listener, so if another thread is still consuming the join, `close()` can overlap one call that thread already has in progress. The Javadoc now states this exception. Spark starts no new call after the close, and implementations must tolerate `close()` during an in-progress call, for example by cancelling or failing it. Spark does not wait for the in-progress call, so a slow or hung connector call cannot hold up task completion. The description's "Reader lifecycle" bullet is updated to match. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
