dongjoon-hyun opened a new pull request, #59568: URL: https://github.com/apache/spark/pull/59568
### What changes were proposed in this pull request? This PR fixes `PythonForeachWriter.UnsafeRowBuffer` so that `close()` cannot free memory that the `WriterThread` still reads. - `remove()` returns a copy of the row instead of a row pointing into a `HybridRowQueue` page. - `close()` frees the queue under the buffer's lock, taken uninterruptibly, and makes any later or waiting `remove()` throw. ### Why are the changes needed? `PythonForeachWriter` sends the buffered rows to the Python worker from its own `WriterThread`. If the task is killed while `close()` waits for that thread, for example by `query.stop()`, the task completes without stopping it. The task completion listener then frees the queue's pages while the `WriterThread` may still be reading a row from them, which can crash the executor when `spark.memory.offHeap.enabled=true`. In addition, if `allRowsAdded()` is interrupted, a `WriterThread` waiting for rows never wakes up. This bug has existed since SPARK-44705 ([v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0), 2025-05-19). ### Does this PR introduce _any_ user-facing change? Yes, this fixes a possible executor crash and thread leak when a task with a Python `foreach` writer is killed. ### How was this patch tested? Added two test cases to `PythonForeachWriterSuite`. Both fail without the fix: a row taken out before `close()` reads the `spark.memory.debugFill` pattern of a freed page, and a thread waiting for rows stays blocked after `close()`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
