j1wonpark opened a new pull request, #57653: URL: https://github.com/apache/spark/pull/57653
### What changes were proposed in this pull request? This PR propagates the operation tags from the `writeStream.start()` request to the Spark Connect session used by a Python `foreachBatch` callback. The tags are transferred through the internal server-worker protocol when the Python worker starts and restored before processing micro-batches. A regression test verifies that the callback session preserves the tags and can execute a nested action with them. ### Why are the changes needed? Spark Connect executes Python `foreachBatch` callbacks in a separate Python worker that creates a local Connect client. Operation tags are client thread-local, so the new client does not contain the tags from the originating request. As a result, operations executed inside the callback, such as actions, writes, and SQL commands, cannot be identified or interrupted using those tags. ### Does this PR introduce _any_ user-facing change? Yes. Operations executed inside a Spark Connect Python `foreachBatch` callback now inherit the operation tags captured when the streaming query starts. This does not introduce a new API. ### How was this patch tested? - Added `test_streaming_foreach_batch_operation_tags` and verified that it fails before the fix and passes after the fix. - `python/run-tests --testnames pyspark.sql.tests.connect.streaming.test_parity_foreach_batch` - `build/sbt -Phive package` - `build/sbt connect/scalastyle` - `SparkConnectSessionHolderSuite` with the `python foreachBatch process` test filter ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5.6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
