siying opened a new pull request, #60068: URL: https://github.com/apache/spark/pull/60068
### What changes were proposed in this pull request? This PR makes stateful stream-stream joins retain watermarked input attributes as implicit references. This prevents column pruning from removing event-time attributes that the physical join needs for late-row filtering, even when event time is not part of the join condition or the downstream projection. It also adds a regression test for an inner stream-stream join where both inputs have watermarks and the final projection selects only the join key. ### Why are the changes needed? When a downstream projection does not use either event-time column, `ColumnPruning` can insert projects below the join that retain only the join key. The stream-stream join then loses the watermark metadata needed to reject late rows and can incorrectly emit a late matching pair. ### Does this PR introduce _any_ user-facing change? Yes. Stateful stream-stream joins now correctly discard late input rows when watermarked event-time columns are projected away downstream. ### How was this patch tested? Added a regression test and ran it with: - `StreamingInnerJoinWithoutVCFSuite` using RocksDB and HDFS state stores - `StreamingInnerJoinWithVCFSuite` with changelog checkpointing enabled and disabled - Catalyst Scalastyle - SQL test Scalastyle The regression test fails without the source change by emitting the late key and passes with the fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
