dongjoon-hyun commented on PR #60117:
URL: https://github.com/apache/spark/pull/60117#issuecomment-6663620411
Thank you for checking. Yes, this is for `master` and `branch-4.x`.
1. The difference for new queries comes from SPARK-59693, not from this PR.
Keeping the filter above `EventTimeWatermark` follows the query as written, and
dropping a row older than the watermark is within its guarantee, which only
covers rows within the delay. I'd leave a migration guide entry, if any, to a
separate change.
2. For a stream-stream join, no. `PushPredicateThroughJoin` does not push a
throwable conjunct into a join side in 4.3.0 either, so the columns it reads
stay above the join in both versions, and the state schema is the same. I ran
it with `where(s != '' AND CASE WHEN lv < -100 THEN raise_error('neg') ELSE lv
> 0 END)` over a join, where `s` is an `array_join` alias over `rv`. A 4.3.0
checkpoint restarted on `master` gave the same rows and state row counts. The
watermark case does change, though. The same filter over `withWatermark` gave
count 2 on 4.3.0 and count 1 after the restart, as in Issue 2. I ran it only on
`master`, but 4.2.2 and 4.3.1 have the same code. I left it as is, because
pushing the throwable conjunct down again would bring back the evaluation-order
bug that SPARK-60602 fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
