sarutak commented on PR #59940: URL: https://github.com/apache/spark/pull/59940#issuecomment-6637512034
Thank you for the fourth round, @dongjoon-hyun. Items 1 and 2 reproduce end to end as you described: with the split on, both queries fail with `DIVIDE_BY_ZERO`, and with the split off they return the 8 rows. I pushed cbfe1acf920, which fixes them as you suggest, pins the change for streaming checkpoints (item 8), and handles items 5, 7, 10 and 11; replies are inline. I also merged master again, since the CI of the previous head failed on the Java 21 requirement of SPARK-60669 with this branch's older workflow files. On items 3, 4 and 9, I agree that the other changes should come first. I plan to open them as separate PRs against master and rebase this PR on them once they are in: SPARK-60731 for the written order of the split, the change of the `FilterExec` subexpression path to compute each column only before the first condition reading it, the change of `InferFiltersFromConstraints` to add the inferred filters after the condition, and, for item 6, an allowlist change for arithmetic and casts that can not fail outside ANSI mode. This PR can then drop the state that only works around them, so there is no hurry to merge it before those. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
