Sunninsky commented on issue #26868: URL: https://github.com/apache/datafusion/issues/26868#issuecomment-6791317393
Three more candidates for 55.3.0, all in the sort-merge join (`datafusion/physical-plan/src/joins/sort_merge_join/`). As far as I can see none of them is on `branch-55` or in the list above: they are on `main` only (labelled `v56.0.0`), and the 55.1.0 and 55.2.0 changelogs have no sort-merge-join entry. Each one is a failed task or a wrong result, not a slowdown: - #25134 (fix for #25133): a mark, semi or anti sort-merge join on a floating-point key panics with ``called `Option::unwrap()` on a `None` value`` when a key group ends at the end of an input batch and the key is NaN. Through Comet this is a Spark `LEFT SEMI JOIN` / `LEFT ANTI JOIN` on a `DOUBLE` key with NaN rows, and Spark returns the rows. The cherry-pick (`-x`) onto `branch-55` applies cleanly. - #26027 (fix for #26001): a LEFT / RIGHT / FULL sort-merge join whose filter uses a column that is `NOT NULL` on the NULL-padded side fails with `Column '...' is declared as non-nullable but contains null values`. Through Comet that column is the `count(*)` of an aggregate directly under the join. The cherry-pick conflicts in `materializing_stream.rs` (around `filter_schema`). - #24573: with a join filter, LEFT / RIGHT / FULL emit their last filtered batch ahead of rows still staged in the output buffer, while the join advertises the order of its streamed side (`maintains_input_order`). A native aggregate above it trusts that order, closes a group early and returns it twice: 8 rows where Spark returns 7. The cherry-pick conflicts in `tests.rs` only. (Only 5 of its 20 hunks are the ordering fix; we carry just those in Comet.) We found them in DataFusion Comet, which vendors 55.2.0: Spark queries compared with Spark's own result, with the batch size set to 1 to 8 rows, because test data of ordinary size never makes a key group end at a batch boundary. On the 55.2.0 code the tests of #25134 (144 cases) and #24573 (3 ordering tests) fail, and with each fix they pass. The tests of #26027 are sqllogictests, so we wrote three small unit tests for it (LEFT, RIGHT and FULL, the filter column declared `NOT NULL` on the padded side), which fail with the error above on 55.2.0 and pass with the fix. Could they be added? I can open the backport PRs against `branch-55` if that helps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
