jayzhan211 commented on code in PR #26090:
URL: https://github.com/apache/datafusion/pull/26090#discussion_r4370152526
##########
datafusion/physical-plan/src/joins/sort_merge_join/exec.rs:
##########
@@ -493,6 +617,18 @@ impl ExecutionPlan for SortMergeJoinExec {
}
fn input_distribution_requirements(&self) -> InputDistributionRequirements
{
+ // Null-aware NOT IN needs facts about the entire right input.
+ // Require a single partition on both sides to use the existing
+ // sort-merge join's partition pairing without cross-partition
coordination.
+ //
+ // TODO: Allow multiple left partitions by sharing the right-side
summary
Review Comment:
I ran the issue's reproducer on this branch with `-m 100M --mem-pool-type
fair`, and the `NOT IN` query still fails, now in the sort: `Failed to allocate
additional 152.7 MB for ExternalSorter[0]`. The `unnest` produces a single
20M-row batch. With the left side pinned to one partition, nothing splits it
before the sort, whereas the `NOT EXISTS` plan hash-repartitions it into 4
pieces. With normally batched input the PR works well: hash join runs out of
memory and this path returns 19999000:
```sql
create view big as select value as x from generate_series(1, 20000000);
create view small as select value as y from generate_series(1, 1000);
select count(*) from big where x not in (select y from small);
```
Could you update the PR description to use this reproducer and show its
actual output? The current output only covers the `SET`/`CREATE` statements.
It's fine to handle the giant-batch case later with this TODO; please link the
TODO to a tracking issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
