SubhamSinghal opened a new issue, #26363:
URL: https://github.com/apache/datafusion/issues/26363
### Describe the bug
With `datafusion.optimizer.enable_piecewise_merge_join = true`, every
existence join that `PiecewiseMergeJoinExec` plans (`LeftSemi`, `LeftAnti`,
`RightSemi`, `RightAnti`, so `EXISTS` / `NOT EXISTS` with a range predicate)
fails when the join key is `RunEndEncoded`:
```
Internal error: Min/Max accumulator not implemented for type
RunEndEncoded(non-null Int32, Int32).
```
The same queries return the right rows with the flag off, and PWMJ's classic
joins (`INNER` / `LEFT` / `RIGHT` / `FULL`) return the right rows on the same
REE tables.
### To Reproduce
```sql
CREATE TABLE l AS SELECT column1 AS id, arrow_cast(column2,
'RunEndEncoded("run_ends": non-null Int32, "values": Int32)') AS v
FROM (VALUES (1, 1), (2, 1), (3, NULL), (4, 5));
CREATE TABLE r AS SELECT column1 AS rid, arrow_cast(column2,
'RunEndEncoded("run_ends": non-null Int32, "values": Int32)') AS v
FROM (VALUES (20, 2), (21, NULL), (22, 6));
SET datafusion.optimizer.enable_piecewise_merge_join = true;
-- LeftSemi / LeftAnti
SELECT id FROM l WHERE EXISTS (SELECT 1 FROM r WHERE l.v < r.v);
SELECT id FROM l WHERE NOT EXISTS (SELECT 1 FROM r WHERE l.v < r.v);
-- RightSemi / RightAnti
SELECT r.rid FROM l RIGHT SEMI JOIN r ON l.v < r.v;
SELECT r.rid FROM l RIGHT ANTI JOIN r ON l.v < r.v;
```
All four fail with the internal error above. The plan is
`PiecewiseMergeJoin: operator=Lt, join_type=LeftSemi, ...` (resp. `RightSemi`,
...).
### Expected behavior
The rows the flag-off plan returns:
| query | expected |
|---|---|
| `EXISTS` | 1, 2, 4 |
| `NOT EXISTS` | 3 |
| `RIGHT SEMI` | 20, 22 |
| `RIGHT ANTI` | 21 |
### Additional context
The existence paths reduce a batch of keys to a single extreme key in
`extreme_key` (`joins/piecewise_merge_join/existence_join.rs`), used by the
left existence stream and by the right existence buffered fold in `exec.rs`.
For non-nested keys it calls `min_batch` / `max_batch` from
`datafusion-functions-aggregate-common`, which have no `RunEndEncoded` arm.
Classic joins never take the extreme, which is why they are unaffected.
#26247 / #26248 report the same error message for `MIN`/`MAX` aggregates,
but that fix is in `get_min_max_result_type`: the planner casts REE to its
value type before the accumulator runs. PWMJ calls the kernels directly on the
evaluated join keys, so it does not go through that coercion and #26248 does
not fix this.
### Willingness to contribute
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
