jayzhan211 opened a new issue, #26002: URL: https://github.com/apache/datafusion/issues/26002
### Describe the bug `SortMergeJoinExec` ignores a join filter that references no columns. For a `FULL JOIN` with such a filter (e.g. a volatile `random() > 2`), sort-merge join returns rows as matched even though the filter is false for every pair. Hash join returns the correct result. ### To Reproduce ```sql SET datafusion.optimizer.prefer_hash_join = false; SET datafusion.execution.target_partitions = 2; CREATE TABLE a AS SELECT * FROM (VALUES (1, 10), (2, 20), (3, 30)) t(k, v); CREATE TABLE b AS SELECT * FROM (VALUES (1, 100), (2, 200), (4, 400)) t(k, w); EXPLAIN SELECT * FROM a FULL JOIN b ON a.k = b.k AND random() > 2; -- SortMergeJoinExec: join_type=Full, on=[(k@0, k@0)], filter=random() > 2 SELECT * FROM a FULL JOIN b ON a.k = b.k AND random() > 2 ORDER BY a.k, b.k; +------+------+------+------+ | k | v | k | w | +------+------+------+------+ | 1 | 10 | 1 | 100 | | 2 | 20 | 2 | 200 | | 3 | 30 | NULL | NULL | | NULL | NULL | 4 | 400 | +------+------+------+------+ ``` (With `target_partitions = 1` the planner picks `HashJoinExec` and the result is correct.) ### Expected behavior `random() > 2` is never true, so no pair matches and every row is NULL-padded, as with `SET datafusion.optimizer.prefer_hash_join = true`: ``` +------+------+------+------+ | k | v | k | w | +------+------+------+------+ | 1 | 10 | NULL | NULL | | 2 | 20 | NULL | NULL | | 3 | 30 | NULL | NULL | | NULL | NULL | 1 | 100 | | NULL | NULL | 2 | 200 | | NULL | NULL | 4 | 400 | +------+------+------+------+ ``` ### Additional context Reproduced on current `main` (Oct 3 2026). `LEFT`, `RIGHT` and `INNER` joins with the same filter return correct results. In `sort_merge_join/materializing_stream.rs` the filter is only evaluated under `if !filter_columns.is_empty()`, so a filter whose expression reads no columns is skipped and every candidate pair is kept. A possible fix is to evaluate the filter even when it has no columns, building the filter batch with `RecordBatchOptions::with_row_count(...)` so its row count matches the candidate pairs. Related: #26001 (same code path, filter batch construction for outer joins). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
