viirya commented on code in PR #9163:
URL: https://github.com/apache/arrow-datafusion/pull/9163#discussion_r1482631795
##########
datafusion/physical-plan/src/joins/sort_merge_join.rs:
##########
@@ -1209,7 +1209,14 @@ impl SMJStream {
) {
// The reverse of the selection mask. For the rows not
pass join filter above,
// we need to join them (left or right) with null rows
for outer joins.
- let not_mask = compute::not(mask)?;
+ let not_mask = if mask.null_count() > 0 {
+ // If the mask contains nulls, we need to use
`prep_null_mask_filter` to
+ // handle the nulls in the mask as false.
+
compute::not(&compute::prep_null_mask_filter(mask))?
Review Comment:
Using the test in `sort_merge_join.slt` as example:
```
CREATE TABLE IF NOT EXISTS t1(t1_id INT, t1_name TEXT, t1_int INT) AS VALUES
(11, 'a', 1),
(22, 'b', 2),
(33, 'c', 3),
(44, 'd', 4);
CREATE TABLE IF NOT EXISTS t2(t2_id INT, t2_name TEXT, t2_int INT) AS VALUES
(11, 'z', 3),
(22, 'y', 1),
(44, 'x', 3),
(55, 'w', 3);
```
For the query `SELECT t1_id, t1_int, t2_int FROM t1 LEFT JOIN t2 ON t1_id =
t2_id AND t1_int >= t2_int`, `33 3 NULL` is one row to output without join
filter. Join filter returns null on this row, so its value in mask selection
array is null and the row is not picked.
Here we take reverse mask. As it is null, its reverse mask is still false,
but this row should be selected here actually. So we need to call
`prep_null_mask_filter` to process the mask before taking reverse array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
