mbutrovich opened a new issue, #12001: URL: https://github.com/apache/arrow-rs/issues/12001
A predicate shorter than the array works today: the rows past its end are not selected. The rustdoc of [`filter`](https://github.com/apache/arrow-rs/blob/9ffd63d528e924d9c0faceef0027e3766b615c98/arrow-select/src/filter.rs#L175-L206) does not say so, so I can't tell whether callers may rely on it. ```rust let values = Int32Array::from(vec![1, 2, 3, 4, 5]); let predicate = BooleanArray::from(vec![true, false, true]); let out = filter(&values, &predicate).unwrap(); // [1, 3]: rows 3 and 4 are not selected assert_eq!(out.as_primitive::<Int32Type>(), &Int32Array::from(vec![1, 3])); ``` On `main` at 9ffd63d528 this holds for every type I tried, through `filter`, an optimized `FilterPredicate`, and `filter_record_batch`: primitives, booleans, byte and view arrays, fixed-size binary, lists, list views, fixed-size lists, structs, maps, dictionaries, run-end encoded arrays, and both union modes. [`filter_array`](https://github.com/apache/arrow-rs/blob/9ffd63d528e924d9c0faceef0027e3766b615c98/arrow-select/src/filter.rs#L597-L604) and [`BatchCoalescer::push_batch_with_filter`](https://github.com/apache/arrow-rs/blob/9ffd63d528e924d9c0faceef0027e3766b615c98/arrow-select/src/coalesce.rs#L760-L764) reject only a longer predicate. arrow-select's tests rely on the behavior: `fuzz_filter` [truncates the predicate](https://github.com/apache/arrow-rs/blob/9ffd63d528e924d9c0faceef0027e3766b615c98/arrow-select/src/filter.rs#L2253-L2261) by up to 9 rows, and [`test_filter_run_end_encoding_array_max_value_gt_predicate_len`](https://github.com/apache/arrow-rs/blob/9ffd63d528e924d9c 0faceef0027e3766b615c98/arrow-select/src/filter.rs#L1786-L1804), added in #6675, expects 2 rows from a 10-row run array and a 3-row predicate. As a contract it would allow a caller to evaluate a predicate over only a prefix of a batch, such as under a `LIMIT`, and pass it without padding it to the batch length. I found no caller that does this yet: DataFusion's `FilterExec` and Comet's rank limit build predicates over whole batches. Should the rustdoc of `filter`, `filter_record_batch` and `FilterPredicate::filter` state that a shorter predicate selects none of the rows past its end? Or should a shorter predicate be an error, as in Arrow C++, whose [`FilterRecordBatch`](https://github.com/apache/arrow/blob/373d64c02cb23663b63c5853099fca43a46c39fc/cpp/src/arrow/compute/kernels/vector_selection_filter_internal.cc#L929-L931) requires equal lengths? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
