yordan-pavlov commented on a change in pull request #1248:
URL: https://github.com/apache/arrow-rs/pull/1248#discussion_r799334843
##########
File path: arrow/src/compute/kernels/filter.rs
##########
@@ -17,19 +17,55 @@
//! Defines miscellaneous array kernels.
+use std::ops::AddAssign;
+use std::sync::Arc;
+
+use num::Zero;
+
+use TimeUnit::*;
+
use crate::array::*;
-use crate::buffer::buffer_bin_and;
-use crate::datatypes::DataType;
-use crate::error::Result;
+use crate::buffer::{buffer_bin_and, Buffer, MutableBuffer};
+use crate::datatypes::*;
+use crate::error::{ArrowError, Result};
use crate::record_batch::RecordBatch;
use crate::util::bit_chunk_iterator::{UnalignedBitChunk,
UnalignedBitChunkIterator};
+use crate::util::bit_util;
-/// Function that can filter arbitrary arrays
-pub type Filter<'a> = Box<dyn Fn(&ArrayData) -> ArrayData + 'a>;
+/// If the filter selects more than this fraction of rows, use
+/// [`SlicesIterator`] to copy ranges of values. Otherwise iterate
+/// over individual rows using [`IndexIterator`]
+///
+/// Threshold of 0.8 chosen based on
<https://dl.acm.org/doi/abs/10.1145/3465998.3466009>
+///
+const FILTER_SLICES_SELECTIVITY_THRESHOLD: f64 = 0.8;
Review comment:
that's a great idea and I suspect that in order to get the best
performance in more cases, it will often be necessary to choose one of multiple
possible implementations at runtime in other places in the code as well; also
love the reference to the research paper
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
