yordan-pavlov commented on a change in pull request #1248:
URL: https://github.com/apache/arrow-rs/pull/1248#discussion_r799350951
##########
File path: arrow/src/compute/kernels/filter.rs
##########
@@ -17,19 +17,55 @@
//! Defines miscellaneous array kernels.
+use std::ops::AddAssign;
+use std::sync::Arc;
+
+use num::Zero;
+
+use TimeUnit::*;
+
use crate::array::*;
-use crate::buffer::buffer_bin_and;
-use crate::datatypes::DataType;
-use crate::error::Result;
+use crate::buffer::{buffer_bin_and, Buffer, MutableBuffer};
+use crate::datatypes::*;
+use crate::error::{ArrowError, Result};
use crate::record_batch::RecordBatch;
use crate::util::bit_chunk_iterator::{UnalignedBitChunk,
UnalignedBitChunkIterator};
+use crate::util::bit_util;
-/// Function that can filter arbitrary arrays
-pub type Filter<'a> = Box<dyn Fn(&ArrayData) -> ArrayData + 'a>;
+/// If the filter selects more than this fraction of rows, use
+/// [`SlicesIterator`] to copy ranges of values. Otherwise iterate
+/// over individual rows using [`IndexIterator`]
+///
+/// Threshold of 0.8 chosen based on
<https://dl.acm.org/doi/abs/10.1145/3465998.3466009>
+///
+const FILTER_SLICES_SELECTIVITY_THRESHOLD: f64 = 0.8;
Review comment:
out of curiosity though, have you measured what's the impact on
performance (of choosing slices or index iterator at runtime)? I suspect that
the benchmarks that filter out half of the data will be most affected /
improved? possibly very selective filters (that filter out most of the data)
will see improvement as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
