alexanderbianchi commented on code in PR #3739:
URL: https://github.com/apache/iceberg-rust/pull/3739#discussion_r4648124324


##########
crates/iceberg/src/arrow/reader/predicate_visitor.rs:
##########
@@ -236,6 +237,58 @@ impl PredicateConverter<'_> {
         }
     }
 
+    fn set_predicate(
+        &mut self,
+        reference: &BoundReference,
+        predicate: &BoundPredicate,
+        negated: bool,
+    ) -> Result<Box<PredicateResult>> {
+        let Some(idx) = self.bound_reference(reference)? else {
+            return Ok(Box::new(move |batch| {
+                Ok(constant_bool_array(negated, batch.num_rows()))
+            }));
+        };
+        let BoundPredicate::Set(set) = predicate else {
+            unreachable!("set visitor receives a set predicate")
+        };
+        let index = set.membership_index();
+        let set = set.clone();
+        let arrow_literals = move || {
+            set.literals()
+                .iter()
+                .map(get_arrow_datum)
+                .collect::<Result<Vec<_>>>()
+        };
+        let scalars = if index.is_some() {
+            OnceCell::new()
+        } else {
+            OnceCell::with_value(arrow_literals()?)
+        };
+        Ok(Box::new(move |batch| {
+            let column = project_column(&batch, idx)?;
+            if let Some(result) = index
+                .as_ref()
+                .and_then(|index| probe_set(index, &column, negated))
+            {
+                return Ok(result);
+            }
+            // Schema evolution can give a file a different physical type from 
the bound literals.
+            let scalars = scalars
+                .get_or_try_init(&arrow_literals)
+                .map_err(|err| ArrowError::ExternalError(Box::new(err)))?;
+            let mut acc = constant_bool_array(negated, batch.num_rows());

Review Comment:
   This also happens on main. - I tried a file containing 0, 32, and null, with 
the table column promoted to Int64. For a list containing 0 through 31 plus 
2147483648, IN should return 0 and NOT IN should return 32. Both returned no 
rows. The large value becomes null when we cast it down. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to