anjalitripathibiz-byte opened a new issue, #3716:
URL: https://github.com/apache/iceberg-rust/issues/3716
### Apache Iceberg Rust version
0.10.1 (the code is unchanged on main at 14e0919489)
### Describe the bug
A row filter that contains an `IN` list whose literal `get_arrow_datum`
cannot convert panics the reader thread, instead of returning an error. The
comparison visitors (`less_than`, `eq`, ...) propagate that error with `?`.
`r#in` and `not_in` call `.unwrap()` instead:
```rust
// crates/iceberg/src/arrow/reader/predicate_visitor.rs, fn r#in (and fn
not_in)
let literals: Vec<_> = literals
.iter()
.map(|lit| get_arrow_datum(lit).unwrap())
.collect();
```
I hit it from SQL through iceberg-datafusion 0.10.1, with a user-supplied
literal outside the column's 32-bit range:
```sql
-- c1 is an Iceberg `int` column
SELECT c2 FROM t2 WHERE c1 IN (3, -9223372036854775808);
```
```
thread '...' panicked at
.../iceberg-0.10.1/src/arrow/reader/predicate_visitor.rs:586:
called `Result::unwrap()` on an `Err` value: FeatureUnsupported =>
Converting datum from type Int to arrow not supported yet
```
The panic was observed on 0.10.1. On main I only checked that the two
`.unwrap()` calls are still there (lines 590 and 620); I did not re-run the
query against main.
### To Reproduce
An `int` column and a scan whose row filter is
`Reference::new("c1").is_in([Datum::int(3), <a datum that binds to type Int
with a Long literal>])`. Through DataFusion, the SQL above against an
`IcebergStaticTableProvider` gets there.
### Expected behavior
Either:
1. return the `FeatureUnsupported` error instead of panicking (PR #3448
changes these two lines to `?` as part of the time-literal fix for #3446), or,
better,
2. not produce an out-of-range literal for an `int` column at all. A literal
outside `i32` can never equal an `int` value, so for `IN` it can be dropped
from the set, and for `NOT IN` it is always true. That is how
`AboveMax`/`BelowMin` are already handled for comparisons.
With (1) alone, the query above fails rather than returning the rows where
`c1 = 3`.
Found by a differential SQL fuzzer comparing DataFusion over Iceberg with
DuckDB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]