mattfaltyn opened a new issue, #1870:
URL: https://github.com/apache/iceberg-go/issues/1870
## Apache Iceberg version
`main` at `0f08e66e59cc1ec9b40898823d9eb991409bd080` (also present in v0.6.0)
## Please describe the bug 🐞
Binding an `IN` or `NOT IN` predicate to a narrower numeric field retains
`AboveMaxLiteral` and `BelowMinLiteral` sentinels in the bound set. Metrics
evaluators assume every set member is a regular `TypedLiteral[T]`, so a valid
scan can fail during planning when its filter mixes representable and
out-of-range values.
For example, an application may build an `int64` filter for an Iceberg
`int32` column:
```go
iceberg.IsIn(iceberg.Reference("id"), int64(35), int64(math.MaxInt32)+1)
```
The second value can never equal an `int32` column value and should be
discarded during binding. Instead, it becomes `aboveMaxLiteral[int32]`.
Inclusive metrics evaluation reaches `removeBoundCmp`, asserts that every
member implements `TypedLiteral[int32]`, and returns this error:
```text
interface conversion: iceberg.aboveMaxLiteral[int32] is not
iceberg.TypedLiteral[int32]: missing method Comparator
```
This affects the normal scan-planning path because
`Scan.collectManifestEntriesWithSchema` creates the inclusive metrics evaluator
for the row filter.
### Expected behavior
- Impossible set members are removed when binding to the field type.
- The mixed predicate above behaves like `id IN (35)`.
- A set containing only impossible members simplifies to `AlwaysFalse` for
`IN` and `AlwaysTrue` for `NOT IN`.
- Metrics evaluation completes without an error.
### Reproduction
The following regression test can be added to `table/evaluators_test.go`:
```go
func TestInclusiveMetricsMixedRangeIn(t *testing.T) {
lower, err := iceberg.Int32Literal(30).MarshalBinary()
require.NoError(t, err)
upper, err := iceberg.Int32Literal(79).MarshalBinary()
require.NoError(t, err)
schema := iceberg.NewSchema(0,
iceberg.NestedField{ID: 1, Name: "id", Type:
iceberg.PrimitiveTypes.Int32},
)
file := &mockDataFile{
count: 1,
valueCounts: map[int]int64{1: 1},
nullCounts: map[int]int64{1: 0},
lowerBounds: map[int][]byte{1: lower},
upperBounds: map[int][]byte{1: upper},
}
filter := iceberg.IsIn(
iceberg.Reference("id"),
int64(35),
int64(math.MaxInt32)+1,
)
eval, err := newInclusiveMetricsEvaluator(schema, filter, true, true)
require.NoError(t, err)
shouldRead, err := eval(file)
require.NoError(t, err)
assert.True(t, shouldRead)
}
```
On current `main`, `eval(file)` returns the interface-conversion error
above. An in-range control using `int64(35), int64(36)` succeeds.
### Root cause
`createBoundSetPredicate` calls `Literal.To(boundType)` and unconditionally
inserts the result. Scalar literal binding already handles `AboveMaxLiteral`
and `BelowMinLiteral` specially, but set binding does not.
Dropping those impossible members before the existing zero/one-member
simplification should preserve `IN` and `NOT IN` semantics while preventing
non-comparable sentinel values from reaching expression, manifest, metrics, and
bloom-filter consumers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
