zhuxiangyi opened a new pull request, #10015:
URL: https://github.com/apache/paimon/pull/10015
### Purpose
Bug fix: any ORC read that carries an `IN ()` or `NOT IN ()` predicate fails
while the reader is being created.
`PredicateBuilder.in(idx, emptyList())` is a legitimate always-false leaf
(the builder already special-cases an empty list, exactly as it does for more
than 20 literals), and `notIn` inherits it through `negate()`. But
`OrcPredicateFunctionVisitor.visitIn` turns that empty list into an empty
`Object[]` and hands it to Hive's `SearchArgument.Builder.in(...)`, which
rejects a zero-length call outright:
```
java.lang.IllegalArgumentException: Can't create in expression with no
arguments
at
org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$BuilderImpl.in(SearchArgumentImpl.java:419)
at
org.apache.paimon.format.orc.filter.OrcFilters$In.add(OrcFilters.java:328)
at
org.apache.paimon.format.orc.OrcReaderFactory.createRecordReader(OrcReaderFactory.java:466)
```
So a predicate that should simply match nothing (or everything, for `NOT IN
()`) instead takes the whole read down. Nothing nested or `Transform`-based is
needed to hit it — a plain index-based `in`/`notIn` on an ORC table is enough.
The fix has `visitIn` decline the pushdown (`Optional.empty()`) on an empty
literal list, the same way the visitor already declines every other predicate
it cannot express in a SARG. `visitNotIn` delegates to `visitIn`, so it is
covered by the same branch. Correctness is unaffected: engines re-evaluate data
filters after the scan (Spark keeps every data filter as a post-scan predicate;
Paimon's own row-level filtering is opt-in via `TableRead.executeFilter()`), so
the always-false / always-true semantics still hold — only the stripe-level
pruning for that one predicate is given up.
This is the ORC counterpart of the `ParquetFilters.visitIn`/`visitNotIn`
guard added in #9423 (parquet-mr's `SetColumnFilterPredicate` refuses an empty
set the same way). It is independent of that PR and predates it.
### Tests
- `OrcFilterConverterTest.testInPredicateWithEmptyValuesIsNotPushedDown` —
`in(idx, [])` and `notIn(idx, [])` both resolve to `Optional.empty()`; also
pins that an empty `In` leaf inside `or(...)`/`and(...)` declines the whole
compound rather than crashing (the builder only folds `AlwaysFalse.INSTANCE`,
so the leaf does survive into a compound).
- `OrcFormatReadWriteTest.testEmptyInAndNotInPredicatesDoNotCrashTheReader`
— writes a real ORC file and opens a reader with each predicate through
`OrcFileFormat.createReaderFactory`; reproduces the `IllegalArgumentException`
before the fix, returns every row unfiltered after it.
### API and Format
No changes.
### Documentation
No changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
