dshogan-amzn opened a new pull request, #18933:
URL: https://github.com/apache/iceberg/pull/18933
This PR improves file pruning for IN predicates on columns that use
non-order-preserving partition transforms such as bucket.
Partition projection already limits a manifest scan to matching bucket
values. File metrics evaluation, however, still evaluates every literal in the
original IN predicate against every selected file. A literal assigned to
another bucket can therefore retain a file that cannot contain that literal.
Lists above the metrics evaluator's 200-value limit also skip bounds evaluation
before literals are narrowed to the file's partition.
The change binds the row filter once and pre-indexes eligible IN literals by
transformed partition value. For each file, metrics evaluation uses only
literals compatible with that file's partition. Existing schema-only
InclusiveMetricsEvaluator constructors retain their current behavior.
ManifestReader opts into the partition-aware path through
InclusiveMetricsEvaluator.forRowFilter.
Validation
./gradlew --no-daemon :iceberg-api:test --tests
org.apache.iceberg.expressions.TestInclusiveMetricsEvaluator
./gradlew --no-daemon :iceberg-core:test --tests
org.apache.iceberg.TestManifestReader
./gradlew --no-daemon :iceberg-api:spotlessJavaCheck
:iceberg-core:spotlessJavaCheck
git diff --check
TestManifestReader covers format versions 1 through 4. It verifies ordinary
and 250-value IN predicates, rejects an impossible cross-bucket file, and
retains a file that can match.
A synthetic JMH benchmark uses 1,000 buckets, 10,000 files, and broad or
narrow file bounds. Deterministic file counts and one-fork timing results are
below.
Keys Layout Baseline files Patched files Baseline ms/op Patched ms/op
50 narrow 487 50 15.94 16.41
200 narrow 1,815 199 18.61 17.05
400 narrow 3,250 397 17.32 17.32
400 broad 3,250 3,250 17.07 17.91
The timing confidence intervals overlap in every case. In the 400-key narrow
case, allocation fell from 13.35 MB/op to 11.21 MB/op. The broad case measures
the indexing cost when file counts cannot improve.
AI use
I used an AI coding assistant during investigation and implementation. I
reviewed the code, tests, and benchmark results before submission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]