dshogan-amzn opened a new pull request, #18933:
URL: https://github.com/apache/iceberg/pull/18933

   This PR improves file pruning for IN predicates on columns that use 
non-order-preserving partition transforms such as bucket.
   
   Partition projection already limits a manifest scan to matching bucket 
values. File metrics evaluation, however, still evaluates every literal in the 
original IN predicate against every selected file. A literal assigned to 
another bucket can therefore retain a file that cannot contain that literal. 
Lists above the metrics evaluator's 200-value limit also skip bounds evaluation 
before literals are narrowed to the file's partition.
   
   The change binds the row filter once and pre-indexes eligible IN literals by 
transformed partition value. For each file, metrics evaluation uses only 
literals compatible with that file's partition. Existing schema-only 
InclusiveMetricsEvaluator constructors retain their current behavior. 
ManifestReader opts into the partition-aware path through 
InclusiveMetricsEvaluator.forRowFilter.
   Validation
   
       ./gradlew --no-daemon :iceberg-api:test --tests 
org.apache.iceberg.expressions.TestInclusiveMetricsEvaluator
       ./gradlew --no-daemon :iceberg-core:test --tests 
org.apache.iceberg.TestManifestReader
       ./gradlew --no-daemon :iceberg-api:spotlessJavaCheck 
:iceberg-core:spotlessJavaCheck
       git diff --check
   
   TestManifestReader covers format versions 1 through 4. It verifies ordinary 
and 250-value IN predicates, rejects an impossible cross-bucket file, and 
retains a file that can match.
   
   A synthetic JMH benchmark uses 1,000 buckets, 10,000 files, and broad or 
narrow file bounds. Deterministic file counts and one-fork timing results are 
below.
   Keys Layout  Baseline files  Patched files   Baseline ms/op  Patched ms/op
   50   narrow  487     50      15.94   16.41
   200  narrow  1,815   199     18.61   17.05
   400  narrow  3,250   397     17.32   17.32
   400  broad   3,250   3,250   17.07   17.91
   
   The timing confidence intervals overlap in every case. In the 400-key narrow 
case, allocation fell from 13.35 MB/op to 11.21 MB/op. The broad case measures 
the indexing cost when file counts cannot improve.
   AI use
   
   I used an AI coding assistant during investigation and implementation. I 
reviewed the code, tests, and benchmark results before submission.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to