dshogan-amzn commented on code in PR #18933:
URL: https://github.com/apache/iceberg/pull/18933#discussion_r4615141794
##########
core/src/main/java/org/apache/iceberg/ManifestReader.java:
##########
@@ -400,7 +400,7 @@ private Evaluator evaluator() {
private InclusiveMetricsEvaluator metricsEvaluator() {
if (lazyMetricsEvaluator == null) {
this.lazyMetricsEvaluator =
- new InclusiveMetricsEvaluator(spec.schema(), rowFilter,
caseSensitive);
+ InclusiveMetricsEvaluator.forRowFilter(rowFilter, spec,
caseSensitive);
Review Comment:
Thanks, good catch on where the index lives. A couple of data points:
- The per-reader binding predates this change: ManifestReader already builds
its own Evaluator and InclusiveMetricsEvaluator lazily for each manifest. What
this PR adds per reader is the literal index, one bucket hash and map insert
per IN literal, built only for manifests that survive manifest-level pruning.
With the project() call removed (see the PartitionedInLiterals thread), each
literal is now hashed once.
- The JMH run in the description opens a new ManifestReader per op, so it
rebuilds the index every iteration. In the 400-literal case where nothing can
be pruned (pure overhead), allocation rose from 13.34 to 13.76 MB/op and time
was within noise (18.04 ± 9.27 vs 18.12 ± 8.32 ms/op), compared with decoding
the 10,000-entry manifest.
Caching evaluators per spec in ManifestGroup would also remove the existing
per-reader binding, but it means passing a prebuilt evaluator into
ManifestReader (and DeleteFileIndex).
I think we can target it as a followup PR and as it's own optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]