acasabar-amzn commented on code in PR #18933:
URL: https://github.com/apache/iceberg/pull/18933#discussion_r4604446913


##########
core/src/main/java/org/apache/iceberg/ManifestReader.java:
##########
@@ -400,7 +400,7 @@ private Evaluator evaluator() {
   private InclusiveMetricsEvaluator metricsEvaluator() {
     if (lazyMetricsEvaluator == null) {
       this.lazyMetricsEvaluator =
-          new InclusiveMetricsEvaluator(spec.schema(), rowFilter, 
caseSensitive);
+          InclusiveMetricsEvaluator.forRowFilter(rowFilter, spec, 
caseSensitive);

Review Comment:
   Potential repeated index construction across manifests? 
   
   Since lazyMetricsEvaluator is scoped to each ManifestReader, every reader 
constructs its own InclusiveMetricsEvaluator, which in turn calls 
PartitionedInLiterals.build(expr, spec).
   
   For scans involving many manifests with the same partition spec and row 
filter, this repeatedly binds the same expression and rebuilds the same 
bucket-to-literals index.
   
   Could we consider reusing the evaluator at the scan-planning level, keyed by 
partition spec, similar to the evaluator caching already used in ManifestGroup?
   
   This would reduce index construction from once per manifest to once per 
partition spec per scan, without changing file-pruning behavior.
   
   The impact may be negligible for smaller scans, but it would be worth 
benchmarking with a large number of manifests and a 200–400+ value IN predicate.



##########
api/src/main/java/org/apache/iceberg/expressions/PartitionedInLiterals.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.expressions;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import org.apache.iceberg.PartitionField;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.expressions.ExpressionVisitors.ExpressionVisitor;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.transforms.Transform;
+
+/**
+ * Groups the literals of each bound {@code IN} predicate by the partition 
value they map to.
+ *
+ * <p>For a predicate on a column with a partition field whose transform does 
not preserve order,
+ * such as {@code bucket}, a file can only contain literals that map to the 
file's partition value.
+ * {@link #literalsFor(Set, StructLike)} returns that subset so an evaluator 
compares a file's
+ * metrics against only those literals. Order-preserving transforms, such as 
{@code identity} and
+ * {@code truncate}, are not indexed because their metrics already bound the 
matching literals.
+ *
+ * <p>Instances are built once per bound expression and partition spec, and 
queried once per file.
+ */
+class PartitionedInLiterals {
+  private static final PartitionedInLiterals EMPTY =
+      new PartitionedInLiterals(Collections.emptyMap());
+
+  // Keyed by identity: each bound IN predicate holds one literal set 
instance, which the visitor
+  // passes back to literals(), and two predicates with equal sets may 
reference different columns.
+  private final Map<Set<?>, Index> indexes;
+
+  private PartitionedInLiterals(Map<Set<?>, Index> indexes) {
+    this.indexes = indexes;
+  }
+
+  /**
+   * Index the {@code IN} predicates of a bound expression against a partition 
spec.
+   *
+   * @param boundExpr a bound expression
+   * @param spec the partition spec of the files that will be evaluated
+   * @return an index of IN literals by partition value
+   */
+  @SuppressWarnings("IdentityHashMapUsage")
+  static PartitionedInLiterals build(Expression boundExpr, PartitionSpec spec) 
{
+    if (spec.isUnpartitioned()) {
+      return EMPTY;
+    }
+
+    Map<Set<?>, Index> indexes = new IdentityHashMap<>();
+    ExpressionVisitors.visit(
+        boundExpr,
+        new ExpressionVisitor<Void>() {
+          @Override
+          public <T> Void predicate(BoundPredicate<T> predicate) {
+            Index index = index(predicate, spec);
+            if (index != null) {
+              indexes.put(predicate.asSetPredicate().literalSet(), index);
+            }
+            return null;
+          }
+        });
+
+    return indexes.isEmpty() ? EMPTY : new PartitionedInLiterals(indexes);
+  }
+
+  /**
+   * Return the literals of an {@code IN} predicate that may match a file with 
the given partition.
+   *
+   * @param literalSet the literal set of a bound IN predicate in the indexed 
expression
+   * @param partition the partition of the file being evaluated
+   * @return the literals that map to the file's partition value, or the full 
set if the predicate
+   *     is not indexed
+   */
+  <T> Collection<T> literalsFor(Set<T> literalSet, StructLike partition) {
+    Index index = indexes.get(literalSet);
+    return index != null ? index.literals(partition) : literalSet;
+  }
+
+  @SuppressWarnings("unchecked")
+  private static <T> Index index(BoundPredicate<T> predicate, PartitionSpec 
spec) {
+    if (!predicate.isSetPredicate()
+        || predicate.op() != Expression.Operation.IN
+        || !(predicate.term() instanceof BoundReference)) {
+      return null;
+    }
+
+    List<PartitionField> fields = 
spec.getFieldsBySourceId(predicate.ref().fieldId());
+    if (fields == null) {
+      return null;
+    }
+
+    for (PartitionField field : fields) {
+      Transform<T, ?> transform = (Transform<T, ?>) field.transform();
+      if (transform.preservesOrder() || transform.project(field.name(), 
predicate) == null) {

Review Comment:
   For bucketed IN predicates, transform.project() is more expensive than a 
capability check. Bucket.project() calls ProjectionUtil.transformSet(), which 
applies the bucket transform to every literal and constructs an intermediate 
projected predicate.
   
   Since we're only checking whether the result is non-null, that projected 
predicate is discarded. We then bind the same transform and apply it to every 
literal again when building literalsByValue.
   
   This effectively hashes the same IN literals twice per index construction, 
along with the additional intermediate allocations.
   
   Could we avoid the full projection here through a lightweight eligibility 
check, or otherwise reuse the transformed values?
   
   This also compounds the per-manifest index reconstruction cost discussed 
earlier. It would be interesting to benchmark the impact with larger IN lists, 
especially for string keys.
   
   Not a correctness concern, but potentially an easy opportunity to reduce 
planning overhead.



##########
api/src/test/java/org/apache/iceberg/expressions/TestInclusiveMetricsEvaluatorWithPartitionSpec.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.expressions;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.types.Conversions.toByteBuffer;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.TestHelpers.Row;
+import org.apache.iceberg.TestHelpers.TestDataFile;
+import org.apache.iceberg.exceptions.ValidationException;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.transforms.Transforms;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.IntegerType;
+import org.junit.jupiter.api.Test;
+
+public class TestInclusiveMetricsEvaluatorWithPartitionSpec {
+  private static final Schema SCHEMA =
+      new Schema(required(1, "id", IntegerType.get()), optional(2, "data", 
Types.StringType.get()));
+  private static final int BUCKETS = 16;
+  private static final Function<Integer, Integer> BUCKET =
+      Transforms.<Integer>bucket(BUCKETS).bind(IntegerType.get());
+
+  /**
+   * Id bounds drawn from one bucket, a value from that bucket outside the 
bounds, and a value from
+   * another bucket inside the bounds.
+   */
+  private record BucketCase(int bucket, int lower, int upper, int ownValue, 
int foreignValue) {}
+
+  private static BucketCase bucketCase() {
+    int bucket = BUCKET.apply(100);
+    List<Integer> values = Lists.newArrayList();
+    for (int value = 0; values.size() < 7; value += 1) {
+      if (BUCKET.apply(value) == bucket) {
+        values.add(value);
+      }
+    }
+
+    int lower = values.get(0);
+    int upper = values.get(5);
+    int foreignValue = lower + 1;
+    while (BUCKET.apply(foreignValue) == bucket) {
+      foreignValue += 1;
+    }
+    assertThat(foreignValue).isLessThan(upper);
+
+    return new BucketCase(bucket, lower, upper, values.get(6), foreignValue);
+  }
+
+  private static DataFile fileWithIdBounds(StructLike partition, int lower, 
int upper) {
+    return new TestDataFile(
+        "file.parquet",
+        partition,
+        10,
+        ImmutableMap.of(1, 10L),
+        ImmutableMap.of(1, 0L),
+        null,
+        ImmutableMap.of(1, toByteBuffer(IntegerType.get(), lower)),
+        ImmutableMap.of(1, toByteBuffer(IntegerType.get(), upper)));
+  }
+
+  private static boolean schemaOnly(Expression expr, DataFile file) {
+    return new InclusiveMetricsEvaluator(SCHEMA, expr).eval(file);
+  }
+
+  private static boolean partitionAware(PartitionSpec spec, Expression expr, 
DataFile file) {
+    return InclusiveMetricsEvaluator.forRowFilter(expr, spec, true).eval(file);
+  }
+
+  @Test
+  public void testBucketInUsesLiteralsForFilePartition() {

Review Comment:
   The new tests cover the bucket-narrowing behavior well, and 
`TestManifestReader` already exercises a 250-value `IN` predicate.
   
   Could we also add a direct evaluator test around the exact 200-value 
boundary? In particular:
   
   - 201 original literals narrowed to 200 should still evaluate file bounds 
and allow pruning.
   - 202 original literals narrowed to 201 should retain the existing 
conservative behavior.
   - A compatible literal within the bounds should continue to retain the file.
   
   It would also be useful to test at least one non-integer bucket source 
(e.g., `STRING`), since the indexing logic uses generic `Transform.bind()` and 
currently the new API tests only exercise integer bucket keys.
   
   These would help guard against regressions in the two key aspects of this 
optimization: candidate-set narrowing before the limit check, and 
type-independent bucket mapping.



##########
api/src/main/java/org/apache/iceberg/expressions/InclusiveEvalVisitor.java:
##########
@@ -301,7 +311,7 @@ public <T> Boolean in(Bound<T> term, Set<T> literalSet) {
       return ROWS_CANNOT_MATCH;
     }
 
-    Collection<T> literals = literalSet;
+    Collection<T> literals = literals(literalSet);

Review Comment:
   Short-circuit empty candidate sets before checking bounds.
   
   `PartitionedInLiterals.literalsFor()` can return an empty collection when no 
`IN` literals map to the file's partition value.
   
   Currently, `in()` continues to evaluate the lower bound and returns 
`ROWS_MIGHT_MATCH` if it is missing or NaN, even though an empty candidate set 
already proves the predicate cannot match.
   
   Could we add an early return here?
   
   ```java
   if (literals.isEmpty()) {
     return ROWS_CANNOT_MATCH;
   }
   ```
   
   This would allow the partition-aware metrics evaluator to reject such files 
independently of column statistics.
   
   I recognize that `ManifestReader` normally applies partition pruning first, 
so the practical impact may be limited. Still, this seems like a small, 
correctness-preserving improvement. It may also be worth extending the 
null-bucket test to cover missing bounds.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to