huaxingao commented on code in PR #17956:
URL: https://github.com/apache/iceberg/pull/17956#discussion_r4011170816
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java:
##########
@@ -264,4 +293,98 @@ protected String groupingKeyDesc() {
.map(NestedField::name)
.collect(Collectors.joining(", "));
}
+
+ protected List<PartitionPredicate> partitionPredicates() {
+ return partitionPredicates;
+ }
+
+ private boolean matchesPartitionPredicates(
+ T task, Map<Integer, PartitionPredicateEvaluator> evaluatorsBySpecId) {
+ PartitionPredicateEvaluator evaluator =
+ evaluatorsBySpecId.computeIfAbsent(
+ task.spec().specId(), ignored -> new
PartitionPredicateEvaluator(task.spec()));
+ return evaluator.eval(task.partition());
+ }
+
+ private static List<PartitionField> activePartitionFields(PartitionSpec
spec) {
+ return spec.fields().stream()
+ .filter(field -> !field.transform().isVoid())
+ .collect(Collectors.toList());
Review Comment:
nit: this part is not covered by the new tests. Could you add a case that
drops a partition field and then pushes a predicate on one of the remaining
ones? For example, partition by p0, p1, p2 and then drop p1 — that moves p2
from ordinal 2 to ordinal 1 in the current spec, while the earlier files still
carry three partition values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
