szehon-ho commented on code in PR #17956:
URL: https://github.com/apache/iceberg/pull/17956#discussion_r3993687285
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java:
##########
@@ -181,6 +193,8 @@ protected synchronized List<T> tasks() {
if (tasks == null) {
try (CloseableIterable<? extends ScanTask> taskIterable =
scan.planFiles()) {
List<T> plannedTasks = Lists.newArrayList();
+ Map<Integer, PartitionPredicateEvaluator> evaluatorsBySpecId =
Maps.newHashMap();
+ int numPlannedTasks = 0;
Review Comment:
Done in 5c33e29522. Renamed it to to distinguish the pre-pruning count from
the tasks that survive pruning.
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/BaseSparkScanBuilder.java:
##########
@@ -178,15 +192,15 @@ public Predicate[] pushPredicates(Predicate[] predicates)
{
}
}
- this.filters = expressions;
- this.pushedPredicates = pushablePredicates.toArray(new Predicate[0]);
+ this.filters.addAll(expressions);
Review Comment:
Done in 5c33e29522. The builder is now created inside the loop, so each
iteration plans only the filter under test.
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java:
##########
@@ -229,7 +258,8 @@ protected synchronized List<ScanTaskGroup<T>> taskGroups() {
StructLikeSet plannedGroupingKeys =
collectGroupingKeys(plannedTaskGroups);
LOG.debug(
- "Planned {} task group(s) with {} grouping key type and {} unique
grouping key(s) for table {}",
+ "Planned {} task group(s) with {} grouping key type and {} unique
grouping key(s) for"
+ + " table {}",
Review Comment:
Done in 5c33e29522. Reverted this change and the other unrelated
string-literal reformatting in and .
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java:
##########
@@ -229,7 +258,8 @@ protected synchronized List<ScanTaskGroup<T>> taskGroups() {
StructLikeSet plannedGroupingKeys =
collectGroupingKeys(plannedTaskGroups);
LOG.debug(
- "Planned {} task group(s) with {} grouping key type and {} unique
grouping key(s) for table {}",
+ "Planned {} task group(s) with {} grouping key type and {} unique
grouping key(s) for"
+ + " table {}",
Review Comment:
Done in 5c33e29522. Reverted this change and the other unrelated
string-literal reformatting in SparkScanBuilder and TestFilterPushDown.
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java:
##########
@@ -181,6 +193,8 @@ protected synchronized List<T> tasks() {
if (tasks == null) {
try (CloseableIterable<? extends ScanTask> taskIterable =
scan.planFiles()) {
List<T> plannedTasks = Lists.newArrayList();
+ Map<Integer, PartitionPredicateEvaluator> evaluatorsBySpecId =
Maps.newHashMap();
+ int numPlannedTasks = 0;
Review Comment:
Done in 5c33e29522. Renamed it to numCandidateTasks to distinguish the
pre-pruning count from the tasks that survive pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
