szehon-ho commented on code in PR #7565:
URL: https://github.com/apache/iceberg/pull/7565#discussion_r1190391156
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java:
##########
@@ -133,55 +133,69 @@ public RewritePositionDeleteFiles.Result execute() {
}
}
- private Map<StructLike, List<List<PositionDeletesScanTask>>>
planFileGroups() {
+ private StructLikeMap<List<List<PositionDeletesScanTask>>> planFileGroups() {
Table deletesTable =
MetadataTableUtils.createMetadataTableInstance(table,
MetadataTableType.POSITION_DELETES);
- CloseableIterable<PositionDeletesScanTask> scanTasks =
+
+ CloseableIterable<PositionDeletesScanTask> tasks =
CloseableIterable.transform(
deletesTable.newBatchScan().ignoreResiduals().planFiles(),
t -> (PositionDeletesScanTask) t);
try {
StructType partitionType = Partitioning.partitionType(table);
- StructLikeMap<List<PositionDeletesScanTask>> filesByPartition =
- StructLikeMap.create(partitionType);
-
- for (PositionDeletesScanTask task : scanTasks) {
- StructLike coerced = coercePartition(task, partitionType);
-
- List<PositionDeletesScanTask> partitionTasks =
filesByPartition.get(coerced);
- if (partitionTasks == null) {
- partitionTasks = Lists.newArrayList();
- }
- partitionTasks.add(task);
- filesByPartition.put(coerced, partitionTasks);
- }
-
- StructLikeMap<List<List<PositionDeletesScanTask>>> fileGroupsByPartition
=
- StructLikeMap.create(partitionType);
- filesByPartition.forEach(
- (partition, partitionTasks) -> {
- Iterable<List<PositionDeletesScanTask>> plannedFileGroups =
- rewriter.planFileGroups(partitionTasks);
- List<List<PositionDeletesScanTask>> groups =
ImmutableList.copyOf(plannedFileGroups);
- if (groups.size() > 0) {
- fileGroupsByPartition.put(partition, groups);
- }
- });
+ StructLikeMap<List<PositionDeletesScanTask>> filesPerPartition =
Review Comment:
Honestly its a bit confusing. The API's refer to files(), FileGroups, etc
but the class is FileTask. At some point we will need to change from task to
file terminology. I end up calling them fileTasks, not sure its the best way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
