JingsongLi commented on code in PR #10994:
URL: https://github.com/apache/paimon/pull/10994#discussion_r4625010180
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/orphan/FlinkOrphanFilesClean.java:
##########
@@ -154,23 +158,88 @@ public void processElement(
env.fromCollection(branches)
.name("branch-source")
.process(
- new ProcessFunction<String, Tuple2<String,
String>>() {
+ new OpenableProcessFunction<String,
Tuple2<String, String>>() {
+
+ private transient MaintenanceProgress
progress;
+
+ @Override
+ protected void openFunction() {
+ progress =
+ new MaintenanceProgress(
+
subtaskProgressContext(),
+
"enumerate-snapshots-tags-changelogs");
+ }
+
+ @Override
+ public void close() throws Exception {
+ try {
+ if (progress != null) {
+ progress.complete();
Review Comment:
**[P2] Report failed operator attempts as aborted**
Flink still invokes the user function's `close()` during failed-task
cleanup. In a real Flink 1.20.1 batch job, I injected a transient IOException
when enumerating snapshots for a local table. The job failed, but the same
collect-snapshots subtask and attempt logged `event=END` with no completed
branches and never logged `ABORTED`. The other collection stages call
`complete()` from close the same way. Mark success on bounded
end-of-input/finish and let close release the reporter for either outcome, as
the join stage already does.
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileDeletionBase.java:
##########
@@ -161,7 +162,10 @@ public void cleanEmptyDirectories() {
for (Integer bucket : entry.getValue()) {
toDeleteEmptyDirectory.add(pathFactory.bucketPath(entry.getKey(), bucket));
}
- executeAll(toDeleteEmptyDirectory, this::tryDeleteEmptyDirectory);
+ executeAll(
Review Comment:
**[P2] Aggregate progress for short per-partition directory cleanup**
In a real partitioned append table with directory cleanup enabled, expiring
48 overwritten partitions emitted 96 INFO START/END records for empty-directory
deletion in about 70 ms. Each partition starts a new reporter, so production
snapshot expiration can produce thousands of eager log records even when no
heartbeat is needed. Report one cleanup phase on the parent reporter or use the
existing delayed reporting for these short nested calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]