JingsongLi commented on code in PR #10994:
URL: https://github.com/apache/paimon/pull/10994#discussion_r4478805079


##########
paimon-core/src/main/java/org/apache/paimon/table/ExpireSnapshotsImpl.java:
##########
@@ -538,6 +629,13 @@ private void writeEarliestHint(long earliest) {
         }
     }
 
+    private String progressContext() {

Review Comment:
   [P1] Redact storage credentials before writing progress logs
   
   `Path.toString()` includes URI user-info. Java's Jindo adapter explicitly 
handles legacy OSS paths with user-info before making filesystem calls, so 
those paths can reach maintenance with the original authority intact. I 
verified with a synthetic `oss://user:password@...` path that a no-op 
`ExpireSnapshotsImpl.expire()` emits both values in all four INFO records. 
Please sanitize table, directory and file URIs at every new logging call site 
using `SensitiveConfigUtils.sanitizeUri(path.toUri().toString())`, and assert 
redaction in a log-capture test.



##########
paimon-core/src/main/java/org/apache/paimon/table/ExpireSnapshotsImpl.java:
##########
@@ -112,13 +113,27 @@ public ExpireSnapshots config(ExpireConfig expireConfig) {
 
     @Override
     public int expire() {
+        try (MaintenanceProgress progress =
+                new MaintenanceProgress(progressContext(), 
"evaluate-retention")) {

Review Comment:
   [P2] Keep short automatic expiration checks off the INFO and thread-creation 
path
   
   Regular writer commits run snapshot expiration by default, synchronously 
unless configured otherwise. This starts a reporter even when retention selects 
an empty range. A normal no-op check emits four INFO records and the last 
user's close shuts the shared scheduler down. A smoke test of 100 sequential 
no-op `expire()` calls started 100 platform threads even with logging disabled. 
Please use a lifecycle-scoped or genuinely shared timer and keep fast/no-op 
checks quiet while preserving the heartbeat for slow retention evaluation.



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/orphan/FlinkOrphanFilesClean.java:
##########
@@ -418,11 +554,28 @@ public void processElement2(
                                         Tuple2<String, Long> fileInfo = 
element.getValue();
                                         String value = fileInfo.f0;
                                         Path path = new Path(value);
+                                        progress.add("files_checked", 1);
                                         if (!used.contains(path.getName())) {
                                             emittedFilesCount++;
                                             emittedFilesLen += fileInfo.f1;
-                                            cleanFile(path);
-                                            LOG.info("Dry clean: {}", path);
+                                            progress.add("candidate_files", 1);
+                                            progress.add("candidate_bytes", 
fileInfo.f1);
+                                            long item = 
progress.startItem(path.toString());
+                                            try {
+                                                cleanFile(path);
+                                            } finally {
+                                                progress.finishItem(item);
+                                            }
+                                            progress.add(
+                                                    dryRun
+                                                            ? "dry_run_files"
+                                                            : 
"delete_calls_completed",
+                                                    1);
+                                            LOG.debug(

Review Comment:
   [P2] Preserve the file list for explicit distributed dry runs
   
   The old INFO line was the per-file output of a distributed dry run. The 
Flink procedure returns only count and bytes, so moving this to DEBUG leaves 
users at normal log levels unable to see which paths would be removed. Please 
keep a redacted path list for explicit dry runs, either in their logs or 
through an equivalent audit output. [Flink cleanup 
procedure](https://github.com/apache/paimon/blob/f9629393b911634ab490a8cc955a17d7659e030e/docs/docs/flink/procedures/repair.md#remove_orphan_files)



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/orphan/FlinkOrphanFilesClean.java:
##########
@@ -283,31 +343,48 @@ public void processElement(
                                             ProcessFunction<String, 
Tuple2<String, Long>>.Context
                                                     ctx,
                                             Collector<Tuple2<String, Long>> 
out) {
-                                        Path dirPath = new Path(dir);
-                                        List<FileStatus> files = 
tryBestListingDirs(dirPath);
-                                        for (FileStatus file : files) {
-                                            if (!file.isDir()
-                                                    && 
!isManagedBlobPack(file.getPath())
-                                                    && oldEnough(file)) {
-                                                out.collect(
-                                                        Tuple2.of(
-                                                                
file.getPath().toString(),
-                                                                
file.getLen()));
-                                            }
-                                        }
-                                        if (files.isEmpty()) {
+                                        try (MaintenanceProgress progress =
+                                                new MaintenanceProgress(

Review Comment:
   [P2] Aggregate orphan-cleanup diagnostics per Flink subtask
   
   Each candidate directory gets its own START/END pair and scheduled task 
here. The manifest-list reader at line 205 does the same for each snapshot. On 
a subtask listing 100,000 directories, this adds 200,000 INFO records, even if 
every call finishes before the heartbeat interval. Serial calls can also create 
a fresh thread when they are the last reporter user. Please use a reporter 
scoped to the subtask's `open`/`close`, record active paths with 
`startItem`/`finishItem`, and aggregate counters across elements.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to