JingsongLi commented on code in PR #10994:
URL: https://github.com/apache/paimon/pull/10994#discussion_r4478805079
##########
paimon-core/src/main/java/org/apache/paimon/table/ExpireSnapshotsImpl.java:
##########
@@ -538,6 +629,13 @@ private void writeEarliestHint(long earliest) {
}
}
+ private String progressContext() {
Review Comment:
[P1] Redact storage credentials before writing progress logs
`Path.toString()` includes URI user-info. Java's Jindo adapter explicitly
handles legacy OSS paths with user-info before making filesystem calls, so
those paths can reach maintenance with the original authority intact. I
verified with a synthetic `oss://user:password@...` path that a no-op
`ExpireSnapshotsImpl.expire()` emits both values in all four INFO records.
Please sanitize table, directory and file URIs at every new logging call site
using `SensitiveConfigUtils.sanitizeUri(path.toUri().toString())`, and assert
redaction in a log-capture test.
##########
paimon-core/src/main/java/org/apache/paimon/table/ExpireSnapshotsImpl.java:
##########
@@ -112,13 +113,27 @@ public ExpireSnapshots config(ExpireConfig expireConfig) {
@Override
public int expire() {
+ try (MaintenanceProgress progress =
+ new MaintenanceProgress(progressContext(),
"evaluate-retention")) {
Review Comment:
[P2] Keep short automatic expiration checks off the INFO and thread-creation
path
Regular writer commits run snapshot expiration by default, synchronously
unless configured otherwise. This starts a reporter even when retention selects
an empty range. A normal no-op check emits four INFO records and the last
user's close shuts the shared scheduler down. A smoke test of 100 sequential
no-op `expire()` calls started 100 platform threads even with logging disabled.
Please use a lifecycle-scoped or genuinely shared timer and keep fast/no-op
checks quiet while preserving the heartbeat for slow retention evaluation.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/orphan/FlinkOrphanFilesClean.java:
##########
@@ -418,11 +554,28 @@ public void processElement2(
Tuple2<String, Long> fileInfo =
element.getValue();
String value = fileInfo.f0;
Path path = new Path(value);
+ progress.add("files_checked", 1);
if (!used.contains(path.getName())) {
emittedFilesCount++;
emittedFilesLen += fileInfo.f1;
- cleanFile(path);
- LOG.info("Dry clean: {}", path);
+ progress.add("candidate_files", 1);
+ progress.add("candidate_bytes",
fileInfo.f1);
+ long item =
progress.startItem(path.toString());
+ try {
+ cleanFile(path);
+ } finally {
+ progress.finishItem(item);
+ }
+ progress.add(
+ dryRun
+ ? "dry_run_files"
+ :
"delete_calls_completed",
+ 1);
+ LOG.debug(
Review Comment:
[P2] Preserve the file list for explicit distributed dry runs
The old INFO line was the per-file output of a distributed dry run. The
Flink procedure returns only count and bytes, so moving this to DEBUG leaves
users at normal log levels unable to see which paths would be removed. Please
keep a redacted path list for explicit dry runs, either in their logs or
through an equivalent audit output. [Flink cleanup
procedure](https://github.com/apache/paimon/blob/f9629393b911634ab490a8cc955a17d7659e030e/docs/docs/flink/procedures/repair.md#remove_orphan_files)
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/orphan/FlinkOrphanFilesClean.java:
##########
@@ -283,31 +343,48 @@ public void processElement(
ProcessFunction<String,
Tuple2<String, Long>>.Context
ctx,
Collector<Tuple2<String, Long>>
out) {
- Path dirPath = new Path(dir);
- List<FileStatus> files =
tryBestListingDirs(dirPath);
- for (FileStatus file : files) {
- if (!file.isDir()
- &&
!isManagedBlobPack(file.getPath())
- && oldEnough(file)) {
- out.collect(
- Tuple2.of(
-
file.getPath().toString(),
-
file.getLen()));
- }
- }
- if (files.isEmpty()) {
+ try (MaintenanceProgress progress =
+ new MaintenanceProgress(
Review Comment:
[P2] Aggregate orphan-cleanup diagnostics per Flink subtask
Each candidate directory gets its own START/END pair and scheduled task
here. The manifest-list reader at line 205 does the same for each snapshot. On
a subtask listing 100,000 directories, this adds 200,000 INFO records, even if
every call finishes before the heartbeat interval. Serial calls can also create
a fresh thread when they are the last reporter user. Please use a reporter
scoped to the subtask's `open`/`close`, record active paths with
`startItem`/`finishItem`, and aggregate counters across elements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]