luoluoyuyu commented on PR #18163: URL: https://github.com/apache/iotdb/pull/18163#issuecomment-4924306976
> > > If the system restarts before removing all files and after dropping the pipe, will the files still be removed? > > > > > > After startup finds stale pipe hardlink directories, it does not delete them synchronously in the startup path. Instead, it registers a periodic cleanup job: > > ```java > > periodicalJobRegistrar.register( > > PERIODICAL_CLEANUP_JOB_ID, > > new PeriodicalStalePipeDirCleaner(stalePipeDirs)::cleanOneRound, > > PipeConfig.getInstance().getPipeSubtaskExecutorCronHeartbeatEventIntervalSeconds()); > > ``` > > > > > > > > > > > > > > > > > > > > > > > > The cleanup job deletes stale directories in throttled rounds. Each round stops when either condition is met: > > ``` > > private static final long DELETE_MAX_PATH_COUNT_PER_ROUND = 100_000L; > > private static final long DELETE_MAX_TIME_PER_ROUND_MS = 1_000L; > > ``` > > > > > > > > > > > > > > > > > > > > > > > > So one cleanup round deletes at most 100,000 paths or runs for at most 1,000 ms. > > This prevents DataNode startup from being blocked by a large number of leftover pipe TsFile hardlinks. If cleanup is not finished in one round, the next periodic round continues from the remaining files. > > Is it possible to reuse the logic for dropping pipe? And hence avoid introducing a new thread. Yes, I agree. Reusing the existing stale pipe directory cleanup logic is better than introducing a new dedicated thread. I updated the drop-pipe cleanup path to follow the same mechanism: when a pipe is dropped, its pipe-specific TsFile directory is first renamed to a stale directory like <pipeName_createTime>.startup-cleaning-<timestamp>-<index>, then submitted to the existing periodical stale-dir cleaner. The cleaner deletes it in throttled rounds with the existing limits: at most 100,000 paths or 1,000 ms per round. This avoids blocking the drop path with synchronous per-file deletion, avoids adding another cleanup thread, and keeps restart recovery consistent. If the DataNode restarts before deletion finishes, startup will find the stale directory and the same periodical cleanup job will continue removing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
