jt2594838 commented on PR #18163: URL: https://github.com/apache/iotdb/pull/18163#issuecomment-4922799380
> > If the system restarts before removing all files and after dropping the pipe, will the files still be removed? > > After startup finds stale pipe hardlink directories, it does not delete them synchronously in the startup path. Instead, it registers a periodic cleanup job: > > ```java > periodicalJobRegistrar.register( > PERIODICAL_CLEANUP_JOB_ID, > new PeriodicalStalePipeDirCleaner(stalePipeDirs)::cleanOneRound, > PipeConfig.getInstance().getPipeSubtaskExecutorCronHeartbeatEventIntervalSeconds()); > ``` > > The cleanup job deletes stale directories in throttled rounds. Each round stops when either condition is met: > > ``` > private static final long DELETE_MAX_PATH_COUNT_PER_ROUND = 100_000L; > private static final long DELETE_MAX_TIME_PER_ROUND_MS = 1_000L; > ``` > > So one cleanup round deletes at most 100,000 paths or runs for at most 1,000 ms. > > This prevents DataNode startup from being blocked by a large number of leftover pipe TsFile hardlinks. If cleanup is not finished in one round, the next periodic round continues from the remaining files. Is it possible to reuse the logic for dropping pipe? And hence avoid introducing a new thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
