luoluoyuyu commented on PR #18163:
URL: https://github.com/apache/iotdb/pull/18163#issuecomment-4922556362
> If the system restarts before removing all files and after dropping the
pipe, will the files still be removed?
After startup finds stale pipe hardlink directories, it does not delete them
synchronously in the startup path. Instead, it registers a periodic cleanup job:
```java
periodicalJobRegistrar.register(
PERIODICAL_CLEANUP_JOB_ID,
new PeriodicalStalePipeDirCleaner(stalePipeDirs)::cleanOneRound,
PipeConfig.getInstance().getPipeSubtaskExecutorCronHeartbeatEventIntervalSeconds());
```
The cleanup job deletes stale directories in throttled rounds. Each round
stops when either condition is met:
```
private static final long DELETE_MAX_PATH_COUNT_PER_ROUND = 100_000L;
private static final long DELETE_MAX_TIME_PER_ROUND_MS = 1_000L;
```
So one cleanup round deletes at most 100,000 paths or runs for at most 1,000
ms.
This prevents DataNode startup from being blocked by a large number of
leftover pipe TsFile hardlinks. If cleanup is not finished in one round, the
next periodic round continues from the remaining files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
