abhishekagarwal87 commented on issue #12692:
URL: https://github.com/apache/druid/issues/12692#issuecomment-1166225796
As I was debugging this, I found two problems with
`LocalIntermediaryDataManager`.
- Both middle-manager and peons start `LocalIntermediaryDataManager`. There
is no guard to protect from race conditions as multiple services on the same
machine try to fiddle with local files. E.g. here is a potential race condition
- Task Y starts iterating through all the supervisor task directories on
local disk. There is no need to walk through all the supervisor task
directories but `LocalIntermediaryDataManager` does not know whether it is
running inside a peon or a middle manager. Task Y detects that there is a
directory for Task X. It is about to iterate through the contents of X's dir.
- Middle manager detects that X already completed and is expired. So it
will delete the contents of X's dir.
- Task Y gets an error during startup because X's dir cannot be found.
- Though looking further, it turns out the default expiration period for
deleting intermediate data is 1 day. The test certainly did not take a day. so
why did the MM delete the intermediate data at all? Weird. Another dead-end?
Well, thankfully not. Poking around the code further, you find this line of
code in
`LocalIntermediaryDataManager#deleteExpiredSuprevisorTaskPartitionsIfNotRunning`
```
if (checkTime.isAfter(now)) {
expiredSupervisorTasks.add(supervisorTaskId);
}
```
`checkTime` is regularly updated and is usually set to `now + expiryTime`.
If the above code snippet should really be as follows
```
if (now.isAfter(checkTime)) {
expiredSupervisorTasks.add(supervisorTaskId);
}
```
This bug has been around for ages now. It appears to be a simple fix but
fixing it has implications. Because of this bug, stale intermediate data gets
deleted in 5 minutes once the supervisor is completed. 5 minutes since that's
the frequency at which expired data is checked. If we fix this bug, on all the
druid installations out there, intermediate data will lie around for a day.
That can be a problem. Especially on the clusters that have a lot of
short-running batch tasks. We should definitely fix the peons to ignore the
contents belonging to a different task group. For the `expiration` bug, we
should fix it and reduce the default expiration time to 5 minutes in code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
