cecemei commented on code in PR #19074:
URL: https://github.com/apache/druid/pull/19074#discussion_r2886714074
##########
multi-stage-query/src/main/java/org/apache/druid/msq/indexing/cleaner/DurableStorageCleaner.java:
##########
@@ -102,9 +102,14 @@ public void run() throws Exception
if (nextDirName != null && !nextDirName.isEmpty()) {
if (runningTaskIds.contains(nextDirName)) {
// do nothing
- } else if (DurableStorageUtils.QUERY_RESULTS_DIR.equals(nextDirName)
- && DurableStorageUtils.isQueryResultFileActive(currentFile,
knownTaskIds)) {
- // query results should not be cleaned even if the task has finished
running
+ } else if (
+ DurableStorageUtils.QUERY_RESULTS_DIR.equals(nextDirName)
+ && DurableStorageUtils.isQueryResultFileActive(
+ currentFile,
+ taskId ->
Optional.fromNullable(taskStorage.getTaskInfo(taskId)).transform(TaskInfo::getCreatedTime),
Review Comment:
updated to only fetch completed task within retention period, we don't have
filter by type in `CompleteTaskLookup`, it wont be too much work to add such
filter. i just assume that even without this filter we wont have too much tasks
finished within the last 6 hours.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
