gianm commented on code in PR #14662:
URL: https://github.com/apache/druid/pull/14662#discussion_r1277811817
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillUnusedSegmentsTask.java:
##########
@@ -128,6 +146,9 @@ public TaskStatus runTask(TaskToolbox toolbox) throws
Exception
}
// Kill segments
+ unusedSegments = maxSegmentsToKill == null
+ ? unusedSegments
Review Comment:
IMO, kill tasks shouldn't mark stuff unused at all. It's best if there is a
time delay between markings things unused and permanently deleting them. It's
possible for people to make mistakes with what they delete, and we want the
mistakes to be undoable for some period of time.
I think the best thing to do here is actually deprecate the `markUnused`
parameter, and focus on workflows that are like:
- Users delete stuff by marking it unused
- Later, the system deletes it permanently by way of autokill (which runs
these `kill` tasks)
Btw, an important part of making this work is including a buffer period
between when a segment is marked unused, and when autokill would remove it.
This was discussed in #12526 although I am not sure what the current status is.
For _this_ patch, IMO the best approach is to have the Coordinator send a
list of segments to the `kill` task, rather than `interval` +
`maxSegmentsToKill`. This will make it easier to implement the buffer period we
need for autokill later— the Coordinator would be in charge of determining
which specific segments are killable, and it would need the `kill` task to
respect that specific list. This also side-steps the issue with `markAsUnused`,
since the `maxSegmentsToKill` parameter would no longer exist.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/KillUnusedSegmentsTask.java:
##########
@@ -128,6 +146,9 @@ public TaskStatus runTask(TaskToolbox toolbox) throws
Exception
}
// Kill segments
+ unusedSegments = maxSegmentsToKill == null
+ ? unusedSegments
Review Comment:
IMO, kill tasks shouldn't mark stuff unused at all. It's best if there is a
time delay between markings things unused and permanently deleting them. It's
possible for people to make mistakes with what they delete, and we want the
mistakes to be undoable for some period of time.
I think the best thing to do here is actually deprecate the `markUnused`
parameter, and focus on workflows that are like:
- Users delete stuff by marking it unused
- Later, the system deletes it permanently by way of autokill (which runs
these `kill` tasks)
Btw, an important part of making this work is including a buffer period
between when a segment is marked unused, and when autokill would remove it.
This was discussed in #12526 although I am not sure what the current status is.
For _this_ patch, IMO the best approach is to have the Coordinator send a
list of segments to the `kill` task, rather than `interval` +
`maxSegmentsToKill`. This will make it easier to implement the buffer period we
need for autokill later— the Coordinator would be in charge of determining
which specific segments are killable, and it would need the `kill` task to
respect that specific list. This also side-steps the issue with `markAsUnused`,
since the `maxSegmentsToKill` parameter would no longer exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
