cryptoe commented on code in PR #19772:
URL: https://github.com/apache/druid/pull/19772#discussion_r3660387422
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/duty/UnusedSegmentsKiller.java:
##########
@@ -245,22 +256,53 @@ private void rebuildKillQueue()
return;
}
- final Set<String> dataSources =
storageCoordinator.retrieveAllDatasourceNames();
-
final Map<String, Integer> dataSourceToIntervalCounts = new HashMap<>();
- for (String dataSource : dataSources) {
- storageCoordinator.retrieveSomeUnusedSegmentIntervals(dataSource,
MAX_INTERVALS_TO_KILL_IN_DATASOURCE).forEach(
- interval -> {
- dataSourceToIntervalCounts.merge(dataSource, 1, Integer::sum);
- killQueue.offer(new KillCandidate(dataSource, interval));
- }
+
+ // Identify intervals with unused segments which are eligible for kill
+ final Map<DatasourceInterval, Integer> killCandidates =
+ storageCoordinator.retrieveSomeUnusedSegmentIntervals(
+ DateTimes.nowUtc().minus(killConfig.getBufferPeriod()),
Review Comment:
So we have basically pushed in the bufferPeriod inside the sql query so that
we donot populate segments which would be ineligible.
I think this change alone would get us lot of improvements.
I am a bit worried about fetching all the data sources at once but I guess
there is where the inner query limit comes into place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
