jtuglu1 commented on code in PR #19203:
URL: https://github.com/apache/druid/pull/19203#discussion_r2992971711
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -417,8 +418,14 @@ private void startPendingTasksOnRunner()
log.info("Notified task runner to clean up [%,d] tasks with IDs[%s].",
unknownTaskIds.size(), unknownTaskIds);
// Attain futures for all active tasks (assuming they are ready to run).
- // Copy tasks list, as notifyStatus may modify it.
- for (final String queuedTaskId : List.copyOf(activeTasks.keySet())) {
+ // Copy tasks list, as notifyStatus may modify it. Sort by priority
(highest first) so that
+ // higher-priority tasks are submitted to the runner before lower-priority
ones.
+ final List<String> queuedTaskIds = new ArrayList<>(activeTasks.keySet());
+ queuedTaskIds.sort(Comparator.comparingInt(id -> {
+ final TaskEntry entry = activeTasks.get(id);
Review Comment:
Good question. Copying O(10k) is going to be expensive regardless I think.
So `ConcurrentHashmap` iterators are weakly consistent (that is, this is – I
believe – providing the same consistency gaurantees as the current
List.copyOf(). The only difference here is that we create a mutable copy for
sorting. Nothing has changed here w.r.t copying of the data.
I think whether we do want a locked snapshot (e.g. prevent any
modifications) is a separate question. I'm fine with wrapping a mutex around
this lock, we just don't have a write lock over `startStopLock` in this context
(currently we support multiple concurrent readers). While I think the correct
(although slower) approach would be to grab a mutex over the entire map, I
would prefer to keep this as-is (and make the strong snapshot consistency
change in a separate PR) as IMO this would be changing the behavior of the
queue (both performance and correctness-wise).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
