kfaraz commented on code in PR #13223:
URL: https://github.com/apache/druid/pull/13223#discussion_r998904713
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -1950,6 +1950,45 @@ public Boolean
apply(SeekableStreamIndexTaskRunner.Status status)
// make sure the checkpoints are consistent with each other and with the
metadata store
verifyAndMergeCheckpoints(taskGroupsToVerify.values());
+
+ // A pause from the previous Overlord's supervisor, immediately before
leader change
+ // can lead to tasks being in a state where they are active but do not
read.
+ // If this is the first run, resume all existing active tasks to be safe
+ if (getState().isFirstRunOnly()) {
+ Map<String, ListenableFuture<Boolean>> activeTaskToResumeFutureMap = new
HashMap<>();
+ for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+ for (String taskId : taskGroup.tasks.keySet()) {
+ activeTaskToResumeFutureMap.put(taskId,
taskClient.resumeAsync(taskId));
+ }
+ }
+
+ for (Map.Entry<String, ListenableFuture<Boolean>> entry :
activeTaskToResumeFutureMap.entrySet()) {
+ String taskId = entry.getKey();
+ ListenableFuture<Boolean> future = entry.getValue();
+ future.addListener(
+ new Runnable()
Review Comment:
Nit: prefer lambdas as they are easier to read.
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -1950,6 +1950,45 @@ public Boolean
apply(SeekableStreamIndexTaskRunner.Status status)
// make sure the checkpoints are consistent with each other and with the
metadata store
verifyAndMergeCheckpoints(taskGroupsToVerify.values());
+
+ // A pause from the previous Overlord's supervisor, immediately before
leader change
+ // can lead to tasks being in a state where they are active but do not
read.
+ // If this is the first run, resume all existing active tasks to be safe
+ if (getState().isFirstRunOnly()) {
+ Map<String, ListenableFuture<Boolean>> activeTaskToResumeFutureMap = new
HashMap<>();
Review Comment:
I am not sure if the term "active" task is used widely in the codebase. It
seems ambiguous, especially considering that the next line mentions
`activelyReadingTasks`.
I would suggest simply using `tasksToResume`.
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -1950,6 +1950,45 @@ public Boolean
apply(SeekableStreamIndexTaskRunner.Status status)
// make sure the checkpoints are consistent with each other and with the
metadata store
verifyAndMergeCheckpoints(taskGroupsToVerify.values());
+
+ // A pause from the previous Overlord's supervisor, immediately before
leader change
+ // can lead to tasks being in a state where they are active but do not
read.
+ // If this is the first run, resume all existing active tasks to be safe
+ if (getState().isFirstRunOnly()) {
+ Map<String, ListenableFuture<Boolean>> activeTaskToResumeFutureMap = new
HashMap<>();
+ for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+ for (String taskId : taskGroup.tasks.keySet()) {
+ activeTaskToResumeFutureMap.put(taskId,
taskClient.resumeAsync(taskId));
+ }
+ }
+
+ for (Map.Entry<String, ListenableFuture<Boolean>> entry :
activeTaskToResumeFutureMap.entrySet()) {
+ String taskId = entry.getKey();
+ ListenableFuture<Boolean> future = entry.getValue();
+ future.addListener(
+ new Runnable()
+ {
+ @Override
+ public void run()
+ {
+ try {
+ if (entry.getValue().get()) {
+ log.info("Resumed task [%s]", taskId);
+ } else {
+ log.warn("Failed to resume task [%s]", taskId);
+ killTask(taskId, "Could not resume task");
Review Comment:
Please make the kill message more informative.
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -1950,6 +1950,45 @@ public Boolean
apply(SeekableStreamIndexTaskRunner.Status status)
// make sure the checkpoints are consistent with each other and with the
metadata store
verifyAndMergeCheckpoints(taskGroupsToVerify.values());
+
+ // A pause from the previous Overlord's supervisor, immediately before
leader change
Review Comment:
Please put the changes in a separate method as this method is already very
cumbersome to read.
Style: return early rather than indenting the whole code block.
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -1950,6 +1950,45 @@ public Boolean
apply(SeekableStreamIndexTaskRunner.Status status)
// make sure the checkpoints are consistent with each other and with the
metadata store
verifyAndMergeCheckpoints(taskGroupsToVerify.values());
+
+ // A pause from the previous Overlord's supervisor, immediately before
leader change
+ // can lead to tasks being in a state where they are active but do not
read.
+ // If this is the first run, resume all existing active tasks to be safe
+ if (getState().isFirstRunOnly()) {
+ Map<String, ListenableFuture<Boolean>> activeTaskToResumeFutureMap = new
HashMap<>();
+ for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+ for (String taskId : taskGroup.tasks.keySet()) {
+ activeTaskToResumeFutureMap.put(taskId,
taskClient.resumeAsync(taskId));
+ }
+ }
+
+ for (Map.Entry<String, ListenableFuture<Boolean>> entry :
activeTaskToResumeFutureMap.entrySet()) {
+ String taskId = entry.getKey();
+ ListenableFuture<Boolean> future = entry.getValue();
+ future.addListener(
+ new Runnable()
+ {
+ @Override
+ public void run()
+ {
+ try {
+ if (entry.getValue().get()) {
Review Comment:
Do publishing tasks and other non-paused tasks return `true` for a resume
call?
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -1950,6 +1950,45 @@ public Boolean
apply(SeekableStreamIndexTaskRunner.Status status)
// make sure the checkpoints are consistent with each other and with the
metadata store
verifyAndMergeCheckpoints(taskGroupsToVerify.values());
+
+ // A pause from the previous Overlord's supervisor, immediately before
leader change
+ // can lead to tasks being in a state where they are active but do not
read.
+ // If this is the first run, resume all existing active tasks to be safe
+ if (getState().isFirstRunOnly()) {
Review Comment:
Please put the changes in a separate method as this method is already very
big.
Style: return early rather than indenting the whole block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
