kfaraz commented on code in PR #13223:
URL: https://github.com/apache/druid/pull/13223#discussion_r1000115785
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -1950,6 +1950,56 @@ public Boolean
apply(SeekableStreamIndexTaskRunner.Status status)
// make sure the checkpoints are consistent with each other and with the
metadata store
verifyAndMergeCheckpoints(taskGroupsToVerify.values());
+
+ // A pause from the previous Overlord's supervisor, immediately before
leader change,
+ // can lead to tasks being in a state where they are active but do not
read.
+ resumeAllActivelyReadingTasks();
+ }
+
+ /**
+ * If this is the first run, resume all tasks in the set of
activelyReadingTaskGroups
+ * Paused tasks will be resumed
+ * Other tasks in this set are not affected adversely by the resume operation
+ */
+ private void resumeAllActivelyReadingTasks()
+ {
+ if (!getState().isFirstRunOnly()) {
+ return;
+ }
+
+ Map<String, ListenableFuture<Boolean>> tasksToResume = new HashMap<>();
+ for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+ for (String taskId : taskGroup.tasks.keySet()) {
+ tasksToResume.put(taskId, taskClient.resumeAsync(taskId));
+ }
+ }
+
+ for (Map.Entry<String, ListenableFuture<Boolean>> entry :
tasksToResume.entrySet()) {
+ String taskId = entry.getKey();
+ ListenableFuture<Boolean> future = entry.getValue();
+ future.addListener(
+ new Runnable()
+ {
+ @Override
+ public void run()
+ {
+ try {
+ if (entry.getValue().get()) {
+ log.info("Resumed task [%s]", taskId);
+ } else {
+ log.warn("Failed to resume task [%s]", taskId);
+ killTask(taskId, "Could not resume task in the first
supervisor run after Overlord change.");
Review Comment:
Nit:
```suggestion
killTask(taskId, "Killing forcefully as task could not be
resumed in the first supervisor run after Overlord change.");
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -1950,6 +1950,56 @@ public Boolean
apply(SeekableStreamIndexTaskRunner.Status status)
// make sure the checkpoints are consistent with each other and with the
metadata store
verifyAndMergeCheckpoints(taskGroupsToVerify.values());
+
+ // A pause from the previous Overlord's supervisor, immediately before
leader change,
+ // can lead to tasks being in a state where they are active but do not
read.
+ resumeAllActivelyReadingTasks();
+ }
+
+ /**
+ * If this is the first run, resume all tasks in the set of
activelyReadingTaskGroups
+ * Paused tasks will be resumed
+ * Other tasks in this set are not affected adversely by the resume operation
+ */
+ private void resumeAllActivelyReadingTasks()
+ {
+ if (!getState().isFirstRunOnly()) {
+ return;
+ }
+
+ Map<String, ListenableFuture<Boolean>> tasksToResume = new HashMap<>();
+ for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+ for (String taskId : taskGroup.tasks.keySet()) {
+ tasksToResume.put(taskId, taskClient.resumeAsync(taskId));
+ }
+ }
+
+ for (Map.Entry<String, ListenableFuture<Boolean>> entry :
tasksToResume.entrySet()) {
+ String taskId = entry.getKey();
+ ListenableFuture<Boolean> future = entry.getValue();
+ future.addListener(
+ new Runnable()
+ {
+ @Override
+ public void run()
+ {
+ try {
+ if (entry.getValue().get()) {
+ log.info("Resumed task [%s]", taskId);
+ } else {
+ log.warn("Failed to resume task [%s]", taskId);
Review Comment:
```suggestion
log.warn("Failed to resume task [%s] in first supervisor
run.", taskId);
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -1950,6 +1950,56 @@ public Boolean
apply(SeekableStreamIndexTaskRunner.Status status)
// make sure the checkpoints are consistent with each other and with the
metadata store
verifyAndMergeCheckpoints(taskGroupsToVerify.values());
+
+ // A pause from the previous Overlord's supervisor, immediately before
leader change,
+ // can lead to tasks being in a state where they are active but do not
read.
+ resumeAllActivelyReadingTasks();
+ }
+
+ /**
+ * If this is the first run, resume all tasks in the set of
activelyReadingTaskGroups
Review Comment:
Thanks for the comments!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
