kfaraz commented on code in PR #20505:
URL: https://github.com/apache/druid/pull/20505#discussion_r4230612655
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/TaskActionToolbox.java:
##########
@@ -140,25 +151,61 @@ public boolean canBatchSegmentAllocation()
}
/**
- * Checks if the given publish action should be failed without allowing any
- * more retries. A failed publish action should be retried only if there is
- * another task waiting to publish offsets for an overlapping set of
partitions.
+ * Performs the segment publish action when the given task is unblocked for
publish.
+ * A streaming task must wait for previously created tasks that are
+ * yet to publish offsets for an overlapping set of partitions.
*/
- public boolean shouldFailSegmentPublishImmediately(
- SegmentPublishResult result,
+ public ListenableFuture<SegmentPublishResult> publishSegmentsWhenReady(
Task task,
String supervisorId,
- DataSourceMetadata startMetadata
+ DataSourceMetadata startMetadata,
+ BiFunction<Task, TaskActionToolbox, SegmentPublishResult> publishAction
)
{
- if (result.isSuccess() || !result.isRetryable() || startMetadata == null) {
- return false;
+ // First try publishing synchronously
+ try {
+ final SegmentPublishResult firstAttemptResult =
publishAction.apply(task, this);
+ if (firstAttemptResult.isSuccess()
+ || !firstAttemptResult.isOffsetMismatch()
+ || !firstAttemptResult.isRetryable()) {
+ return Futures.immediateFuture(firstAttemptResult);
+ }
+ }
+ catch (Exception e) {
+ return Futures.immediateFailedFuture(e);
}
- return !getSupervisorManager().isAnotherTaskGroupPublishingToPartitions(
+ // Try publishing later if the failure was due to offset mismatch
+ final ListenableFuture<Boolean> taskReadyToPublishFuture =
supervisorManager.isTaskReadyToPublishSegments(
supervisorId,
task.getId(),
startMetadata
);
+
+ return Futures.transform(
+ taskReadyToPublishFuture,
+ readyToPublish -> {
+ if (Boolean.TRUE.equals(readyToPublish)) {
+ // Task is already unblocked for publish, retrying will not fix
offset mismatch
+ return doNotRetryOffsetMismatchFailure(publishAction.apply(task,
this));
Review Comment:
updated - made the logic inline instead of separate method.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/TaskActionToolbox.java:
##########
@@ -63,6 +72,7 @@ public TaskActionToolbox(
this.supervisorManager = supervisorManager;
this.jsonMapper = jsonMapper;
this.segmentAllocationQueue = segmentAllocationQueue;
+ this.actionExec = scheduledExecutorFactory.create(4,
"TaskActionToolbox-%s");
Review Comment:
added, thanks for the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
