FrankChen021 commented on code in PR #18226:
URL: https://github.com/apache/druid/pull/18226#discussion_r4429536072
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java:
##########
@@ -1015,6 +1030,11 @@ public void onFailure(Throwable t)
}
ingestionState = IngestionState.COMPLETED;
+ if (unrecoverableReadException != null) {
+ final String errorMsg = unrecoverableReadException.getMessage();
+ toolbox.getTaskReportFileWriter().write(task.getId(),
getTaskCompletionReports(errorMsg, handoffWaitMs));
+ return TaskStatus.failure(task.getId(), errorMsg);
Review Comment:
The failure path only triggers when the next offset is below the partition's
earliest available offset. Since retention removes the log from the head, every
offset below it, including the task group's starting/checkpoint offsets, is
gone too, so a replacement task created from the stale group sequences gets
`OffsetOutOfRangeException` below the earliest offset on its first poll and
fails without reading or publishing anything.
Duplicates are also prevented by the compare-and-swap on datasource
metadata: the failed task committed up to the unavailable offset, so a publish
starting from older offsets is rejected.
Once the group has no tasks, `createNewTasks()` →
`verifyAndMergeCheckpoints()` clears it and resets `partitionOffsets`, so the
next group reads its start offsets from metadata. The availability check then
reports "Previous sequenceNumber … is no longer available" and the supervisor
goes unhealthy, which is the intended outcome. The few extra fast-failing tasks
count toward `taskUnhealthinessThreshold`, making the problem visible rather
than retrying silently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
