jtuglu1 commented on code in PR #19431:
URL: https://github.com/apache/druid/pull/19431#discussion_r3206560746
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -4099,6 +4099,12 @@ private void createNewTasks() throws
JsonProcessingException
);
// check that there is a current task group for each group of partitions
in [partitionGroups]
+ // Fetch metadata offsets once and collect all stale partitions across all
groups before committing
+ // any state changes. New TaskGroups are staged locally and only written
to activelyReadingTaskGroups
+ // if no reset is required, keeping state consistent: either all new
groups are committed or none are.
+ final Map<PartitionIdType, SequenceOffsetType> metadataOffsets =
getOffsetsFromMetadataStorage();
Review Comment:
LMK what you think about this – I think this shouldn't impact task groups
without invalid offsets as currently they'd just be added to the
activelyReadingTaskGroups then be delayed in processing (see few lines down)
since generateStartingSequencesForPartitionGroup would just throw an exception
(and force a new runInternal()).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
