jon-wei commented on a change in pull request #8671: More Kinesis resharding
adjustments
URL: https://github.com/apache/incubator-druid/pull/8671#discussion_r335254721
##########
File path:
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
##########
@@ -1946,53 +1956,152 @@ private boolean updatePartitionDataFromStream()
}
}
- if (supportsPartitionExpiration()) {
- // Look for expired shards and remove them from metadata storage and the
partition groups
- Set<PartitionIdType> expiredPartitions = new HashSet<>();
- for (PartitionIdType partitionTd : closedPartitions) {
- if (!partitionIds.contains(partitionTd)) {
- expiredPartitions.add(partitionTd);
+ if (!partitionIds.equals(previousPartitionIds)) {
+ // the set of partition IDs has changed, have any running tasks stop
early so that we can adjust to the
+ // repartitioning quickly by creating new tasks
+ for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+ if (!taskGroup.taskIds().isEmpty()) {
+ // a new partition was added and we are managing active tasks - set
an early publish time 2 minutes in the
+ // future to give things time to settle
+ earlyStopTime = DateTimes.nowUtc().plusMinutes(2);
Review comment:
I updated this to use a configurable `repartitionTransitionDuration`
property under the supervisor tuning config, along with docs on what the
property is for (giving the stream time to write records to the new shards
before transitioning to a new set of tasks)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
