jon-wei opened a new pull request #8644: Fix Kinesis resharding issues URL: https://github.com/apache/incubator-druid/pull/8644 This PR makes several adjustments to areas related to Kinesis resharding: - The Kinesis supervisor now filters out closed shards before sending partition/sequence num mappings in task specs. Previously this would cause tasks to fail because the supervisor does not include closed shards in the setEndOffsets call. - When a shard expires, the supervisor will now remove the expired shard from metadata storage and the `partitionGroups`. - The logic for determining shard ID -> task group mappings is changed. Previously this was done with (index of shard ID in shard list % task count). This fluid mapping caused issues with the same shard being assigned to multiple task groups when expired shards were removed from the shard list. This now attempts to parse the shard number from the shard ID, instead of using the list position. If the parse attempt fails, the number % task count will be a hash of the shard ID instead. - (Unrelated to sharding) Fixes a bug in SeekableStreamIndexTaskRunner with the publishing sequences: When iterating through the list of sequences to publish, it was possible for the publish future to complete before the task was done iterating through the sequence list; this would cause sequences to be skipped, as the publish future removes items from the sequence list. This block now snapshots the sequence list before publishing them. - Adds shard split/merge test to Kinesis supervisor tests, adds end-of-shard handling test to Kinesis task test This PR has: - [x] been self-reviewed. - [x] using the [concurrency checklist](https://github.com/apache/incubator-druid/blob/master/dev/code-review/concurrency.md) - [x] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths. - [x] been tested in a test Druid cluster.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
