[jira] [Commented] (KAFKA-5724) AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero
[ https://issues.apache.org/jira/browse/KAFKA-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16134658#comment-16134658 ] huxihx commented on KAFKA-5724: --- [~hachikuji] Did you get any time to look into this jira? Two possible ways to solve this problem: 1. Strictly honor the rule that partition must start from zero; 2. Pass more information than partition count when invoking `assign` , in doing so can we correctly do the assignment based on the metadata. What do you think? > AbstractPartitionAssignor does not take into consideration that partition > number may start from non-zero > > > Key: KAFKA-5724 > URL: https://issues.apache.org/jira/browse/KAFKA-5724 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.11.0.0 >Reporter: Allen Wang >Assignee: huxihx > > In AbstractPartitionAssignor.assign(Cluster metadata, Map Subscription> subscriptions), it invokes assign(partitionsPerTopic, > subscriptions). It assumes that partition number starts from 0, and it > constructs TopicPartition in the range of [0, partitionsPerTopic). > This assumption is not correct. The correct way to handle it is to follow the > same approach in producer's DefaultPartitioner, where it uses [0, > numberOfPartition) as an index to the actual partition. > There are use cases where partition number may not start from zero. It can > happen if users use advanced tooling to manually craft the partition number > when creating topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5724) AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero
[ https://issues.apache.org/jira/browse/KAFKA-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124786#comment-16124786 ] huxihx commented on KAFKA-5724: --- [~hachikuji] Currently, it is possible that user could employee KafkaAdminClient to create a topic with an assignment where partitions are neither consecutive nor zero-based. > AbstractPartitionAssignor does not take into consideration that partition > number may start from non-zero > > > Key: KAFKA-5724 > URL: https://issues.apache.org/jira/browse/KAFKA-5724 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.11.0.0 >Reporter: Allen Wang >Assignee: huxihx > > In AbstractPartitionAssignor.assign(Cluster metadata, Map Subscription> subscriptions), it invokes assign(partitionsPerTopic, > subscriptions). It assumes that partition number starts from 0, and it > constructs TopicPartition in the range of [0, partitionsPerTopic). > This assumption is not correct. The correct way to handle it is to follow the > same approach in producer's DefaultPartitioner, where it uses [0, > numberOfPartition) as an index to the actual partition. > There are use cases where partition number may not start from zero. It can > happen if users use advanced tooling to manually craft the partition number > when creating topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5724) AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero
[ https://issues.apache.org/jira/browse/KAFKA-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123631#comment-16123631 ] Allen Wang commented on KAFKA-5724: --- [~hachikuji] Please note that TopicCommand can take the replica assignment as an argument to create topic. So I believe it does not require much advanced tooling to get into this situation. > AbstractPartitionAssignor does not take into consideration that partition > number may start from non-zero > > > Key: KAFKA-5724 > URL: https://issues.apache.org/jira/browse/KAFKA-5724 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.11.0.0 >Reporter: Allen Wang >Assignee: huxihx > > In AbstractPartitionAssignor.assign(Cluster metadata, Map Subscription> subscriptions), it invokes assign(partitionsPerTopic, > subscriptions). It assumes that partition number starts from 0, and it > constructs TopicPartition in the range of [0, partitionsPerTopic). > This assumption is not correct. The correct way to handle it is to follow the > same approach in producer's DefaultPartitioner, where it uses [0, > numberOfPartition) as an index to the actual partition. > There are use cases where partition number may not start from zero. It can > happen if users use advanced tooling to manually craft the partition number > when creating topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5724) AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero
[ https://issues.apache.org/jira/browse/KAFKA-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123584#comment-16123584 ] Allen Wang commented on KAFKA-5724: --- [~hachikuji] We have our own topic creation tool where we take advantage of AdminUtils.assignReplicasToBrokers() where you can pass a starting partition ID. The API will return an assignment which can be passed to TopicCommand or AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK() API. > AbstractPartitionAssignor does not take into consideration that partition > number may start from non-zero > > > Key: KAFKA-5724 > URL: https://issues.apache.org/jira/browse/KAFKA-5724 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.11.0.0 >Reporter: Allen Wang >Assignee: huxihx > > In AbstractPartitionAssignor.assign(Cluster metadata, Map Subscription> subscriptions), it invokes assign(partitionsPerTopic, > subscriptions). It assumes that partition number starts from 0, and it > constructs TopicPartition in the range of [0, partitionsPerTopic). > This assumption is not correct. The correct way to handle it is to follow the > same approach in producer's DefaultPartitioner, where it uses [0, > numberOfPartition) as an index to the actual partition. > There are use cases where partition number may not start from zero. It can > happen if users use advanced tooling to manually craft the partition number > when creating topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5724) AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero
[ https://issues.apache.org/jira/browse/KAFKA-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123563#comment-16123563 ] Jason Gustafson commented on KAFKA-5724: [~allenxwang] I fear this is not the only place we make this assumption. Can you clarify specifically how an advanced tool creates a topic with a non-zero first partition? > AbstractPartitionAssignor does not take into consideration that partition > number may start from non-zero > > > Key: KAFKA-5724 > URL: https://issues.apache.org/jira/browse/KAFKA-5724 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.11.0.0 >Reporter: Allen Wang >Assignee: huxihx > > In AbstractPartitionAssignor.assign(Cluster metadata, Map Subscription> subscriptions), it invokes assign(partitionsPerTopic, > subscriptions). It assumes that partition number starts from 0, and it > constructs TopicPartition in the range of [0, partitionsPerTopic). > This assumption is not correct. The correct way to handle it is to follow the > same approach in producer's DefaultPartitioner, where it uses [0, > numberOfPartition) as an index to the actual partition. > There are use cases where partition number may not start from zero. It can > happen if users use advanced tooling to manually craft the partition number > when creating topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5724) AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero
[ https://issues.apache.org/jira/browse/KAFKA-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122966#comment-16122966 ] ASF GitHub Bot commented on KAFKA-5724: --- GitHub user huxihx opened a pull request: https://github.com/apache/kafka/pull/3655 KAFKA-5724: AbstractPartitionAssignor should support assignment for topics with non-consecutive partitions Current design does consider the siutation when user creates a topic via KafkaAdminClient whose partitions are not consecutive or zero-based. In such case, consumer does not work since assignor failed to assign partitions. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huxihx/kafka KAFKA-5724 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3655.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3655 commit 9ffa5a8961039affcc829b09912584b56063ddf3 Author: huxihx Date: 2017-08-11T07:20:52Z KAFKA-5724: AbstractPartitionAssignor does not take into consideration that partition number may start from non-zero Current design does consider the siutation when user creates a topic via KafkaAdminClient whose partitions are not consecutive or zero-based. In such case, consumer does not work since assignor failed to assign partitions. > AbstractPartitionAssignor does not take into consideration that partition > number may start from non-zero > > > Key: KAFKA-5724 > URL: https://issues.apache.org/jira/browse/KAFKA-5724 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.11.0.0 >Reporter: Allen Wang >Assignee: huxihx > > In AbstractPartitionAssignor.assign(Cluster metadata, Map Subscription> subscriptions), it invokes assign(partitionsPerTopic, > subscriptions). It assumes that partition number starts from 0, and it > constructs TopicPartition in the range of [0, partitionsPerTopic). > This assumption is not correct. The correct way to handle it is to follow the > same approach in producer's DefaultPartitioner, where it uses [0, > numberOfPartition) as an index to the actual partition. > There are use cases where partition number may not start from zero. It can > happen if users use advanced tooling to manually craft the partition number > when creating topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
