[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271797#comment-14271797 ] Neha Narkhede commented on KAFKA-1753: -- [~joestein], KAFKA-1792 is waiting on a proposal on how the user will interact with the partition reassignment, given that we automated the assignment part. Once we have that, it will be easier to check it in. add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249692#comment-14249692 ] Dmitry Pekar commented on KAFKA-1753: - [~charmalloc] So we need a merge of KAFKA-1792 before starting this or should I provide a specialized algorithm for decommission broker in the scope of this ticket? add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249913#comment-14249913 ] Joe Stein commented on KAFKA-1753: -- [~Dmitry Pekar] patch this feature option on this ticket. We should see about committing the other two fixes you patched to trunk. add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250158#comment-14250158 ] Neha Narkhede commented on KAFKA-1753: -- [~charmalloc] The only thing we are waiting on for KAFKA-1792 is extensive testing that proves the effectiveness of the algorithm. If we implemented some published and accepted algorithm, then we wouldn't need this. But since we haven't, we would want to do the same sort of testing that would convince us that the algorithm yields a balanced cluster with minimal replica movement. If we have that, we can move forward with KAFKA-1792 as well as this ticket. add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14233760#comment-14233760 ] Neha Narkhede commented on KAFKA-1753: -- [~charmalloc] That sounds right. Though the tool would have to provide an option to allow the user to decommission the broker. Internally what will happen is what you described above. add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230239#comment-14230239 ] Joe Stein commented on KAFKA-1753: -- So, I think with the changes in KAFKA-1792 this now becomes the same type of algo but the partitions on the broker being decommissioned are what we are evenly spreading to the rest of the cluster nodes essentially. add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206698#comment-14206698 ] Gwen Shapira commented on KAFKA-1753: - --replace-broker 1 --broker-list 2,3,4 can work. I'm wondering about the semantics when broker list is not mentioned. Is it an error, or do we assume just move partitions to any broker, try to balance things out. The nice thing about --decommission-broker semantics is that I can have a 30 node cluster and decommission one without caring especially where partitions go to. A broker list with 29 brokers is pretty annoying :) add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206705#comment-14206705 ] Joe Stein commented on KAFKA-1753: -- A broker list with 29 brokers is pretty annoying that is a good point so does it make sense (and what we are saying) then to have --decommission-broker 5 in which case you don't supply the list and it spreads it around the cluster where it should and (for KAFKA-1752) --replace-broker 1 --broker-list 2,3,4 for explicit actions add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206712#comment-14206712 ] Gwen Shapira commented on KAFKA-1753: - Yes, I think this will be usable for most cases (at least any usage I can think of...) add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206845#comment-14206845 ] Dmitry Pekar commented on KAFKA-1753: - Ok. In that case I will add --decomission-broker X with semantic exactly the same as --replace-broker X --broker-list all except X add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204782#comment-14204782 ] Dmitry Pekar commented on KAFKA-1753: - [~gwenshap] [~nehanarkhede] [~charmalloc] I was playing with my initial implementation of --replace-broker command and found that it could be easily generalized to support delete broker scenario without introducing additional UX complexity. IMHO, it should be easy for a user to understand that to delete broker X he should replace broker X with some broker Y from the cluster (because the cluster should preserve all partition replicas located previously on broker X). So --replace-broker command would cover --decommission-broker scenario. What do you think about dropping --decommission-broker in favor of --replace-broker? add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205848#comment-14205848 ] Neha Narkhede commented on KAFKA-1753: -- [~Dmitry Pekar] It will help to know how you are thinking of implementing this? Decommissioning a broker involves moving all partitions away from it and evenly distributing those onto the remaining brokers. This is done mainly to ensure that the replication factor does not drop when the broker is removed or brought down. add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201162#comment-14201162 ] Neha Narkhede commented on KAFKA-1753: -- +1 on decommission-broker add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199035#comment-14199035 ] Dmitry Pekar commented on KAFKA-1753: - Proposal for implementation is following: add --remove-broker CLI option. Value of option is broker id. Example --remove-broker 1 means to remove broker[id=1] from cluster. Please, negotiate. add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199524#comment-14199524 ] Gwen Shapira commented on KAFKA-1753: - +1 Great idea IMO. Will make life easier for a pretty common use-case. I like the --decomission-broker name better than --remove-broker (just because it matches the Hadoop terminology) add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199564#comment-14199564 ] Joe Stein commented on KAFKA-1753: -- +1 naming the option --decomission-broker add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199670#comment-14199670 ] Jun Rao commented on KAFKA-1753: Is the idea just moving all data off the decommissioned topic using the partition reassignment tool under the cover? add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199675#comment-14199675 ] Joe Stein commented on KAFKA-1753: -- yes add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
