Re: Problems trying to make kafka 'rack-aware'

2018-09-21 Thread Bryan Duggan
Hi Eno, many thanks for trying that. That is very helpful for me. That basic check didn't work for me but I have since discovered what my issue was. Despite using a version of kafka that supports rack-awareness we have been deliberately setting 'inter.broker.protocol.version' to an older vers

Re: Problems trying to make kafka 'rack-aware'

2018-09-21 Thread Eno Thereska
Hi Bryan, I did a simple check with starting a broker with no rack id and then restarting with a rack id and I can confirm I could get the rack id from zookeeper after the restart. This was on trunk. Does that basic check work for you (i.e., without reassigning partitions)? Thanks Eno On Fri, Se

Re: Problems trying to make kafka 'rack-aware'

2018-09-21 Thread Bryan Duggan
I didn't get a response to this, but I've been investigating more and can now frame the problem slightly differently (hopefully, more accurately). According to this document https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper Which defines broker data struct

Problems trying to make kafka 'rack-aware'

2018-09-20 Thread Bryan Duggan
Hi, I have a kafka cluster consisting of 3 brokers across 3 different AWS availability zones.  It hosts several topics, each of which has a replication factor of 3. The cluster is currently not 'rack-aware'. I am trying to do the following;     - add 3 additional brokers (one in each of the