RE: ConsumerRebalanceFailedException

To: users@kafka.apache.org Subject: Re: ConsumerRebalanceFailedException Could you send around the consumer log when it throws ConsumerRebalanceFailedException. It should state the reason for the failed rebalance attempts. Thanks, Neha On Tue, Feb 25, 2014 at 12:01 PM, Yu, Libo libo...@citi.com

ConsumerRebalanceFailedException

Hi all, I tried to reproduce this exception. In case one, when no broker was running, I launched all consumers and got this exception. In case two, while the consumers and brokers were running, I shutdown all brokers one by one and did not see this exception. I wonder why in case two this

RE: broker offline

-register upon receiving the session timeout. You can re-produce this issue by signal pause the ZK process. Guozhang On Fri, Feb 14, 2014 at 12:15 PM, Yu, Libo libo...@citi.com wrote: Hi team, We have three brokers on our production cluster. I noticed two of them somehow got offline and then re

broker offline

Hi team, We have three brokers on our production cluster. I noticed two of them somehow got offline and then re-registered with zookeeper and got back online. It seems the issue was caused by some zookeeper issue. So I want to know what may be the possible cases of the issue. If I want to

RE: some brokers cannot register themselves with zookeeper

version are you using? Pre-0.8.1 there is a bug that can cause a registration path to be deleted: https://issues.apache.org/jira/browse/KAFKA-992 And this has been fixed in 0.8.1 Guozhang On Tue, Feb 11, 2014 at 1:16 PM, Yu, Libo libo...@citi.com wrote: Hi team, This is an issue that has

RE: How to compile with a newer version of zookeeper

When I telnet to the zookeeper and type status, this is what I got: Zookeeper version: 3.3.3-1203054, built on 11/17/2011 05:47 GMT Is that 3.3.4? So 0.8 final also uses 3.3.4, is that right? Thanks. Regards, Libo -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com]

RE: Consumer's behavior when brokers are temporarily not available.

rebalances. Thanks, Jun On Thu, Feb 6, 2014 at 10:46 AM, Yu, Libo libo...@citi.com wrote: While the broker is not available (caused by zookeeper issue), the rebalance will fail. Should rebalance succeed in this case? Thanks. Regards, Libo -Original Message- From: Guozhang

Consumer's behavior when brokers are temporarily not available.

Hi folks, This is what we experienced recently: Some zookeeper's issue made broker unavailable for a short period of time. On the consumer side, this triggered rebalance and rebalanced failed after four tries. So while should we expect while the broker is not up? Should consumer keep trying to

RE: Consumer's behavior when brokers are temporarily not available.

, Feb 6, 2014 at 9:05 AM, Yu, Libo libo...@citi.com wrote: Hi folks, This is what we experienced recently: Some zookeeper's issue made broker unavailable for a short period of time. On the consumer side, this triggered rebalance and rebalanced failed after four tries. So while should we

RE: Consumer's behavior when brokers are temporarily not available.

, Feb 6, 2014 at 9:05 AM, Yu, Libo libo...@citi.com wrote: Hi folks, This is what we experienced recently: Some zookeeper's issue made broker unavailable for a short period of time. On the consumer side, this triggered rebalance and rebalanced failed after four tries. So while should we expect

num.partitions

Hi team, I believe num.partitions is for automatic topic creation. Is that right? The default number of partition for kafka-create-topic.sh is 1. So Will num.partitions impact kafka-create-topic.sh? Thanks. Regards, Libo

RE: purging a topic

You would need to first stop the consumer, update the offset in ZK and then restart the consumer. Also, have you looked at the tool ImportZkOffsets? Thanks, Jun On Tue, Jan 14, 2014 at 12:38 PM, Yu, Libo libo...@citi.com wrote: Hi folks, I am writing a tool to purge the pending topics

purging a topic

Hi folks, I am writing a tool to purge the pending topics for a user. Assume the user has never consumed this topic previously. If I create all the nodes on the path /consumers/[myuser]/offsets/[mytopic]/[partition] and put the maximum available offset to the node, is that enough to let the

RE: ConsumerRebalanceFailedException

Hi Jun, zookeeper.session.timeout.ms is used in a broker's configuration and manages brokers' registration with zk. Does it apply to consumer as well? Thanks. Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Monday, December 30, 2013 11:13 AM To:

RE: upgrade from beta1 to 0.81

...@gmail.com wrote: 0.8.1 is working in stable at LinkedIn now. Guozhang On Thu, Dec 19, 2013 at 10:52 AM, Yu, Libo libo...@citi.com wrote: I also want to know how stable the 0.81 will be, compared with 0.8 or 0.8-beta1. Regards, Libo -Original Message- From: Jason

upgrade from beta1 to 0.81

Hi folks, As the tools in 0.8 are not stable and we don't want to take the risk. we want to skip 0.8 and upgrade from beta1 to 0.81 directly. So my question is whether we can do an in place upgrade and let 0.81 use beta1's zk and kf data. Assume that we will disable log compaction. Thanks.

RE: a consumer question

On Tue, Dec 17, 2013 at 4:57 PM, Guozhang Wang wangg...@gmail.com wrote: If there is no more messages, hasNext will return false instead of throwing an exception. Guozhang On Tue, Dec 17, 2013 at 11:53 AM, Yu, Libo libo...@citi.com wrote: Sorry, a typo. Correct my question. When

RE: a consumer question

: Tuesday, December 17, 2013 12:40 AM To: users@kafka.apache.org Subject: Re: a consumer question If there is a message, hasNext() returns true, not throwing an exception. Thanks, Jun On Mon, Dec 16, 2013 at 11:29 AM, Yu, Libo libo...@citi.com wrote: Hi folks, For this parameters

RE: cluster expansion

brokers or you could add partitions to the new brokers for each topic. In 0.8.0 there is now an add-partitions tool. Cheers Rob Turner. On 13 December 2013 14:42, Yu, Libo libo...@citi.com wrote: Hi folks, There are three brokers running 0.8-beta1 in our

RE: cluster expansion

-Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Monday, December 16, 2013 10:26 AM To: users@kafka.apache.org Subject: RE: cluster expansion They will be evenly distributed across the nodes in the cluster. Thanks, Neha On Dec 16, 2013 6:42 AM, Yu, Libo libo

RE: cluster expansion

distributed on all six brokers. If I use reassignment tool in 0.81 with 0.8 broker, will that work and get around the bugs? Your broker also needs to be on 0.8.1 for it to work correctly. On Mon, Dec 16, 2013 at 9:06 AM, Yu, Libo libo...@citi.com wrote: If we have six brokers, and a topic has

cluster expansion

Hi folks, There are three brokers running 0.8-beta1 in our cluster currently. Assume all the topics have six partitions. I am going to add another three brokers to the cluster and upgrade all of them to 0.8. My question is after the cluster is up, will the partition be evenly distributed to all

error from adding a partition

Hi folks, I got this error when I tried to test the partition addition tool. bin/kafka-add-partitions.sh --partition 1 --topic libotesttopic --zookeeper xx.xxx.xxx.xx: adding partitions failed because of

RE: set new retention size on the fly

. 0.8.1 makes all per-topic configuration dynamic and updatable via a command line tool. -Jay On Tue, Dec 3, 2013 at 1:23 PM, Yu, Libo libo...@citi.com wrote: Hi Neha, 0.8.1 includes the ability to dynamically change per topic configs. Do you mean number of partitions or retention size

RE: set new retention size on the fly

+release+plan Thanks, Jun On Wed, Dec 4, 2013 at 8:15 AM, Yu, Libo libo...@citi.com wrote: Thanks for the clarification. I am just curious about how this works out. If we can change the retention size with kafka-topics.sh --alter, will the new retention size be updated

set new retention size on the fly

Hi folks, For 0.8, it is possible to add a partition dynamically. Is it possible to increase the retention size on the fly? This feature will be very useful for operation. I know rolling start can pick up the change but it takes too much effort. Thanks. Libo

RE: set new retention size on the fly

, Yu, Libo libo...@citi.com wrote: Hi folks, For 0.8, it is possible to add a partition dynamically. Is it possible to increase the retention size on the fly? This feature will be very useful for operation. I know rolling start can pick up the change but it takes too much effort. Thanks

RE: ConsumerRebalanceFailedException

will not be consumed by any consumers. Thanks, Jun On Fri, Nov 29, 2013 at 10:44 AM, Yu, Libo libo...@citi.com wrote: You are right, Joe. I checked our brokers' log. We have three brokers. All of them failed to connect to zk at some point. So they were offline and later reregistered themselves with the zk

RE: ConsumerRebalanceFailedException

/ On Fri, Nov 29, 2013 at 11:31 AM, Yu, Libo libo...@citi.com wrote: We found our consumer stopped working after this exception occurred. Can the consumer recover from such an exception? Regards, Libo -Original Message- From: Florin Trofin [mailto:ftro...@adobe.com

addition of a partition

Hi team, I am reading this link: https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-5.AddPartitionTool and this JIRA https://issues.apache.org/jira/i#browse/KAFKA-1030. I have a couple of questions. After adding a partition by using the tool, should the

about compiling broker in 0.8 branch

Hi team, For the current 0.8 branch, is it recommended to compile it with Scala 2.10? I remember someone said previously it is best to compile the broker with Scala 2.80. Thanks. Regards, Libo

RE: retention size

the producer sticks to a partition for the metadata refresh period - so if your test run isn't long enough some partitions may be more loaded than the others. On Thu, Nov 21, 2013 at 06:28:39PM +, Yu, Libo wrote: Hi team, We have 3 brokers in a cluster. The replication factor is 2. I set

retention size

Hi team, We have 3 brokers in a cluster. The replication factor is 2. I set the default retention size to 3G bytes. I published 12G data to a topic, which is enough to fully load all partitions. I assume on each broker the partition size should be 3G. However, it is only 1.4G for one

NodeExists Keeper exception

Hi team, Still this is from beta1. I notice this exception occurred frequently in our broker logs. [2013-11-14 21:09:58,714] INFO Got user-level KeeperException when proce ssing sessionid:0x24250e816b000df type:create cxid:0x26b zxid:0x fffe txntype:unknown reqpath:n/a Error

broker exception

Hi team, This exception occurs regularly on our brokers. When it occurs, a broker will lose its leader role but still in ISR. And running preferred-leader-election script may rebalance the leadership but in some cases it does not help. [2013-11-14 08:04:40,001] INFO Processed session

will this cause message loss?

Hi team, We are using beta1. I am going to delete all topics and create them with more partitions. But I don't want to lose any messages. Assume the consumers are online all the time for the following steps. The consumer's auto.offset.reset is set to largest. 1 stop publishing to the brokers.

RE: Leader not local

? Guozhang On Thu, Nov 14, 2013 at 8:23 AM, Yu, Libo libo...@citi.com wrote: Hi team, I saw this line within a long time span in our logs for the same topic and partition. [2013-11-14 11:13:41,647] WARN [KafkaApi-1] Produce request with correlation id 529240 from client on partition

RE: broker exception

...@gmail.com] Sent: Thursday, November 14, 2013 12:27 PM To: users@kafka.apache.org Subject: Re: broker exception Are you using ZK 3.3.4? This seems to be caused by a bug in 3.3.3 and 3.3.0. https://issues.apache.org/jira/browse/ZOOKEEPER-1115 Thanks, Jun On Thu, Nov 14, 2013 at 5:25 AM, Yu, Libo

RE: add partition tool in 0.8

Thanks for your reply, Joel. Regards, Libo -Original Message- From: Joel Koshy [mailto:jjkosh...@gmail.com] Sent: Thursday, November 07, 2013 5:00 PM To: users@kafka.apache.org Subject: Re: add partition tool in 0.8 kafka-add-partitions.sh is in 0.8 but not in 0.8-beta1. Therefore

RE: Purgatory

I read it and tried to understand it. It would be great to add a summary at the beginning about what it is and how it may impact a user. Regards, Libo -Original Message- From: Joel Koshy [mailto:jjkosh...@gmail.com] Sent: Friday, November 08, 2013 2:01 AM To: users@kafka.apache.org

add partition tool in 0.8

Hi team, Here is what I want to do: We are using 0.8-beta1 currently. We already have some topics and want to add partitions for them. kafka-add-partitions.sh is in 0.8 but not in 0.8-beta1. Therefore we cannot use this tool with 0.8-beta1. If I download latest 0.8 and compile it, can I use

question about default key

Hi team, If I don't specify a key when publishing a message, a default key will be generated. In this case, how long is the default key and will the consumer get this default key? Thanks. Libo

RE: question about default key

Got it. Thanks. Regards, Libo -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Thursday, October 24, 2013 10:09 AM To: users@kafka.apache.org Subject: Re: question about default key The default key is null. Thanks, Neha On Oct 24, 2013 6:47 AM, Yu, Libo

key partitioner

Hi team, According to the document, the default partitioner hashes the key string and assign the message to a broker. Could you give a brief introduction to the hash algorithm? If a long timestamp (in hex format) is used as key, will the messages be distributed evenly to all partitions?

KeyedMessage question

Hi team, For the message type in KeyedMessage, I can use String or byte[]. Is there any difference in terms of the actual data transferred? Regards, Libo

producer API thread safety

Hi team, Is it possible to use a single producer with more than one threads? I am not sure If its send() is thread safe. Regards, Libo

RE: Strategies for improving Consumer throughput

You can use a thread pool to write to hbase. And create another pool of consumer threads. Or add more consumer processes. The bottleneck is writing to Hbase in this case. Regards, Libo -Original Message- From: Graeme Wallace [mailto:graeme.wall...@farecompare.com] Sent: Wednesday,

bandwidth usage issue

Hi team, Here is a usage case: Assume each host in a kafka cluster a gigabit network adaptor. And the incoming traffic is 0.8gbps and at one point all the traffic goes to one host. The remaining bandwidth is not enough for the followers to replicate messages from this leader. To make sure no

RE: [jira] [Updated] (KAFKA-1046) Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

Hi team, Is it safe to apply the 0.8 patch to 0.8 beta1? Regards, Libo -Original Message- From: Joe Stein [mailto:crypt...@gmail.com] Sent: Friday, September 13, 2013 4:10 PM To: d...@kafka.apache.org; users@kafka.apache.org Subject: Re: [jira] [Updated] (KAFKA-1046) Added support for

RE: monitoring followers' lag

, September 10, 2013 8:48 PM To: users@kafka.apache.org Subject: Re: monitoring followers' lag It should be kafka.server:type=ReplicaFetcherManager,name=Replica-MaxLag - can you confirm and mind updating the wiki if this is the case? Thanks, Joel On Tue, Sep 10, 2013 at 10:38 AM, Yu, Libo libo

implicit default minimum retention size per partition is 4GB.

Hi team, For default broker configuration, replica.lag.max.messages is 4000 and message.max.bytes is 1Mb. In the extreme case, the follower(s) could lag by 4000 messages. The leader must save at least 4000 messages to allow follower(s) to catch up. So the minimum retention size is 4000Mb=4Gb.

RE: monitoring followers' lag

/documentation.html#monitoring ? Thanks, Jun On Tue, Sep 10, 2013 at 7:59 AM, Yu, Libo libo...@citi.com wrote: Hi team, I wonder if anybody can give detailed instructions on how to monitor the followers' lag by using JMX. Thanks. Regards, Libo

RE: monitoring followers' lag

, September 10, 2013 11:01 AM To: users@kafka.apache.org Subject: Re: monitoring followers' lag Have you looked at the updated docs in http://kafka.apache.org/documentation.html#monitoring ? Thanks, Jun On Tue, Sep 10, 2013 at 7:59 AM, Yu, Libo libo...@citi.com wrote: Hi team, I wonder

RE: is it possible to commit offsets on a per stream basis?

. Thanks, Neha On Mon, Sep 9, 2013 at 9:08 AM, Yu, Libo libo...@citi.com wrote: If one connector is used for a single stream, when there are many topics/streams, will that cause any performance issue, e.g. too many connections or too much memory or big latency? Regards, Libo

RE: zookeeper session time out

, Aug 28, 2013 at 1:09 PM, Yu, Libo libo...@citi.com wrote: Hi team, We notice when the incoming throughput is very high, the broker has to delete old log files to free up disk space. That caused some kind of blocking (latency) and frequently the broker's zookeeper session times out

is it possible to commit offsets on a per stream basis?

Hi team, This is our current use case: Assume there is a topic with multiple partitions. 1 Create a connector first and create multiple streams from the connector for a topic. 2 Create multiple threads, one for each stream. You can assume the thread's job is to save the message into the

RE: is it possible to commit offsets on a per stream basis?

at the same time? This is a better approach as there is no complex locking involved. Thanks, Neha On Thu, Aug 29, 2013 at 10:28 AM, Yu, Libo libo...@citi.com wrote: Hi team, This is our current use case: Assume there is a topic with multiple partitions. 1 Create a connector first and create

zookeeper session time out

Hi team, We notice when the incoming throughput is very high, the broker has to delete old log files to free up disk space. That caused some kind of blocking (latency) and frequently the broker's zookeeper session times out. Currently our zookeeper time out threshold is 4s. We can increase it.

Question about preferred replica leader election tool

Hi, We have three brokers in our kafka cluster. For all topics, the replica factor is two. Here is the distribution of leaders. After I ran the leader election tool, nothing happened. In this list, the first broker in ISR is the leader. I assume after running the tool, the first broker is

RE: questions about ISR

a large replica.lag.time.max.ms may delay the committing of a message. Thanks, Jun On Tue, Aug 27, 2013 at 6:37 AM, Yu, Libo libo...@citi.com wrote: Thanks, Jun. That is very helpful. However, I still have a couple of questions. We have a min fetch rate JMX in the broker. How to find out

RE: questions about ISR

ISR When a broker is restarted, it will automatically catch up from the leader and will join ISR when it's caught up. Are you not seeing this happening? Thanks, Jun On Fri, Aug 23, 2013 at 11:33 AM, Yu, Libo libo...@citi.com wrote: Hi, When a broker is not in a topic's ISR, will it try

RE: delete a topic

wrote: We currently don't have the delete topic feature in Kafka 0.8. So any manual attempts to do so might have a negative impact on functionality. Thanks, Neha On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo libo...@citi.com wrote: Hi

consumer question

Hi team, Right now, from a stream, an iterator can be obtained which has a blocking hasNext(). So what is the implementation behind the iterator? I assume there must be queue and the iterator monitors the queue. And a separate thread fetches data and feeds to the queue when it is almost empty.

RE: Producer message ordering problem

An auto-increment index can be assigned to a message as a key when it is being published. The consumer can monitor this index when receiving. If the expected message does not show up, buffer all received messages in a hashtable (use index as hash key) until it is received. Then handle all

RE: delete a topic

-1021 Thanks, Neha On Fri, Aug 23, 2013 at 6:34 AM, Yu, Libo libo...@citi.com wrote: Hi Neha, Wipe out the cluster Do you mean you uninstall the cluster and reinstall it? Or you just delete all kafka data and zookeeper data for the cluster? This is not a blocking issue for us. Our blocking

RE: KeeperErrorCode = BadVersion

ISR or electing a new leader for the same partition. Could you please file a JIRA to improve the quality of logging in this case? Thanks, Neha On Fri, Aug 23, 2013 at 10:28 AM, Yu, Libo libo...@citi.com wrote: Hi team, During normal operation, all of a sudden, we found many exceptions

questions about ISR

Hi, When a broker is not in a topic's ISR, will it try to catch up to go back to ISR itself? Or do we have to restart it? We can increase replica.lag.time.max.ms and replica.lag.max.messages to let brokers stay longer in ISR. Is that good practice? Still this is related to the first questions.

RE: delete a topic

-729 Thanks, Neha On Fri, Aug 23, 2013 at 10:52 AM, Yu, Libo libo...@citi.com wrote: I will give it a try. I know how to delete log files. But to delete the zookeeper data, do I only need to run the delete script? Regards, Libo -Original Message- From: Neha Narkhede

RE: ordering

] Sent: Thursday, August 22, 2013 12:01 AM To: users@kafka.apache.org Subject: Re: ordering Actually, I am not sure if I understand the trouble that you mentioned. Could you elaborate that a bit more? Thanks, Jun On Wed, Aug 21, 2013 at 12:30 PM, Yu, Libo libo...@citi.com wrote: Hi

delete a topic

Hi team, When I delete a topic, the topic is deleted from zookeeper but its log files are not deleted from Brokers. When I restart a broker, the broker will try to sync the log files whose topic has been deleted. Manually deleting the log files will resolve the issue. Should broker ignore

RE: issue with kafka-preferred-replica-election.sh

: https://issues.apache.org/jira/browse/KAFKA-1019 Guozhang On Wed, Aug 21, 2013 at 11:27 AM, Yu, Libo libo...@citi.com wrote: We never deleted it. Either it was never created or deleted somehow. Regards, Libo -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent

RE: delete a topic

0.8. So any manual attempts to do so might have a negative impact on functionality. Thanks, Neha On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo libo...@citi.com wrote: Hi team, When I delete a topic, the topic is deleted from zookeeper but its log files are not deleted from

partition does not exist (but it is there).

This is from the broker 3's log: [2013-08-22 15:40:02,984] WARN [KafkaApi-3] Fetch request: Partition [tes t.replica1.default,0] doesn't exist on 3 (kafka.server.KafkaApis) Here is what list topic command shows: topic: test.replica1.defaultpartition: 0leader: 3 replicas: 3 isr:

RE: issue with kafka-preferred-replica-election.sh

12:23 AM To: users@kafka.apache.org Subject: Re: issue with kafka-preferred-replica-election.sh Libo, Sorry for the late reply. I will file a JIRA for this one. Could you please provide the full process and environment to reproduce this issue? Guozhang On Tue, Aug 20, 2013 at 1:10 PM, Yu, Libo

mx4j-tools.jar missing

Hi team, I got the message from a broker log: [2013-08-21 08:22:55,502] INFO Will not load MX4J, mx4j-tools.jar is not in th e classpath (kafka.utils.Mx4jLoader$) Should it be in kafka_2.8.0-0.8.0-SNAPSHOT.jar or kafka-assembly-0.8.0-SNAPSHOT-deps.jar? Is that a packaging error with sbt?

broker never comes back to ISR

Hi team, We have three kafka brokers in a production cluster. We use replication factor 3 for all topics. We notice quite frequently one broker is not in isr. Sometimes after it is restarted, it will go back to isr. Sometimes even after it is restarted, it will not go back to isr. In today's

RE: issue with kafka-preferred-replica-election.sh

, Just want to clarify, in your case after you created your topic, the /brokers/topics/my_topic/partitions was never created or it was deleted somehow? Guozhang On Wed, Aug 21, 2013 at 7:25 AM, Yu, Libo libo...@citi.com wrote: For the path /brokers/topics/my_topic/partitions, if you remove

RE: broker never comes back to ISR

of the possibilities is the replica fetcher thread is somehow dead. Thanks, Neha On Aug 21, 2013 8:00 AM, Yu, Libo libo...@citi.com wrote: I checked the log of normal restart. The replication manager should start to handle leader and isr request after the server is up. What may stop it from doing

RE: How to get broker back to ISR

Hi Neha, Which broker is controller broker and how is it defined? Regards, Libo -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Tuesday, August 20, 2013 10:56 AM To: users@kafka.apache.org Subject: Re: How to get broker back to ISR Once the broker is

ordering

Hi, This is from kafka faq: * Each partition is not consumed by more than one consumer thread/process in each consumer group. This allows to have each process consume in a single threaded fashion to guarantee ordering to the consumer within the partition (if we split up a partition of

controller data

Hi team, I found this line from the log: [2013-08-21 08:22:55,513] INFO conflict in /controller data: 2 stored data: 3 (kafka.utils.ZkUtils$) I checked the zookeeper and the data in /controller is 3. We have 3 brokers in our cluster. The broker ids are 0,1,2 or 1,2,3? Regards, Libo

RE: How to get broker back to ISR

to /123.456.789. (kafka.network.Processor) .. Regards, Libo -Original Message- From: Yu, Libo [ICG-IT] Sent: Wednesday, August 21, 2013 3:15 PM To: 'users@kafka.apache.org' Subject: RE: How to get broker back to ISR Hi Neha, Which broker is controller broker and how is it defined

issue with kafka-preferred-replica-election.sh

Hi, I tried to run kafka-preferred-replica-election.sh on our kafka cluster. But I got this expection: Failed to start preferred replica election org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for

RE: issue with kafka-preferred-replica-election.sh

-preferred-replica-election.sh have you already created the topic uattoqaaa.default with the number of partitions? Guozhang On Tue, Aug 20, 2013 at 8:35 AM, Yu, Libo libo...@citi.com wrote: Hi, I tried to run kafka-preferred-replica-election.sh on our kafka cluster. But I got this expection

RE: issue with kafka-preferred-replica-election.sh

the create topic tool, it might not be created yet. Guozhang On Tue, Aug 20, 2013 at 10:24 AM, Yu, Libo libo...@citi.com wrote: It is created with 3 partitions. Regards, Libo -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: Tuesday, August 20, 2013 12:21

RE: issue with kafka-preferred-replica-election.sh

to partitions. If you start the election command right after the create topic tool, it might not be created yet. Guozhang On Tue, Aug 20, 2013 at 10:24 AM, Yu, Libo libo...@citi.com wrote: It is created with 3 partitions. Regards, Libo -Original Message- From: Guozhang Wang

multi-subscription

Hi, In Jun's presentation about kafka, multi-subscription is mentioned as a feature of kafka. Do that mean one consumer can consume multiple partitions on different brokers? Thanks. Regards, Libo

cluster reboot and leader selection

Hi, We have three hosts (1, 2, 3) in a kafka cluster. When we restart the cluster, If broker on host 2 restarts first, then most of partitions will use broker 2 as leader. Ideally, we want to see one third topics use broker 1 as leader, other one third topics use broker 2 and the remaining one

java version for kafka clients

Hi We have integrated kafka consumer and producer into our java application. We've noticed some issues when loading classes. And it seems it is caused by different JDK versions. So I wonder which version of JDK is recommended for developing kafka clients. Regards, Libo

RE: Changing the number of partitions after a topic is created

Hi Jun, Is it possible to specify a different partition number for each topic in the server.properties for 0.8? There is no documentation about that in the file. Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Friday, July 05, 2013 12:06 AM To:

RE: PID files for monitoring two different instances of Kafka on same machine

On linux, echo $! pid.txt will save the PID of the previous background job to a file. If you run java to launch kafka directly rather than using the provided scripts, it can save process pid into the file. Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com]

RE: Changing the number of partitions after a topic is created

the # of partitions when you create a topic using the create-topic command line tool. Thanks, Jun On Fri, Jul 5, 2013 at 6:45 AM, Yu, Libo libo...@citi.com wrote: Hi Jun, Is it possible to specify a different partition number for each topic in the server.properties for 0.8? There is no documentation

RE: [ANNOUNCEMENT] Apache Kafka 0.8.0-beta1 Released

-l Then you should see the following tag. 0.8.0-beta1 You can then checkout from the tag and see the commits. Thanks, Jun On Wed, Jul 3, 2013 at 3:02 PM, Yu, Libo libo...@citi.com wrote: Hi, I am trying to find out the main difference between the BETA1 and the 0.8 version we are testing

check out 0.8 beta from git repo

Hi, I want to check out beta version from git repo. Which branch should I use? Thank.s origin/0.8 origin/0.8.0-beta1-candidate1 Regards, Libo

partition numbers and mirrormaker

Hi I asked this question previously and didn't get an answer. If a topic has N partitions on source cluster, after being mirrored, on the destination cluster it has only one partition. To avoid this issue, before mirroring, I create the same topic with N partitions on the Destination cluster.

RE: how brokers sync with each other

means wait until all replicas in ISR have received the message) it would slow down your producer considerably. (You could use acks = 1 in that case though but with weaker guarantees - i.e., your message could be lost in failure scenarios). Joel On Fri, Jun 28, 2013 at 5:41 PM, Yu, Libo libo

RE: failover strategy

is disjoint from the set of topics you are mirroring from B-A (i.e., to avoid a mirroring loop). Joel On Fri, Jun 28, 2013 at 5:29 PM, Yu, Libo libo...@citi.com wrote: Hi, I can think of two failover strategies. I am not sure which one is the right way to go. First method. set up kafka

Is it possible to get latest offset from kafka server?

Hi, I know kafka.tools.ConsumerOffsetChecker can find out the latest offset for a topic from log files internally. Is there any public API that users can use for that? This would be useful for the failover scenario. When one cluster is down, the publisher will publish to the backup cluster. But

RE: out of disk space

Thanks for your answer, Jun. That explains what I found. I thought it was for the machine. If there are many consumers (as in our case), that number is determined by the most productive consumer. I would prefer a limit for the machine. Regards, Libo From: Yu, Libo [ICG-IT] Sent: Monday, June 10

RE: rebalance issues from mirrormaker

: Yu, Libo [ICG-IT] Sent: Monday, June 10, 2013 11:21 AM To: 'users@kafka.apache.org' Subject: rebalance issues from mirrormaker Hi, I come across several critical issues when benchmarking mirrormakers. 1 If a topic has N partitions on source cluster, after mirroring, on the destination cluster

  1   2   >