Re: expanding cluster and reassigning parititions without restarting producer

No I don't see anything like that, the question was aimed at learning if it is worthwhile to make the effort of reimplementing the Python producer in Java, I so I will not make all the effort just to be disappointed afterwards. understand I have nothing to worry about, so I will try to simulate thi

Re: consumer ack for high-level consumer?

Got it! Thanks for your response. 2014-11-07 13:14 GMT+08:00 Guozhang Wang : > 0. Yes, if consumer crashed before commit its offset it can cause > duplicates. > > 1. Yes, since from the consumer client's point of view, once the message is > returned from the iterator it is considered as "consum

Re: One question about "New Producer Configs"

Dear Genlong, "New Producer Configs" is for upcoming versions. If you are using 0.8.1, please refer to "Producer Configs". acks=n, in which n means number of in-sync replicas, not number of partitions. regards, Chia-Chun 2014-11-10 10:51 GMT+08:00 hi <995174...@qq.com>: > Dear sir or madam, >

Re: Consumer thread dies

If C1 dies, C2 will be owning that partition. However, C1 has to really die, which typically means that either you close the consumer connector or the jvm of C1 is gone. In your case, it seems that C1 didn't die, it just hung. Do you know why C1 hung? Thanks, Jun On Fri, Nov 7, 2014 at 3:34 PM,

Re: Add partitions with replica assignment in same command

Yes, it seems that we need to fix the tool to support that. It's probably more intuitive to have TopicCommand just take the replica-assignment (for the new partitions) when altering a topic. Could you file a jira? Thanks, Jun On Fri, Nov 7, 2014 at 4:17 PM, Allen Wang wrote: > I am trying to f

One question about "New Producer Configs"

Dear sir or madam, There is one question when I'm using Kafka: From the "documentation:New Producer Configs" ,I get that acks=n means the number of acknowledgments the producer requires the leader to have received before considering a request complete. But why can the producer send mess

Re: Rebalance not happening even after increasing max retries causing conflict in ZK

A rebalance should trigger on all consumers when you add a new consumer to the group. If you don't see the zookeeper watch fire, the consumer may have somehow lost the watch. We have seen this behavior on older zk versions, I wonder it that bug got reintroduced. To verify if this is the case, you c

Re: Interrupting controlled shutdown breaks Kafka cluster

Solon, You may be hitting this: KAFKA-1305 . You can try with 0.8.2-beta version and see if this issue is re-producible. Guozhang On Fri, Nov 7, 2014 at 11:52 AM, Solon Gordon wrote: > We're using 0.8.1.1 with auto.leader.rebalance.enable=true.

Re: Interrupting controlled shutdown breaks Kafka cluster

We fixed a couple issues related to automatic leader balancing and controlled shutdown. Would you mind trying out 0.8.2-beta? On Fri, Nov 7, 2014 at 11:52 AM, Solon Gordon wrote: > We're using 0.8.1.1 with auto.leader.rebalance.enable=true. > > On Fri, Nov 7, 2014 at 2:35 PM, Guozhang Wang wrot

Re: expanding cluster and reassigning parititions without restarting producer

The producer might get an error code if the leader of the partitions being reassigned also changes. However it should retry and succeed. Do you see a behavior that suggests otherwise? On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan wrote: > Hi All, > I recently had an issue producing from python w

Re: corrupt recovery checkpoint file issue....

Guozhang, In OffsetCheckpoint.write(), we don't catch any exceptions. There is only a finally clause to close the writer. So, it there is any exception during write or sync, the exception will be propagated back to the caller and swapping will be skipped. Thanks, Jun On Fri, Nov 7, 2014 at 9:47

Re: Adding replicas to existing topic cause data loss in some partitions

Any error in the controller/state-change log when you increased the replication factor? If you describe those topics, are both replicas in ISR? The answers to those questions will help us understand whether this is a broker side or consumer-side issue. Thanks, Jun On Thu, Nov 6, 2014 at 11:56 PM

Re: kafka test jars in sbt?

For sbt, you need to use something like: "org.apache.kafka" %% "kafka" %"0.8.2-beta" % "test" classifier "test" That tells sbt to pull in the kafka artifact with the "test" classifier only when running tests. The %% tells sbt to fill in the scala version (so it'll map to "kafka_2.10" like in your

Re: powered by kafka

I'm not Jay, but fixed it anyways ;) Gwen On Sun, Nov 9, 2014 at 10:34 AM, vipul jhawar wrote: > Hi Jay > > Thanks for posting the update. > > However, i checked the page history and the hyperlink is pointing to the > wrong domain. > Exponential refers to www.exponential.com. I sent the twitter

Re: powered by kafka

Hi Jay Thanks for posting the update. However, i checked the page history and the hyperlink is pointing to the wrong domain. Exponential refers to www.exponential.com. I sent the twitter handle, should have sent the domain. Please correct. Thanks On Sat, Nov 8, 2014 at 3:45 PM, vipul jhawar wr

Re: Consumer thread dies

Hi, Further I looked at this scenario, Is it correct that above scenario can be handled if I use SimpleConsumer approach instead of using "ConsumerConnector.createMessageStreams()" , this way I have better control on partition. This way my partition is not bound with any specific consumer thread.

Re: powered by kafka

Updated. Thanks! On Sat, Nov 8, 2014 at 12:16 PM, Jimmy John wrote: > Livefyre (http://web.livefyre.com/) uses kafka for the real time > notifications, analytics pipeline and as the primary mechanism for general > pub/sub. > > thx... > jim > > On Sat, Nov 8, 2014 at 7:41 AM, Gwen Shapira > wrot

Re: Issues Running Kafka Producer Java example

The producer code here looks fine. It may be an issue with the consumer, or how the consumer is used. If you are running the producer before starting a consumer, make sure you get all messages by setting auto.offset.reset=smallest (in the console consumer you can use --from-beginning) Also, you c