Re: Difference between storing offset on Kafka and on Zookeeper server?

Are there plans to move consumer group coordination off of Zk as well? And if so:-- what's the approximate planned release for that?-- and what dependencies on Zk will be left in Kafka after that? thanks!Marina From: Ewen Cheslack-Postava To: "users@kafka.apache.org"

Offset Management in Kafka

Hi I am looking for offset management in Kafka. Is there any link which can explain how offset are updated in Kafka once a consumer read a message.

Re: Urban Airship Connect

Hi, Great info. What is your approximate/average message size ? And, how many topics and partitions do you use in your 4-node cluster. Also, please share your hardware specs Thanks, Prabhjot On Sat, Oct 10, 2015 at 2:00 AM, Cory Kolbeck wrote: > The first level Kafkas are

Re: How Kafka work to send message to consumer?

Yes you are right. On Wed, Oct 14, 2015 at 10:38 PM, Kiran Singh wrote: > So it's consumer responsibility to keep on checking whether a new data > available on Kafka server or not. > Am I right? > > On Thu, Oct 15, 2015 at 10:21 AM, Mayuresh Gharat < >

Re: Difference between storing offset on Kafka and on Zookeeper server?

There are a couple of advantages. First, scalability. Writes to Kafka are cheaper than writes to ZK. Kafka-based offset storage is going to be able to handle significantly more consumers (and scale out as needed since writes are spread across all partitions in the offsets topic). Second, once you

Re: Offset Management in Kafka

This might help : https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka http://www.slideshare.net/jjkoshy/offset-management-in-kafka Thanks, Mayuresh On Thu, Oct 15, 2015 at 5:39 AM, Kiran Singh wrote: > Hi > > I am

Time synchronicity prerequisites

Does Kafka rely on synchronized server clocks? Or with other words: Will Kafka/Zookeeper work reliable when server clocks are not synchronized (with that i mean the clocks are permanently out of sync and difference is in the magnitude of hours) Thanks Hendrik -- Hendrik Saly (salyh,

Getting error code 15 while connect to the offset manager.

Hi I am trying to implement fetch and commit offset API. But i am getting error code 15 "ConsumerCoordinatorNotAvailableCode" while connecting to kafka. Can any one explain this error? Thanks and Regards Kiran Singh

Re: GC is running forever

We had an issue that sounds somewhat similar. It impacted many long-running java apps but Kafka more than most. In our case, it actually turned out to be a kernel bug. Here is a reference with more details: https://groups.google.com/forum/#!topic/mechanical-sympathy/QbmpZxp6C64 Thanks, Paul

Thread safe way of creating a topic programmatically

Hi, # I have a use case where multiple app-servers would try to create same topic. In the past it was mentioned that "AdminUtils.createTopic" is not thread safe to create a topic, could some one confirm if it's still the case?. (or) One of the safest approach is all the app-serves would go to DB

Re: How to get group list in kafka?

Hello Kiran, kafka-consumer-groups.sh will be released as part of 0.9.0.0. What version of Kafka are you using? Info on ConsumerGroups is persisted in Zookeeper and you can write a simple tool like here to get this info. ​ On Thu, Oct 15, 2015 at

How are people dealing with time and detecting delayed messages?

I hope this question makes sense, I’m kinda a newbie when it comes to reasoning about distributed systems. Let’s say I have a consumer that needs to be able to detect when a given message was delayed by some period of time (could be due to network partition or producer errors or whatever). By

GC is running forever

Hello Kafka Experts. We have experienced a long running GC problem for a while. GC will eventually eat up all the cpu cycles on the physical box. Have chatted with Gwen regarding this issue during recent ended Hadoop conference at NYC, per her recommendation we switched to G1, but we still see

Re: How to get group list in kafka?

Hi Ashish I am using kafka_2.10-0.8.2.2.jar On Fri, Oct 16, 2015 at 9:25 AM, Ashish Singh wrote: > Hello Kiran, > > kafka-consumer-groups.sh will be released as part of 0.9.0.0. > > What version of Kafka are you using? Info on ConsumerGroups is persisted in > Zookeeper and

Re: Getting error code 15 while connect to the offset manager.

Hi Mayuresh I am following link https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka for implementing commit and fech offset API and i am using kafka_2.10-0.8.2.2 jar file. On Thu, Oct 15, 2015 at 9:40 PM, Mayuresh Gharat

Offset Commit and fetch API.

Hi I am little bit confused in regards to offset. Please correct me if i am wrong any where. Note: I am using Java API only for most of these operation. 1. By default offset store in Zookeeper for a topic? Is there any setting in zookeeper/broker/topic related to same? 2. If a offset is stored

("offsets.storage","kafka") not working with Consumer API

Hello I am trying to use ConsumerConnector API. But if i used ("offsets.storage","kafka") property "createJavaConsumerConnector" function will hang. And I removed this, API work fine. I am using kafka_2.10-0.8.2.2.jar Is there any issue or I am missing any thing. Thanks Kiran Singh

Re: New consumer client compatible with old broker

> On Oct 15, 2015, at 11:29 AM, tao xiao wrote: > > Hi team, > > Does new consumer client (the one in trunk) work with 0.8.2.x broker? I am > planning to use the new consumer in our development but don't want to > upgrade the broker to the latest. is it possible to do that?