Consumer Events Problem

2013-12-09 Thread Sanket Maru
I am working on a small project and discovered that our consumer hasn't been executed for over a month now. How can i check the unprocessed events ? From which date the events are available and what is the retention policy on the producer side ? Thanks, Sanket Maru

Re: Consumer Events Problem

2013-12-09 Thread Philip O'Toole
What version are you running? Philip On Mon, Dec 9, 2013 at 4:30 AM, Sanket Maru san...@ecinity.com wrote: I am working on a small project and discovered that our consumer hasn't been executed for over a month now. How can i check the unprocessed events ? From which date the events are

Re: Consumer Events Problem

2013-12-09 Thread Sanket Maru
I am using kafka 0.8.0 On Mon, Dec 9, 2013 at 6:09 PM, Philip O'Toole phi...@loggly.com wrote: What version are you running? Philip On Mon, Dec 9, 2013 at 4:30 AM, Sanket Maru san...@ecinity.com wrote: I am working on a small project and discovered that our consumer hasn't been

Re: Consumer Events Problem

2013-12-09 Thread Philip O'Toole
OK, I am only familiar with 0.72. Philip On Mon, Dec 9, 2013 at 4:54 AM, Sanket Maru san...@ecinity.com wrote: I am using kafka 0.8.0 On Mon, Dec 9, 2013 at 6:09 PM, Philip O'Toole phi...@loggly.com wrote: What version are you running? Philip On Mon, Dec 9, 2013 at 4:30 AM,

Re: How do you keep track of offset in a partition

2013-12-09 Thread Bhargav
Hi, How to know where to start consuming from, if we have already consumed a few messages. Some where in between latest and the earliest? How to identify the timestamp or offset from the api. Thanks, Bhargav --

Re: Consumer Events Problem

2013-12-09 Thread Jun Rao
By default, each topic is kept on the broker for 7 days. Older data, whether consumed or not, will be deleted. To check # of unconsumed messages, you can either use the ConsumerOffsetChecker tool (see https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F)

Re: How do you keep track of offset in a partition

2013-12-09 Thread Jun Rao
Consumer offsets are typically checkpointed into ZK. To check # of unconsumed messages, you can either use the ConsumerOffsetChecker tool (see https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F) or use the maxlag jmx in the consumer application (see

Re: Partition reassignment stucked

2013-12-09 Thread Maxime Nay
Hi, We used the code checked in this branch a few hours before the official 0.8.0 final release : https://github.com/apache/kafka/tree/0.8 So hopefully it should be the exact same code as the official release. The controller logs are empty. In a previous exchange you advised us to not use trunk

Re: Partition reassignment stucked

2013-12-09 Thread Neha Narkhede
I wouldn't call 0.8.1 production ready just yet. We are still in the process of deploying it at LinkedIn. Until it is ready, there isn't a good cluster expansion solution other than spinning up a new cluster. This is probably a little easier if you have a VIP in front of your kafka cluster.

Re: Partition reassignment stucked

2013-12-09 Thread Maxime Nay
Ok, thanks for your help. When 0.8.1 will be production ready, will you announce it somewhere (will you release it right away) ? Thanks, Maxime 2013/12/9 Neha Narkhede neha.narkh...@gmail.com I wouldn't call 0.8.1 production ready just yet. We are still in the process of deploying it at

Re: storing last processed offset, recovery of failed message processing etc.

2013-12-09 Thread S Ahmed
Say am I doing this, a scenerio that I just came up with that demonstrates #2. Someone signs up on a website, and you have to: 1. create the user profile 2. send email confirmation email 3. resize avatar Now once a person registers on a website, I write a message to Kafka. Now I have 3

Re: storing last processed offset, recovery of failed message processing etc.

2013-12-09 Thread Benjamin Black
You might look at Curator http://curator.apache.org/ On Mon, Dec 9, 2013 at 12:36 PM, S Ahmed sahmed1...@gmail.com wrote: Say am I doing this, a scenerio that I just came up with that demonstrates #2. Someone signs up on a website, and you have to: 1. create the user profile 2. send

Migration Process from Kafka 0.7 to 0.8

2013-12-09 Thread Arnaud Lawson
Hi, I am working on a migration plan - moving from kafka 0.7 to kafka 0.8 - and came across the webpage below that was last edited on April 26, 2013. Is this the most up-to-date information page regarding the migration process? And are there other useful webpages?

Re: Partition reassignment stucked

2013-12-09 Thread Neha Narkhede
We will announce it on this mailing list. It is probably a month away from a release. Thanks, Neha On Mon, Dec 9, 2013 at 12:02 PM, Maxime Nay maxime...@gmail.com wrote: Ok, thanks for your help. When 0.8.1 will be production ready, will you announce it somewhere (will you release it right

Presentation on our use of Kafka at Flurry

2013-12-09 Thread Ian Friedman
Hello fellow kafka users, I just wanted to share a presentation one of my colleagues made at SK Planet's Tech Planet conference. It's about how Flurry transitioned from a map-reduce batching data processing system to using Kafka to do continuous processing in our data ingestion pipeline. I

Re: Partition reassignment stucked

2013-12-09 Thread Maxime Nay
That's good to know. Thanks for your help! 2013/12/9 Neha Narkhede neha.narkh...@gmail.com We will announce it on this mailing list. It is probably a month away from a release. Thanks, Neha On Mon, Dec 9, 2013 at 12:02 PM, Maxime Nay maxime...@gmail.com wrote: Ok, thanks for your

Re: Consumer Events Problem

2013-12-09 Thread Sanket Maru
For mine topic the offset isn't increasing which means the consumer has stopped. I wanted to get the count(#) of events that are still remaining to be processed. Is that possible ? On Mon, Dec 9, 2013 at 9:44 PM, Jun Rao jun...@gmail.com wrote: By default, each topic is kept on the broker for