If you look at the example simple consumer:
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
You'll see:
if (currentOffset readOffset) {
System.out.println(Found an old offset: + currentOffset +
Expecting: + readOffset);
continue;
}
and a
Hi Jay,
Do they record these meetups?
Thanks,
Chris
On Thu, Oct 17, 2013 at 5:03 PM, Jay Kreps jay.kr...@gmail.com wrote:
FYI.
-- Forwarded message --
From: Jakob Homan jgho...@gmail.com
Date: Thu, Oct 17, 2013 at 11:08 AM
Subject: Special Bay Area HUG: Tajo and Samza
Thanks Ian.
Is your consumer multi-threaded? If so can you share how you coordinated
each of the threads so you knew it was 'okay' to commit across all the
threads? I'm stuck on how to do this without really complicating the
consumer.
Thanks,
Chris
On Tue, Jul 9, 2013 at 5:51 PM, Ian Friedman
Hi,
I'm working through a production-level High Level Consumer app and have a
couple of error/shutdown questions to understand how the offset storage is
handled.
Test case - simulate an error writing to destination application, for
example a database, offset is 'lost'
Scenario
- write 500
-commit updates the offsets in
Zookeeper, you will drop that message.
The solution to this to call commitOffsets() explicitly.
Philip
On Tue, Jul 9, 2013 at 11:16 AM, Chris Curtin curtin.ch...@gmail.com
wrote:
Hi,
I'm working through a production-level High Level Consumer app and have
the
number of partitions out there). That way commitOffsets() will actually
only apply to that partition. Auto-commit the same way. It might give you
the level of control you need.
Philip
On Tue, Jul 9, 2013 at 2:22 PM, Chris Curtin curtin.ch...@gmail.com
wrote:
Hi Philip,
Correct, I
Enhancement submitted: https://issues.apache.org/jira/browse/KAFKA-966
On Tue, Jul 9, 2013 at 3:53 PM, Chris Curtin curtin.ch...@gmail.com wrote:
Thanks. I know I can write a SimpleConsumer to do this, but it feels like
the High Level consumer is _so_ close to being robust enough
I'd look at a variation of #2. Can your messages by grouped into a 'class
(for lack of a better term)' that are consumed together? For example a
'class' of 'auditing events' or 'sensor events'. The idea would to then
have a topic for 'class'.
A couple of benefits to this:
- you can define your
.
thanks,
rob
From: Chris Curtin [curtin.ch...@gmail.com]
Sent: Wednesday, May 29, 2013 7:36 AM
To: users
Subject: Re: one consumerConnector or many?
I'd look at a variation of #2. Can your messages by grouped into a 'class
(for lack of a better term
Hi Tim,
On Wed, May 22, 2013 at 3:25 PM, Timothy Chen tnac...@gmail.com wrote:
Hi,
I'm currently trying to understand how Kafka (0.8) can scale with our usage
pattern and how to setup the partitioning.
We want to route the same messages belonging to the same id to the same
queue, so its
Yes. However be aware that starting and stopping processes will cause a
rebalance of the consumers, so your code may find itself receiving events
from a different partition suddenly (so don't assume the partition you are
reading isn't going to change!) Also as things are starting up you may
find
I'll try to answer some, the Kafka team will need to answer the others:
On Wed, May 8, 2013 at 12:17 PM, Yu, Libo libo...@citi.com wrote:
Hi,
I read this link
https://cwiki.apache.org/KAFKA/consumer-group-example.html
and have a few questions (if not too many).
1 When you say the
= metadata.broker.list).
Jun
On Mon, Apr 29, 2013 at 11:19 AM, Chris Curtin curtin.ch...@gmail.com
wrote:
Thanks, I missed that the addition of consumers can cause a re-balance.
Thought it was only on Leader changes.
I've updated the wording in the example.
I'll pull down the beta and test my
the wiki with customized
encoder (Producer) and decoder (Consumer) at some point.
Thanks,
Jun
On Wed, May 1, 2013 at 6:44 AM, Chris Curtin curtin.ch...@gmail.com
wrote:
I've tested my examples with the new (4/30) release and they work, so
I've
updated the documentation.
Thanks
Hi,
I pulled 0.8.0 head at 3 pm eastern 4/30, did the sbt update; sbt package;
sbt assembly-package-dependency and my code won't compile.
It is missing the Yammer Metrics libraries.
Last pull was about 45 days ago and they were stored in core\lib\metrics-*
Do I now need to pull them myself?
Just added the High Level Consumer example.
On Mon, Apr 29, 2013 at 1:52 AM, Jun Rao jun...@gmail.com wrote:
We have updated the 0.8 documentation in our website (
http://kafka.apache.org/index.html). Please review the docs. We have the
following blockers for the 0.8 beta release:
are changed to
zookeeper.connect,
zookeeper.session.timeout.ms, and zookeeper.sync.time.ms, respectively.
I will add a link to your wiki in our website.
Thanks again.
Jun
On Mon, Apr 29, 2013 at 5:54 AM, Chris Curtin curtin.ch...@gmail.com
wrote:
Hi Jun,
I finished and published
rebalancing algorithm
section of http://kafka.apache.org/07/design.html
Thanks,
Jun
On Mon, Apr 29, 2013 at 8:16 AM, Chris Curtin curtin.ch...@gmail.com
wrote:
Jun, can you explain this a little better? I thought when using Consumer
Groups that on startup Kafka connects to ZooKeeper
In a nutshell: High Level uses Consumer Groups to handle the tracking of
message offset consumption. SimpleConsumer leaves it all up to you.
The 0.7.x quick start shows examples of both:
http://kafka.apache.org/quickstart.html
On Fri, Apr 26, 2013 at 12:32 PM, Oleg Ruchovets
Did you create the topic without a # of partitions then try to
delete/recreate it? I've had that happen to me before. Try shutting down
everything (including zookeeper) and restarting.
On Tue, Apr 23, 2013 at 9:08 PM, Jun Rao jun...@gmail.com wrote:
Does this happen on every message that you
I following these instructions to get the first 'play' cluster going:
https://cwiki.apache.org/KAFKA/kafka-08-quick-start.html
Instead of running the 3 brokers on the same machine, I ran on on each
machine.
Note that you will need to do a little bit of zookeeper setup to get a
cluster running,
, that
appears to be using one instance of zookeeper, which makes whichever
server
that runs zookeeper as the single point of failure?
Jason
On Tue, Apr 23, 2013 at 8:28 AM, Chris Curtin
curtin.ch...@gmail.comwrote:
I following these instructions to get the first 'play' cluster
Also keep in mind that anything done at the transport (SSL for example)
layer won't solve your 'at rest' problems.
All messages are written to disk, so unless the broker does some encryption
logic you haven't solved the data visibility issues.
I also think this should be a producer/consumer
for the
0.8 high level consumer?
Thanks,
Jun
On Fri, Mar 29, 2013 at 8:28 AM, Chris Curtin curtin.ch...@gmail.com
wrote:
Hi,
I've added an example program for using a SimpleConsumer for 0.8.0. Turns
out to be a little more complicated once you add Broker failover. I'm not
100% thrilled
You need to reference the version of Yammer shipping with Kafka. It is
under \core\lib\metrics-*
On Thu, Apr 4, 2013 at 11:41 AM, Oleg Ruchovets oruchov...@gmail.comwrote:
I am executing a simple code like this:
public class FirstKafkaTester {
public ProducerInteger, String
Now with Video:
http://vimeo.com/63040812
(I did notice that I misspoke about reading from replicas, sorry).
On Wed, Mar 20, 2013 at 8:11 AM, Chris Curtin curtin.ch...@gmail.comwrote:
Hi,
It went really well last night. Lots of good questions. Here are the
slides, and hopefully the video
://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
Thanks,
Chris
On Mon, Mar 25, 2013 at 8:02 AM, Chris Curtin curtin.ch...@gmail.comwrote:
Hi David,
Thanks for the feedback. I've seen the example before and after in
different books/articles and it doesn't matter to me.
Anyone
Thanks for finding those. Looks like a copy and paste issue. I've updated
the document.
Thanks,
Chris
On Sat, Mar 23, 2013 at 11:27 AM, Jonathan Hodges hodg...@gmail.com wrote:
Many thanks for contributing! The docs are very helpful. I found a couple
small possible typos. The
Hi David,
Thanks for the feedback. I've seen the example before and after in
different books/articles and it doesn't matter to me.
Anyone else want to help define a style guide or is there one I didn't see
already?
Thanks,
Chris
On Thu, Mar 21, 2013 at 7:46 PM, David Arthur mum...@gmail.com
Okay, how do we do this logistically? I've take the Producer code that I
wrote for testing purposes and wrote a description around it. How do I get
it to you guys?
Simple Consumer is going to take a little longer since my test Consumers
are non-trivial and I'll need to simplify them.
Thanks,
Hi Jun,
I've been thinking for a while about how to contribute to the project and
thought that working on some documentation for the website might be a good
way. Do you have an outline of what you'd like the site to look like that I
(AND OTHERS hint, hint) could pick a topic, write the article
Hi,
I noticed the same thing. In 0.8.0 the offset passed to the fetch is where
you want to start, not where you left off. So the last offset read from the
previous batch is truly the 'last offset' so you need to save it and ask
for it +1. Otherwise you keep asking for that last offset, which is
Hi,
0.8.0 HEAD from 3/4/2013.
As I think through building a robust SimpleConsumer I ran some failure
tests today and want to make sure I understand what is going on.
FYI I know that I should be doing a metadata lookup to find the leader, but
I wanted to see what happens if things are going well
http://www.twitter.com/allthingshadoop
*/
On Tue, Mar 5, 2013 at 8:30 AM, Chris Curtin curtin.ch...@gmail.com
wrote:
Hi Jun,
I wasn't explicitly setting the ack anywhere.
Am I reading the code correctly that in SyncProducerConfig.scala the
DefaultRequiredAcks is 0? Thus not waiting
again.
Also, I recommend that you try the latest code in 0.8. A bunch of issues
have been fixes since Jan. You will have to wipe out all your ZK and Kafka
data first though.
Thanks,
Jun
On Mon, Mar 4, 2013 at 8:32 AM, Chris Curtin curtin.ch...@gmail.com
wrote:
Hi,
(Hmm, take 2
Hi,
I was able to implement my own lookup code but have a few concerns about
this long term:
- the Broker class is marked as 'private' in the Scala code. IntelliJ gives
me an error about using it, but the runtime lets me use it and get the
host/port out.
- I have to know a lot about the structure
36 matches
Mail list logo
