Have there been any changes with 0.8.2 in how the marker gets moved when you use the high-level consumer?
One problem I have always had was: what if I pull something from the stream, but then I have an error in processing it? I don't really want to move the marker. I would almost like the client to have a callback mechanism for processing, and the marker only gets moved if the high level consumer successfully implements my callback/processor (with no exceptions, at least). On Sun, Feb 8, 2015 at 9:49 AM, Gwen Shapira <gshap...@cloudera.com> wrote: > On Sun, Feb 8, 2015 at 6:39 AM, Christopher Piggott <cpigg...@gmail.com> > wrote: > > > > The consumer used Zookeeper to store offsets, in 0.8.2 there's an > option > > to use Kafka itself for that (by setting *offsets.storage = kafka > > > > Does it still really live in zookeeper, and kafka is proxying the > requests > > through? > > > > > They don't live in Zookeeper. They live in a secret Kafka topic (__offsets > or something like that). > > For migration purposes, you can set dual.commit.enable = true and then > offsets will be stored in both Kafka and ZK, but the intention is to > migrate to 100% Kafka storage. > > > > > On Sun, Feb 8, 2015 at 9:25 AM, Gwen Shapira <gshap...@cloudera.com> > > wrote: > > > > > Hi Eduardo, > > > > > > 1. "Why sometimes the applications prefer to connect to zookeeper > instead > > > brokers?" > > > > > > I assume you are talking about the clients and some of our tools? > > > These are parts of an older design and we are actively working on > fixing > > > this. The consumer used Zookeeper to store offsets, in 0.8.2 there's an > > > option to use Kafka itself for that (by setting *offsets.storage = > > kafka*). > > > We are planning on fixing the tools in 0.9, but obviously they are less > > > performance sensitive than the consumers. > > > > > > 2. Regarding your tests and disk usage - I'm not sure exactly what > fills > > > your disk - if its the kafka transaction logs (i.e. log.dir), then we > > > expect to store the size of all messages sent times the replication > > faction > > > configured for each topic. We keep messages for the amount of time > > > specified in *log.retention* parameters. If the disk is filled within > > > minutes, either set log.retention.minutes very low (at risk of losing > > data > > > if consumers need restart), or make sure your disk capacity matches the > > > rates in which producers send data. > > > > > > Gwen > > > > > > > > > On Sat, Feb 7, 2015 at 3:01 AM, Eduardo Costa Alfaia < > > > e.costaalf...@unibs.it > > > > wrote: > > > > > > > Hi Guys, > > > > > > > > I have some doubts about the Kafka, the first is Why sometimes the > > > > applications prefer to connect to zookeeper instead brokers? > Connecting > > > to > > > > zookeeper could create an overhead, because we are inserting other > > > element > > > > between producer and consumer. Another question is about the > > information > > > > sent by producer, in my tests the producer send the messages to > brokers > > > and > > > > a few minutes my HardDisk is full (my harddisk has 250GB), is there > > > > something to do in the configuration to minimize this? > > > > > > > > Thanks > > > > -- > > > > Informativa sulla Privacy: http://www.unibs.it/node/8155 > > > > > > > > > >
