RE: repeated for request offset out of range

Can you tell us how to reproduce this behavior? Thanks, Neha On Aug 23, 2013 4:44 AM, "Wu, Jiang2" wrote: > We are using high-level API, not the SimpleConsumer. > > Thanks, > Jiang > > -Original Message- > From: Jun Rao [mailto:jun...@gmail.com] > Sent: Friday, August 23, 2013 12:13 AM >

Re: Questions about the leader election

The replication state machine and leader election mechanism is described here - http://kafka.apache.org/documentation.html#replication Let us know how the docs can be improved. Thanks, Neha On Thu, Aug 22, 2013 at 8:51 PM, James Wu wrote: > Hi, > > I am wondering what is the mechanism that Ka

Re: partition does not exist (but it is there).

Libo, Did you see this error in the broker 3's log, right after it was restarted? One of the reasons for this error is that the broker hasn't received the first LeaderAndIsr request from the controller yet. When a broker starts up, the only way it knows what partitions it owns is through the first

Re: message loss

sis, choosing consistency over > availability would open kafka to more use cases no? > > Sent from my iPhone > > On Aug 22, 2013, at 1:59 PM, Neha Narkhede > wrote: > > > Scott, > > > > Kafka replication aims to guarantee that committed writes are not lost. >

Re: repeated for request offset out of range

er.File=/logs/kafka/kafka-request.log > log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout > log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n > > Other logs including server.log, controller.log, state-change.log are all > correct. > > Regards, &g

Re: delete a topic

at 1:01 PM, Vadim Keylis wrote: > How do you guys delete a topic if such need arise? > > > On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede >wrote: > > > I mentioned this in a different thread. I think the > 0.8.0-beta1-candidate1 > > includes the script, but

Re: message loss

Scott, Kafka replication aims to guarantee that committed writes are not lost. In other words, as long as leader can be transitioned to a broker that was in the ISR, no data will be lost. For increased availability, if there are no other brokers in the ISR, we fall back to electing a broker that i

Re: repeated for request offset out of range

Could you check the kafka-request.log for these FetchRequests and see if they are sent by the same clientId ? Also, from the same kafka-request.log, could you check if the Kafka server received an OffsetRequest after these FetchRequests ? Thanks, Neha On Thu, Aug 22, 2013 at 10:23 AM, Wu, Jiang2

Re: delete a topic

I mentioned this in a different thread. I think the 0.8.0-beta1-candidate1 includes the script, but we have removed it later. I think when we publish 0.8-final, this should be resolved. Thanks, Neha On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede wrote: > Vadim, > > The JIRA trac

Re: No leaders elected after topics created

; > > On Thu, Aug 22, 2013 at 11:05 AM, Neha Narkhede >wrote: > > > As part of topic creation, the controller broker has to elect leader for > > each of the topics and this involves a few writes to zookeeper. So it > might > > take several seconds to complete lead

Re: delete a topic

Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps wrote: > > > We should really remove the delete script from 0.8 if we plan to release > > without delete support. Right now it appears to work but doesn't which is > > clearly not good. > > > > -Jay > > > &

Re: No leaders elected after topics created

As part of topic creation, the controller broker has to elect leader for each of the topics and this involves a few writes to zookeeper. So it might take several seconds to complete leader election for 300 topics and that time would increase if you have many partitions per topic. Which delete part

Re: delete a topic

We currently don't have the delete topic feature in Kafka 0.8. So any manual attempts to do so might have a negative impact on functionality. Thanks, Neha On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo wrote: > Hi team, > > When I delete a topic, the topic is deleted from zookeeper but its log > f

Re: More questions on avro serialization

int id of the > schema is written instead of the 16 byte SHA. Is this the new preferred > way? Which is compatible with > https://issues.apache.org/jira/browse/AVRO-1124? > > Thanks again > > On Aug 21, 2013, at 8:38 PM, Neha Narkhede > wrote: > > > We define the

Re: How does Kafka decide which Consumer out of multiple Consumer clients to assign to a single topic partition

We range partition a sorted list of topic-partitions over a sorted list of consumer instances. This makes the rebalancing algorithm deterministic. As soon as you bring up the 2nd consumer instance, if its position in the sorted list is before the position of the 1st consumer client, it will end up

Re: More questions on avro serialization

We define the LinkedIn Kafka message to have a magic byte (indicating Avro serialization), MD5 header followed by the payload. The Hadoop consumer reads the MD5, looks up the schema in the repository and deserializes the message. Thanks, Neha On Wed, Aug 21, 2013 at 8:15 PM, Mark wrote: > Does

Re: broker never comes back to ISR

transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=359 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame) > > > Thread 5135: (state = BLOCKED) > > > Thread 5134: (state = BLOCKED) > - java.lang.Object.wait(long) @bci=0 (Interpreted frame) > -

Re: controller data

This log message is marked INFO since it is normal, though I think the message content can be improved. The broker that does not get elected as the controller will have this log message that indicates it tried to become controller but some other broker succeeded instead. Thanks, Neha On Wed, Aug

Re: How to get broker back to ISR

; To: 'users@kafka.apache.org' > Subject: RE: How to get broker back to ISR > > Hi Neha, > > Which broker is controller broker and how is it defined? > > Regards, > > Libo > > > -Original Message- > From: Neha Narkhede [mailto:neha.narkh

Re: High level java consumer hang in kafka 0.7 because of LinkedBlockingQueue

Thanks for listing the possible conditions for a stalled consumer, Philip. I added this to the FAQ - https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F Frank, Let us know if the FAQ above helps to understand your issue or can be improved. Thanks, Neha

RE: broker never comes back to ISR

Could you take a thread dump on that broker and send it across? One of the possibilities is the replica fetcher thread is somehow dead. Thanks, Neha On Aug 21, 2013 8:00 AM, "Yu, Libo" wrote: > I checked the log of normal restart. The replication manager should start > to handle > leader and isr

Re: Kafka startup/restart process

ss besides setting timeout reasonable high? > > > > > > Thanks so much again, > > > Vadim > > > > > > > > > On Mon, Aug 19, 2013 at 9:11 PM, Neha Narkhede < > neha.narkh...@gmail.com > > > >wrote: > > > > > > >

Re: How to get broker back to ISR

Once the broker is restarted, the controller broker will send it a list of partitions that it should follow. The broker starts fetching from the respective leaders and enters the ISR. Depending on the duration of shutdown, the broker can take some time to enter ISR. Thanks, Neha On Aug 20, 2013 4:

Re: questtion about log.retention.bytes

Paul, I'm trying to understand the 2nd problem you reported. Are you saying that you set the log.retention.bytes=11534336 (11MB) but nevertheless your log grew to 114MB. Which means the config option didn't really work as expected? Thanks, Neha On Mon, Aug 19, 2013 at 8:46 PM, Jun Rao wrote:

Re: Kafka startup/restart process

ing one verses the other? > > Thanks, > Vadim > > > On Sun, Aug 18, 2013 at 11:05 PM, Vadim Keylis >wrote: > > > thanks so much. Greatly appreciated. > > > > > > On Sun, Aug 18, 2013 at 10:00 PM, Neha Narkhede >wrote: > > > >> I

Re: kafka_2.8.0-0.8.0-beta1.jar doesn't include kafka.utils.TestUtils

this to the main package > target rather than moving our test code into the main source directory. > > -Jay > > > On Mon, Aug 19, 2013 at 9:25 AM, Neha Narkhede >wrote: > > > You raised a good point. The same is true for all the tools that are > > under src/c

Re: Failed to start preferred replica election

n > the tool would run successfully. > > Thanks so much. > > > On Mon, Aug 19, 2013 at 12:59 PM, Neha Narkhede >wrote: > > > I think the error message can be improved to at least print which > > partitions it couldn't move the leader for. What could be happ

Re: Failed to start preferred replica election

I think the error message can be improved to at least print which partitions it couldn't move the leader for. What could be happening is that the 2 brokers that were down might not have entered the ISR yet. So the tool will not be able to move any leaders to them. You can run kafka-list-topics with

Re: kafka_2.8.0-0.8.0-beta1.jar doesn't include kafka.utils.TestUtils

You raised a good point. The same is true for all the tools that are under src/core/src/test/scala/other/kafka. I think we need to fix the packaging under src/core/test. Probably move the tools under core/src/main/scala/kafka/tools so that it goes in the main Kafka jar. And either include the test

Re: Kafka startup/restart process

er replicated partition > count during control shutdown that is configured in the property file? > > Thanks, > Vadim > > > On Sun, Aug 18, 2013 at 6:11 PM, Neha Narkhede >wrote: > > > Vadim, > > > > Controlled shutdown takes 2 parameters - number of retries and

Re: Kafka startup/restart process

Vadim, Controlled shutdown takes 2 parameters - number of retries and shutdown timeout. In every retry, controlled shutdown attempts to move leaders off of the broker that needs to be shutdown. If the controlled shutdown runs out of retries, it proceeds to shutting down the broker even if it still

Re: Leader election in the failure mode

If request processing is slow on some brokers that also have significantly more leaders than others, you should run the preferred replica election tool. We are looking into adding this functionality in the Kafka brokers where the brokers would automatically run preferred replica election. Thanks,

Re: Exception with kafka 0.8

The invalid argument exception on a socket connection looks weird. If you enable debug on kafka.controller.ControllerChannelManager, it will tell you which broker the newly elected controller is trying to talk to. Then you will have to make sure that every broker can connect to every other broker.

Re: General LinkedIn Question

Jonathan, We are in the process of replacing our legacy ActiveMQ based applications with Kafka. As such all messaging, logging and queuing applications use Kafka at LinkedIn. Thanks, Neha On Fri, Jul 26, 2013 at 7:31 AM, Jonathan Hodges wrote: > Hi LinkedIn Guys and Girls, > > Out of curiosit

Re: Meetup in Raleigh/Durham, NC

Great, thanks for presenting and sharing this David! On Wed, Jul 24, 2013 at 6:16 AM, David Arthur wrote: > Here are the slides from the meetup http://www.slideshare.net/** > mumrah/kafka-talk-tri-hug > > We had 40-50 people show up which is

Re: Consumer stops consuming after 5 gb

Samir, When you observe that the consumer hangs, could you send around the thread dump. Also can you check the value of the FetchRate mbean ? Thanks, Neha On Tue, Jul 23, 2013 at 10:17 AM, Samir Madhavan < samir.madha...@fluturasolutions.com> wrote: > We just observed that the consumer is occu

Re: clean up kafka environmet

node. Thanks, Neha On Tue, Jul 23, 2013 at 9:36 AM, Neha Narkhede wrote: > If the console producer/consumer works fine, it would be safe to assume > the broker is up. > > Thanks, > Neha > > > On Tue, Jul 23, 2013 at 8:44 AM, Oleg Ruchovets wrote: > >> Hi Jun , >

Re: clean up kafka environmet

If the console producer/consumer works fine, it would be safe to assume the broker is up. Thanks, Neha On Tue, Jul 23, 2013 at 8:44 AM, Oleg Ruchovets wrote: > Hi Jun , > >I made such tests: > *bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic > test > * > > Thi

Re: 0.8 Durability Question

No. It only means that messages are written to all replicas in memory. Data is flushed to disk asynchronously. Thanks, Neha On Jun 13, 2013 3:29 AM, "Jonathan Hodges" wrote: > Looking at Jun’s ApacheCon slides ( > http://www.slideshare.net/junrao/kafka-replication-apachecon2013) slide 21 > title

RE: message order, guarenteed?

Kafka guarantees order per topic partition per source client. Thanks, Neha On Jun 9, 2013 5:33 PM, "S Ahmed" wrote: > I understand that there are no guarantees per say that a message may be a > duplicate (its the consumer's job to guarantee that), but when it comes to > message order, is kafka

Re: Dilemma - ZK consumer problems - upgrade to 0.8?

Which version of ZK are you using ? On Fri, Jun 7, 2013 at 12:15 PM, Evan Chan wrote: > Hey guys, > > I sincerely apologize if this has been covered before, I haven't quite > found a similar situation. > > We are using Kafka 0.7.2 in production, and we are using the ZK high level > Scala consum

Re: consumer offset (0.7)

You can use the ConsumerOffsetChecker tool that ships with Kafka. Thanks, Neha On Wed, Jun 5, 2013 at 2:19 PM, Ian Friedman wrote: > Hey guys, > > Is there any way to tell where the consumer offset is in terms of the > total size of the topic or topic-partition? I know i can read the offset >

Re: 2 kafkas 1 zookeeper

It should be fine as long as you pick different zookeeper namespaces for 0.7.2 and 0.8 Thanks, Neha On Wed, Jun 5, 2013 at 10:26 AM, Andrew Neilson wrote: > We are interested in having two kafka clusters (one 0.7.2, one 0.8) share > the same 3 node zookeeper tier (3.4.3). Has anyone else tried

Re: Compression in kafka 0.8

The only way of storing compressed data on the server is to have the producer send it compressed. The server sends the data in compressed format to the consumers. Thanks, Neha On Tue, Jun 4, 2013 at 9:55 PM, Vadim Keylis wrote: > Good evening. I would like to preserve logs in kafka for 2 days.

Re: macbook air and kafka

> > So would you say it is actually enjoyable to develop on it? Or its not > > > really that ideal compared to a desktop. > > > What about the heat? > > > > > > > > > On Mon, Jun 3, 2013 at 9:54 PM, Neha Narkhede > >wrote: > > > >

Re: Producer only finding partition on 1 of 2 Brokers, even though ZK shows 1 partition exists on both Brokers?

You are probably hitting https://issues.apache.org/jira/browse/KAFKA-278. Can you please try the workaround mentioned in the JIRA description? Thanks, Neha On Tue, Jun 4, 2013 at 4:21 PM, Brett Hoerner wrote: > (version 0.7.2) > > For some reason, my producers are only picking up the partition

Re: sync and async

Async producer just wraps sync producer and handles batching. So whether you use async producer or sync producer, it will use the request.required.acks only for the sync producer. Thanks, Neha On Tue, Jun 4, 2013 at 3:05 PM, Yu, Libo wrote: > Hi > > What is the difference between an aync produ

Re: facing the same problem

Gaurang, Did you try pointing to a new zookeeper namespace or use a different zookeeper cluster? Thanks, Neha On Tue, Jun 4, 2013 at 11:50 AM, Gaurang Jhawar wrote: > > http://mail-archives.apache.org/mod_mbox/kafka-users/201301.mbox/%3CCAAG86fpamV2XeB5=xx19xc81q1f93avn9g3r3hyjqz-lnxx...@mail.

Re: kafka 0.8

I was just about to send an update. We can release beta right away. Joe, I remember you were interested in helping out. Let me know if you are still up for managing the release. Thanks, Neha On Jun 4, 2013 9:22 AM, "Soby Chacko" wrote: > Hello, > > Just wanted to know where we are with the bet

Re: Kafka Hadoop Consumer for multiple brokers

It is probably possible to try to make the serde format pluggable. You can try to start that discussion on the camus mailing list. Thanks, Neha On Jun 4, 2013 8:51 AM, "Samir Madhavan" < samir.madha...@fluturasolutions.com> wrote: > Thanks Jun. > > For one of our project, the data is coming in th

Re: macbook air and kafka

It really depends on your air specs. It takes roughly a minute to compile from scratch on my macbook air. Thanks, Neha On Mon, Jun 3, 2013 at 6:33 PM, S Ahmed wrote: > Hi, Curious if anyone uses a macbook air to develop with kafka. > > How are compile times? Does the fan go crazy after a whil

Re: Can't install 0.8

+ kafka-users On Mon, Jun 3, 2013 at 1:15 PM, Neha Narkhede wrote: > Did you do *./sbt assembly-package-dependency ?* > > > On Mon, Jun 3, 2013 at 11:59 AM, Bob Jervis < > bjer...@visibletechnologies.com> wrote: > >> I tried to follow the 0.8 quickstart

Re: Producer stuck

The error or the actual problem is very unclear. Do you mind giving more details? Did your producers throw any errors or stop sending messages? Thanks, Neha On Jun 3, 2013 7:43 AM, "田晓亮" wrote: > 3 zookeepers,1 kafka > > There is 2 producer ,and both of them can connect to kafka through > zookee

Re: consumer can't consume data any more when change the number of kafka partitions

Have you set your consumer fetch.size larger than the max.messsage.size on your producers? Thanks, Neha On Jun 3, 2013 4:04 AM, "shangan chen" wrote: > This is a testing log in non-storm enviroment, it can't fetch any messages > and seems to die there. > > > 13/06/03 18:44:55 INFO consumer.Zooke

Re: What exactly happens if fetch size is smaller than the next batch (0.72 and high-level consumer)

Yes, it is the maxSize parameter in FetchRequest. And the consumer won't stall unless a single message (compressed or not) is larger than the fetch size, in this case 300KB. It doesn't matter how big a batch is. Thanks, Neha On Fri, May 31, 2013 at 11:25 PM, Philip O'Toole wrote: > Hello -- I'l

Re: question about usage of SimpleConsumer

nd each consumer > established a stream other than all consumers share the same stream. It > seems to verify that latest offset is brokers's queue info other than > consumer's state. > > > > > On Fri, May 31, 2013 at 11:49 PM, Neha Narkhede >wrote: > > > g

Re: zookeeper version issue

Actually, the version of zookeeper that Kafka 08 and 07 depend on is 3.3.4. Zookeeper 3.3.3 has known serious bugs with lost watchers. Thanks, Neha On Fri, May 31, 2013 at 10:22 AM, Dennis Haller wrote: > The version of zookeeper embedded with Kafka 0.8 is still v. 3.3.3. > > Regards > Dennis >

Re: question about usage of SimpleConsumer

getOffsetsBefore sends an RPC call to the Kafka brokers to find out the earliest/latest offset for that topic, partition. In your example, it will get you the latest offset at the time of the request. Thanks, Neha On Thu, May 30, 2013 at 11:31 PM, shangan chen wrote: > In Kafka, the consumers a

Re: kafka.common.LeaderNotAvailableException: No leader for any partition

Can you check if the value for zookeeper.connect is set correctly ? We changed the name of some configs in 08, you can find the new config names here - http://kafka.apache.org/08/configuration.html Thanks, Neha On Thu, May 30, 2013 at 1:33 PM, Vadim Keylis wrote: > Your advice is appreciated >

Re: SimpleConsumer and message offsets

If main advantage of SimpleConsumer is so that you can have finer level control on the offset management. So you have to keep track of offsets. If you don't care about offset management, you can use the ZookeeperConsumerConnector. Thanks, Neha On Wed, May 29, 2013 at 11:50 AM, arathi maddula wro

Re: About Kafka Users Group around Hadoop Summit

bug you guys, but is the date fixed? So that I can book my flight > accordingly. > > Regards, > Vaibhav > GumGum > > > On Tue, May 21, 2013 at 10:56 AM, Neha Narkhede >wrote: > > > It will be great to have a user group meeting around Hadoop Summit. I > &

Re: Partitioning and scale

es its own topic with single partition, therefore there could be > > millions of topic with single partition. > > > > I wonder what would be the bottleneck of doing this? > > > > Thanks, > > > > Tim > > > > > > On Wed, May 22, 2013 at 4:3

Re: [0.8] Autocommit bug? Or am I doing it wrong? :)

That sounds odd. Can you turn on DEBUG for kafka.consumer.ZookeeperConsumerConnector in your consumer and check if you see the following log message - "Committed offset for topic " Which version of Kafka are you using? Thanks, Neha On Fri, May 24, 2013 at 8:06 AM, Hargett, Phil < phil.harg...@

Re: consumer offset not saved in zk

> > com.sun.jdmk > jmxtools > > > javax.jms > jms > > > > > > > On Thu, May 2

Re: More Kafka Benchmarking Goodness

Thanks for sharing this, Jason. I had performed similar benchmarks back in 2011 (Results are on slide 27 here - https://cwiki.apache.org/confluence/download/attachments/2786/F_1330_Narkhede_Kafka+%281%29.pptx?version=1&modificationDate=1358352559000). My tests saturated the 1Gb network link bet

Re: consumer offset not saved in zk

You don't want to override the default configs. Also, seems like something else is wrong with your setup ? Could you share the log4j logs of your consumer ? Meanwhile, can you try if you can use the console consumer successfully ? Thanks, Neha On Thu, May 23, 2013 at 2:31 PM, rk vishu wrote: >

Re: heterogenous kafka cluster?

This paper talks about how to do that - http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf It will be interesting to see what part of it Kafka can adopt, if any. Thanks, Neha On Fri, May 17, 2013 at 11:28 PM, Jason Rosenberg wrote: > Letting each broker have a weight sounds like a great idea. > > S

Re: Failed to launch 0.8 stable server

Even if they are wrong, were you pointing to a zookeeper cluster where another Kafka cluster was started up ? Thanks Neha On Thu, May 23, 2013 at 7:53 AM, Yu, Libo wrote: > I will answer my own question. It turns out the zookeeper addresses in > server.properties are > wrong. But it is not ea

Re: kafka 0.8 - unable to run most scala test cases

You can find all unit tests under core/test. Just look for *Test.scala. Thanks, Neha On May 23, 2013 9:59 AM, "Rob Withers" wrote: > I am using 0.8 in eclipse. I used the approach suggested to use the sbt > plugin and that works great (thank you to whomever recommended that). It > pulled Scala

Re: orders of launching kafka servers and zookeepers

even in prod, whether we like it > or not :) > > thanks, > marc > > > On Wed, May 22, 2013 at 12:10 PM, Neha Narkhede >wrote: > > > First launch the zookeeper cluster completely followed by the kafka > > cluster. > > > > Thanks, > > Neha >

Re: Offset in high level consumer

r. > > Any pointers will be of great help. > Thanks > Arathi > On Wed, May 22, 2013 at 7:24 PM, Neha Narkhede >wrote: > > > You can run the ConsumerOffsetChecker tool that ships with Kafka. > > > > Thanks, > > Neha > > > > > > On Wed,

Re: are topics and partitions dynamic?

You can specify it in server.properties and set it to true. For more information on 08 configs, look here http://kafka.apache.org/08/configuration.html Thanks, Neha On May 23, 2013 8:04 AM, "Noel Golding" wrote: > I am currently using kafka-0.8.0. I don't see a reference to > auto.create.topics

Re: consumer offset not saved in zk

I suspect you had auto.commit.enable=false and consumer.timeout.ms=1. Can you confirm the values for the above configs in your example? Thanks, Neha On May 22, 2013 11:22 PM, "rk vishu" wrote: > Hello All, > > I recently started experimenting Kafka for my usecase. I am running 0.8 in > two n

Re: Partitioning and scale

Hi Neha/Chris, > > Thanks for the reply, so if I set a fixed number of partitions and just add > brokers to the broker pool, does it rebalance the load to the new brokers > (along with the data)? > > Tim > > > On Wed, May 22, 2013 at 1:15 PM, Neha Narkhede &g

Re: Offset in high level consumer

You can run the ConsumerOffsetChecker tool that ships with Kafka. Thanks, Neha On Wed, May 22, 2013 at 2:02 PM, arathi maddula wrote: > Hi, > > Could you tell me how to find the offset in a high level Java consumer ? > > Thanks > Arathi >

Re: message ordering guarantees

Thanks, Neha On May 21, 2013 5:42 PM, "Ross Black" wrote: > Hi, > > I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with a > single broker service process. > > I am occasionally seeing messages (from a *single* partition) being > processed out of order to what I expect and I am

Re: Apache Kafka in AWS

Thanks for sharing your experience with the community, Jason! -Neha On Wed, May 22, 2013 at 1:42 PM, Jason Weiss wrote: > All, > > I asked a number of questions of the group over the last week, and I'm > happy to report that I've had great success getting Kafka up and running in > AWS. I am us

Re: Partitioning and scale

- I see that Kafka server.properties allows one to specify the number of partitions it supports. However, when we want to scale I wonder if we add # of partitions or # of brokers, will the same partitioner start distributing the messages to different partitions? And if it does, how can that same c

Re: orders of launching kafka servers and zookeepers

First launch the zookeeper cluster completely followed by the kafka cluster. Thanks, Neha On May 22, 2013 8:43 AM, "Yu, Libo" wrote: > Hi, > > I want to launch kafka on three machines. I can launch zookeepers > on the three machines first. After that, start kafka server on each > machine. Or for

Re: Producer can't initialize if zk host down (0.7.2)

figurations. But it makes > it difficult to roll in new changes to the cluster layout while keeping > services up. > > Jason > > > On Tue, May 21, 2013 at 2:20 PM, Neha Narkhede >wrote: > > > According to zookeeper code, when you try to create a client handle to > >

Re: Producer can't initialize if zk host down (0.7.2)

According to zookeeper code, when you try to create a client handle to zookeeper, it resolves each of the hosts in the zookeeper connection string. If any of these fail, it throws an exception. If you use a zookeeper based producer, it tries to create a ZkClient handle which in turn tries to create

Re: About Kafka Users Group around Hadoop Summit

It will be great to have a user group meeting around Hadoop Summit. I prefer Thursday. Thanks, Neha On Tue, May 21, 2013 at 10:54 AM, Vaibhav Puranik wrote: > Any day will work for me. I am waiting for you to decide on time so that I > can book my ticket accordingly! > > Regards, > Vaibhav > >

Re: Closing socket connections during extended message delivery

Yes. This was added to get around the VIP hard killing connections that are idle. If your producer doesn't use a VIP you can set this high enough so it rarely reconnects. Thanks, Neha On May 21, 2013 8:13 AM, "Jason Weiss" wrote: > Ahh - I see, thank you. > > Would you describe this as a "tunabl

RE: only-once consumer groups

gned to a new thread, > thus > violating manual partitioning? > > Or is a rebalance when a broker dies and new leader partitions are elected? > The simple consumer must get told of a leader election and this is nothing > to do with rebalance, is it so? > > Thanks, > rob >

Re: Update: RE: are commitOffsets botched to zookeeper?

e: > > > Have you considered abstracting offset storage away so people could > implement their own? > > Would you take a patch if I'd stabbed at it, and if yes, what's the > process (pardon the n00b)? > > > > KCBO, > > -- > > "If you can

RE: are commitOffsets botched to zookeeper?

:36 AM, "Seshadri, Balaji" wrote: > Hi Neha, > > Is moving to zookeeper 3.4.x is a big change ?. > > Can you please explain parts it affects consumer API for example ?. > > Thanks, > > Balaji > -Original Message- > From: Neha Narkhede [mailto:neha

Re: hi plz reple me

You can do that using Kafka. Please read the design details here - http://kafka.apache.org/07/design.html Thanks, Neha On May 20, 2013 6:57 AM, "satya prakash" wrote: > i am using kafka .i need to create one message on producer side and send to > multiple consumer...? >

Re: What happens if one broker goes down

g ??? Please help > > For reference : > I'm using kafka 0.7.2 since my consumer is a storm spout and i read that > storm is most compatible with kafka 0.7.2. > IT DOESNT SUPPORT REPLICATION RIGHT? > > > On Fri, May 17, 2013 at 6:56 PM, Neha Narkhede >w

Re: can producers(from same system) send messages to separate broker systems?

The feature I mentioned is only available on 0.8. In 0.7.2, you can tweak producer batch size and the flush interval on the broker for the high priority topics. Note that setting those too low will have performance implications. Thanks, Neha On May 17, 2013 2:25 PM, "Neha Narkhede" wr

Re: heterogenous kafka cluster?

That does seem a little hacky. But I'm trying to understand the requirement behind having to deploy heterogeneous hardware. What are you trying to achieve or optimize? Thanks, Neha On Fri, May 17, 2013 at 2:29 PM, Jason Rosenberg wrote: > Hi, > > I'm wondering if there's a good way to have a h

Re: can producers(from same system) send messages to separate broker systems?

er.list > for each producer , is enough? > > > On Tue, May 14, 2013 at 9:11 PM, Neha Narkhede >wrote: > > > Yes there can be. You just need to make sure they are configured with > > separate broker.list. > > > > Thanks, > > Neha > > On May 14, 201

Re: Update: RE: are commitOffsets botched to zookeeper?

is much more stable compared to 3.4.x. We think it is better not to > > club 2 big changes together. So most likely this will be a post 08 item > for > > stability purposes. > > > > Thanks, > > Neha > > On May 17, 2013 6:31 AM, "Withers, Robert" > >

Re: Setting a broker to be read-only?

There is no easy way to do this in 0.7. If you are using a VIP to produce data to Kafka, you can remove that broker host from the VIP. It will still be available for consumer reads until it is shut down but won't accept new data from the producer. In 0.8, since there are many replicas, you will not

Re: Getting the latest message always from Kafka

For every unit test, you want your server and consumer to start fresh. In your unit test, you maintain state in the KafkaManager across unit tests so your tests don't work as expected. Try moving KafkaManager to the setup method. Currently, the 2nd test is consuming the message produced by the 1st

Re: Our use case and am I right in my definition of Throughput?

Another parameter that you want to tune if you want higher throughput in 0.7 is the broker side flush interval. If you set it to something high, like 50K or 100K, you can maximize the throughput achieved by your producer. Thanks, Neha On Fri, May 17, 2013 at 8:11 AM, Jun Rao wrote: > You earli

RE: WebMethods consumer for Kafka.

> becoming an issue. > > Thanks, > > Balaji > > -Original Message- > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > Sent: Thursday, May 16, 2013 10:00 PM > To: users@kafka.apache.org > Subject: Re: WebMethods consumer for Kafka. > > It seems tha

RE: possible to shutdown a consumerConnector without flushing the offset

nd but egads, it's > tricky. > > thanks, > rob > > From: Neha Narkhede [neha.narkh...@gmail.com] > Sent: Friday, May 17, 2013 7:29 AM > To: users@kafka.apache.org > Subject: Re: possible to shutdown a consumerConnector without flushing the > offset > >

Re: Time difference between message fetch and message send is very high (~900-1200ms)

Kishore, The end to end latency will be much lower in 08. For 07, you can try to tune the producer batch size so it sends data quicker. You can also reduce the log.flush.interval on the broker but that will drive up iops. Thanks, Neha On May 16, 2013 11:39 PM, "Kishore V. Kopalle" wrote: > Hi F

RE: are commitOffsets botched to zookeeper?

hed. > > thanks, > rob > ____ > From: Neha Narkhede [neha.narkh...@gmail.com] > Sent: Friday, May 17, 2013 7:17 AM > To: users@kafka.apache.org > Subject: RE: are commitOffsets botched to zookeeper? > > Sorry I wasn't clear. Zookeeper 3.4.x has this feature. As soo

RE: only-once consumer groups

n if several fetchers are > feeding a thread. Perhaps the message metadata contains the partition a > msg > came from; could you not just update the offset for that partition if the > property "only.once=true" is sent to the consumer connector? > > Thanks, > rob

<    2   3   4   5   6   7   8   9   10   >