Re: kafka sending duplicate content to consumer

and crash without committing the offsets somewhere (either to kafka or in some other system). When you restart the you will start your fetch again from the last committed offset. Kafka has no notion of an already consumed message. 2015-01-23 7:54 GMT+01:00 Tousif tousif.pa...@gmail.com: Hi

kafka sending duplicate content to consumer

Hi, i want know in which situation does kafka send same event multiple times to consumer. Is there a consumer side configuration to tell kafka to send only once and stop retries? -- Regards Tousif Khazi

Re: kafka brokers going down within 24 hrs

any help ? On Mon, Jan 19, 2015 at 11:43 AM, Tousif tousif.pa...@gmail.com wrote: Here are the logs from broker id 0 and 1 and it was captured when broker 1 went down. http://paste.ubuntu.com/9782553/ http://paste.ubuntu.com/9782554/ i'm using netty in storm and here are the configs

Re: kafka brokers going down within 24 hrs

(BoundedByteBufferReceive.scala:54) at kafka.network.Processor.read(SocketServer.scala:347) at kafka.network.Processor.run(SocketServer.scala:245) at java.lang.Thread.run(Thread.java:662) On Thu, Jan 15, 2015 at 7:49 PM, Harsha ka...@harsha.io wrote: Tousif, Which version of kafka and zookeeper are you

Re: kafka brokers going down within 24 hrs

, Tousif tousif.pa...@gmail.com wrote: Thanks harsha for quick response. I don't see any other error. I used to see replica fetcher error but seems to be disappeared after setting replica fetcher threads to 2 as I have 2 partitions. Some times I see zookeeper session expiration. On Jan 14, 2015 8:02

Re: kafka brokers going down within 24 hrs

at 12:01 PM, Chia-Chun Shih chiachun.s...@gmail.com wrote: You can use tools (e.g., VisialVM) to diagnose OOM problem. 2015-01-15 14:15 GMT+08:00 Tousif Khazi tou...@senseforth.com: i see this error ERROR [ReplicaFetcherThread-0-1], Error for partition [realtimestreaming,1] to broker 1

Re: kafka brokers going down within 24 hrs

: Tousif, Do you see any other errors in server.log -Harsha On Wed, Jan 14, 2015, at 01:51 AM, Tousif wrote: Hello, I have configured kafka nodes to run via supervisord and see following exceptions and eventually brokers going out of memory. i have given enough memory

kafka brokers going down within 24 hrs

. (kafka.network.Processor) Error occurred during initialization of VM Could not reserve enough space for object heap Error occurred during initialization of VM Could not reserve enough space for object heap -- Regards, Tousif +918050227279 -- Regards Tousif Khazi