Kafka Consumer stops receiving messages.

Hi All, My kafka consumer suddenly stops receiving messages and lags keep on increasing. Stack trace shows me the following for all the consumer threads. *"pool-43-thread-20" prio=10 tid=0x7f9a1d9d5000 nid=0x17888 waiting on condition [0x7f95d8e0d000]* * java.lang.Thread.State:

kafka 0.9.0 java version

Hi all, what is the oracle jdk version that is "supported" by kafka 0.9.0 ? 6/7/8... ATTENTION: The information in this e-mail is confidential and only meant for the intended recipient. If you are not the intended recipient ,

New consumer pause/reset behaviour

I am trying to enable throttling on a consumer poller. My understanding of the consumer is that it has a single threaded design so that all comms affecting the consumer are sent during a client threads call to poll and commitSynch. We have therefor designed our throttling by using the

RE: kafka 0.9.0 java version

My mistake, missed that :) Do you know if the latest stable zookeper (3.4.6) works with java 1.8 :D ? -Original Message- From: Franco Giacosa [mailto:fgiac...@gmail.com] Sent: Thursday, January 28, 2016 6:02 PM To: users@kafka.apache.org Subject: Re: kafka 0.9.0 java version check

Re: Accumulating data in Kafka Connect source tasks

> On Jan 28, 2016, at 5:06 PM, Ewen Cheslack-Postava wrote: > > Randall, > > Great question. Ideally you wouldn't need this type of state since it > should really be available in the source system. In your case, it might > actually make sense to be able to grab that

Re: How to set Timeout for KafkaConsumer.poll()

Thanks, Jason and Guozhang. I guess the conclusion is that, if there is no other logic to execute other than poll(), a long timeout and a short timeout have no big difference. Guozhang, if I understand right, even with this issue, I don't think a long timeout and a short timeout will be much

Re: How to set Timeout for KafkaConsumer.poll()

That is right, for now. On Thu, Jan 28, 2016 at 2:05 PM, Yifan Ying wrote: > Thanks, Jason and Guozhang. I guess the conclusion is that, if there is no > other logic to execute other than poll(), a long timeout and a short > timeout have no big difference. > > Guozhang, if I

How to set Timeout for KafkaConsumer.poll()

Hi All, I was using the new Kafka Consumer to fetch messages in this way: while (true) { ConsumerRecords records = kafkaConsumer.poll(Long.MAX_VALUE); // do nothing if records are empty } Then I realized that blocking until new messages fetched might be a little

Re: How to set Timeout for KafkaConsumer.poll()

Hey Yifan, As far as how the consumer works internally, there's not a big difference between using a long timeout or a short timeout. Which you choose really depends on the needs of your application. Typically people use a short timeout in order to be able to break from the loop with a boolean

Re: How to set Timeout for KafkaConsumer.poll()

Hi Yifan, There are some cases that could cause a consumer to block longer than the specified poll timeout, for example if Kafka is not up and running at all the consumer would be blocked until it is connected to the broker. Some more details are here:

Re: Accumulating data in Kafka Connect source tasks

Randall, Great question. Ideally you wouldn't need this type of state since it should really be available in the source system. In your case, it might actually make sense to be able to grab that information from the DB itself, although that will also have issues if, for example, there have been

Re: Broker Exception: Attempt to read with a maximum offset less than start offset

I can reproduce this on 0.9.0.0. The error log is [2016-01-28 16:12:32,840] ERROR [Replica Manager on Broker 1]: Error processing fetch operation on partition [ad-events,1] offset 75510318 (kafka.server.ReplicaManager) I also print the sent offset from producer time partition offset

Re: New consumer pause/reset behaviour

Hey Tom, Yes, it is possible that the poll() will rebalance and resume fetching for a previously paused partition. First thought is to use a ConsumerRebalanceListener to re-pause the partitions after the rebalance completes.The rebalance listener offers two hooks: onPartitionsRevoked() is called

How to bind all Kafka tcp port to private net address

My version is kafka_2.11-0.9.0.0. I find that the kafka listen on multi tcp port on a linux server. [gdata@gdataqosconnd2 kafka_2.11-0.9.0.0]$ netstat -plnt|grep java (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp

Re: Accumulating data in Kafka Connect source tasks

Rather than leave this thread so open ended, perhaps I can narrow down to what I think is the best approach. These accumulations are really just additional information from the source that don’t get written to the normal topics. Instead, each change to the accumulated state can be emitted as

Re: kafka 0.9.0 java version

check http://kafka.apache.org/documentation.html#java 2016-01-28 16:58 GMT+01:00 Muresanu A.V. (Andrei Valentin) < andrei.mures...@ing.ro>: > Hi all, > > what is the oracle jdk version that is "supported" by kafka 0.9.0 ? > > 6/7/8... > >