Jun, Joel,
The issue here is exactly which threads are left out, and which threads are
assigned partitions.
Maybe I am missing something but what I want is to balance consuming
threads across machines/processes, regardless of the amount of threads the
machine launches (side effect: this way if you
Hi Jun,
The hasNext() itself throws this error. I have to manually reset state and
sometime it is able to recover and other it is not. Any other clue ?
public boolean hasNext() {
LOG.info("called of hasNext() :");
int retry = 3;
while(retry > 0){
I don't see anything else that is relevant. I traced the first of these
error messages to figure out the ordering. It actually goes something like
this:
2014-10-30T01:51:32.400Z ERROR [ProducerSendThread-]
[k.producer.async.DefaultEventHandler] {}: Failed to collate messages by
top
This pattern seems to repeat:
2014-10-30T01:54:46.004Z ERROR [ProducerSendThread-]
[k.producer.async.DefaultEventHandler] {}: Failed to send requests for
topics myTopic with correlation ids in [1729,1736]
2014-10-30T01:54:46.008Z ERROR [ProducerSendThread-]
[k.prod
Shlomi,
If you are on trunk, and your consumer subscriptions are identical
then you can try a slightly different partition assignment strategy.
Try setting partition.assignment.strategy="roundrobin" in your
consumer config.
Thanks,
Joel
On Wed, Oct 29, 2014 at 06:29:30PM -0700, Jun Rao wrote:
>
The log before that will show you the cause of the error. Could you dig
that out?
Thanks,
Jun
On Wed, Oct 29, 2014 at 6:43 PM, Rajiv Kurian wrote:
> I keep seeing these errors in my code that is just trying to send some data
> using an AsyncProducer:
>
> kafka.common.FailedToSendMessageExcepti
Thanks Jun, looks like I am on the right track.
On Wed, Oct 29, 2014 at 6:51 PM, Jun Rao wrote:
> 1), 2), 3) Yes. If you use SimpleConsumer, you have to figure out the
> leader of each partition and connect to the right broker. Each fetch
> request can send multiple partitions (if with same lead
1), 2), 3) Yes. If you use SimpleConsumer, you have to figure out the
leader of each partition and connect to the right broker. Each fetch
request can send multiple partitions (if with same leader) and you need to
examine the error code per partition.
Not all those error codes are applicable to th
I keep seeing these errors in my code that is just trying to send some data
using an AsyncProducer:
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at kafka.producer.async.DefaultEventHandler.handle(Unknown Source)
~[kafka_2.10-0.8.0.jar:0.8.0]
at kafka.producer
The IllegalStateException typically happens if you call next() before
hasNext() on the iterator.
Thanks,
Jun
On Tue, Oct 28, 2014 at 10:50 AM, Bhavesh Mistry wrote:
> Hi Neha,
>
> Thanks for your answer. Can you please let me know how I can resolve the
> Iterator IllegalStateException ? I wo
What's the output of the ConsumerOffsetChecker tool?
Thanks,
Jun
On Tue, Oct 28, 2014 at 7:31 AM, Natarajan, Murugavel <
murugavel.natara...@softwareag.com> wrote:
> Hi,
>
> I have the following Kafka Setup
> Number of producer : 1
> Number of topics : 1
> Number of partitions : 2
> Number of c
By consumer, I actually mean consumer threads (the thread # you used when
creating consumer streams). So, if you have 4 consumers, each with 4
threads, 4 of the threads will not get any data with 12 partitions. It
sounds like that's not what you get? What's the output of the
ConsumerOffsetChecker
Sorry for getting late on this Sameer.
The logs you showed above are from controller logs, do you see brokers logs
besides that?
Guozhang
On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami wrote:
> Hi Guozhang,
>
> Is there any clue in the above data as to what is wrong?
>
> thanks
>
>
> On Tue, Oct
Hi Guozhang,
Is there any clue in the above data as to what is wrong?
thanks
On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami wrote:
> This is the one from the file specified in log4j.properties -
>
> [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on Controller 0]:
> Topic change listener f
How many topics do you have and can you share message throughput ?.
What's the functionality you are using for ?.
-Original Message-
From: Michael Nussbaum [mailto:michaelnussbau...@gmail.com]
Sent: Wednesday, October 29, 2014 1:57 PM
To: users
Subject: Re: Apache Kafka Consumers in jav
We are using 0.8.1. Will try 0.8.2-beta.
Do you have the JIRA ticket for the NPE?
Thanks,
Allen
On Wed, Oct 29, 2014 at 1:16 PM, Neha Narkhede
wrote:
> Which Kafka version are you on for the consumer? We recently fixed a NPE
> related to ZkClient, could you give 0.8.2-beta a try on your consum
Which Kafka version are you on for the consumer? We recently fixed a NPE
related to ZkClient, could you give 0.8.2-beta a try on your consumer and
retry the test?
On Wed, Oct 29, 2014 at 10:34 AM, Allen Wang wrote:
> After executing PreferredReplicaLeaderElectionCommand on broker instance,
> we
Yes, we're using it in production, and no, no performance issues or
bottlenecks related to Kafka. All consumption/production has continued
normally. Before this we were on open jdk 7.
On Wed, Oct 29, 2014 at 2:41 PM, Seshadri, Balaji
wrote:
> Are you using it in production ?.
>
> Did you experie
Are you using it in production ?.
Did you experience any performance or bottleneck ?.
-Original Message-
From: Michael Nussbaum [mailto:michaelnussbau...@gmail.com]
Sent: Wednesday, October 29, 2014 1:39 PM
To: users@kafka.apache.org
Subject: Re: Apache Kafka Consumers in java 8
I've be
I've been using clojure on open jdk 8 for my producers and consumers for
about a month now without any issue. Anything specific you're interested in?
Cheers,
Michael Nussbaum
On Wed, Oct 29, 2014 at 2:32 PM, Seshadri, Balaji
wrote:
> Have anybody used Kafka with Java 8 ?.
>
> -Original Mess
Have anybody used Kafka with Java 8 ?.
-Original Message-
From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com]
Sent: Wednesday, October 29, 2014 11:11 AM
To: 'd...@kafka.apache.org'; 'users@kafka.apache.org'
Subject: Apache Kafka Consumers in java 8
Hi All,
Can you please share your
I am planning to use the current Java API and have the following use case:
i) A single topic with about 1024 partitions.
ii) A number of processes that want to consume these partitions in a
deterministic way. The machine -> partitions assignment is done outside of
kafka. During the lifetime of a p
After executing PreferredReplicaLeaderElectionCommand on broker instance,
we observed one of the consumers cannot find the leadership and stopped
consuming. The following exception is all over the log file and it appears
that the consumer cannot recover from it:
2014-10-29 00:53:30,492 WARN
suroro
Hi All,
Can you please share your experiences of running Kafka Consumers/producers with
Java 8 ?.
Thanks,
Balaji
Hi, sorry for the extremely late follow up:
No, there is nothing logged on the broker, Using default log4j.properties. I
also tried to activate trace logging for the network processor component
(log4j.logger.kafka.network.Processor=TRACE, requestAppender), without seeing
anything that seemed r
Created https://issues.apache.org/jira/browse/KAFKA-1737
Thanks for support!
Kind regards,
Stevo Slavic.
On Tue, Oct 28, 2014 at 2:07 AM, Guozhang Wang wrote:
> I think AdminUtils just take a zkClient as its parameter, and the zkClient
> should setZkSerializer at the time when it is initialize
26 matches
Mail list logo
