Jun,
The detailed logs are as follows:
24.04.2014 13:37:31812 INFO main kafka.producer.SyncProducer - Disconnecting
from localhost:9092
24.04.2014 13:37:38612 WARN main kafka.producer.BrokerPartitionInfo - Error
while fetching metadata [{TopicMetadata for topic LOGFILE04 ->
No partition metadata for topic LOGFILE04 due to
kafka.common.LeaderNotAvailableException}] for topic [LOGFILE04]: class
kafka.common.LeaderNotAvailableException
24.04.2014 13:37:40712 INFO main kafka.client.ClientUtils$ - Fetching metadata
from broker id:0,host:localhost,port:9092 with correlation id 1 for 1 topic(s)
Set(LOGFILE04)
24.04.2014 13:37:41212 INFO main kafka.producer.SyncProducer - Connected to
localhost:9092 for producing
24.04.2014 13:37:48812 INFO main kafka.producer.SyncProducer - Disconnecting
from localhost:9092
24.04.2014 13:37:48912 WARN main kafka.producer.BrokerPartitionInfo - Error
while fetching metadata [{TopicMetadata for topic LOGFILE04 ->
No partition metadata for topic LOGFILE04 due to
kafka.common.LeaderNotAvailableException}] for topic [LOGFILE04]: class
kafka.common.LeaderNotAvailableException
24.04.2014 13:37:49012 ERROR main kafka.producer.async.DefaultEventHandler -
Failed to collate messages by topic, partition due to: Failed to fetch topic
metadata for topic: LOGFILE04
24.04.2014 13:39:96513 WARN
ConsumerFetcherThread-produceLogLine2_vcmd-devanshu-1398361030812-8a0c706e-0-0
kafka.consumer.ConsumerFetcherThread -
[ConsumerFetcherThread-produceLogLine2_vcmd-devanshu-1398361030812-8a0c706e-0-0],
Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 4; ClientId:
produceLogLine2-ConsumerFetcherThread-produceLogLine2_vcmd-devanshu-1398361030812-8a0c706e-0-0;
ReplicaId: -1; MaxWait: 60000 ms; MinBytes: 1 bytes; RequestInfo:
[LOGFILE04,0] -> PartitionFetchInfo(2,1048576)
java.net.SocketTimeoutException
at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
at kafka.utils.Utils$.read(Unknown Source)
at kafka.network.BoundedByteBufferReceive.readFrom(Unknown Source)
at kafka.network.Receive$class.readCompletely(Unknown Source)
at kafka.network.BoundedByteBufferReceive.readCompletely(Unknown Source)
at kafka.network.BlockingChannel.receive(Unknown Source)
at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source)
at
kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown
Source)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown
Source)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown
Source)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown
Source)
at kafka.metrics.KafkaTimer.time(Unknown Source)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown
Source)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source)
at kafka.metrics.KafkaTimer.time(Unknown Source)
at kafka.consumer.SimpleConsumer.fetch(Unknown Source)
at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown
Source)
at kafka.server.AbstractFetcherThread.doWork(Unknown Source)
at kafka.utils.ShutdownableThread.run(Unknown Source)
Regards,
Yashika
________________________________________
From: Jun Rao <[email protected]>
Sent: Thursday, April 24, 2014 10:49 PM
To: [email protected]
Subject: Re: Kafka Performance Tuning
Before that error messge, the log should tell you the cause of the error.
Could you dig that out?
Thanks,
Jun
On Thu, Apr 24, 2014 at 10:12 AM, Yashika Gupta <[email protected]
> wrote:
> Hi,
>
> I am working on a POC where I have 1 Zookeeper and 2 Kafka Brokers on my
> local machine. I am running 8 sets of Kafka consumers and producers running
> in parallel.
>
> Below are my configurations:
> Consumer Configs:
> zookeeper.session.timeout.ms=120000
> zookeeper.sync.time.ms=2000
> zookeeper.connection.timeout.ms=120000
> auto.commit.interval.ms=60000
> rebalance.backoff.ms=2000
> fetch.wait.max.ms=60000
> auto.offset.reset=smallest
>
> Producer configs:
> key.serializer.class=kafka.serializer.StringEncoder
> request.required.acks=-1
> message.send.max.retries=3
> request.timeout.ms=60000
>
> I have tried various more configuration changes but I am running into the
> same exception again and again.
>
> 17.04.2014 06:31:83216 ERROR pool-5-thread-1
> kafka.producer.async.DefaultEventHandler - Failed to collate messages by
> topic, partition due to: Failed to fetch topic metadata for topic: TOPIC99
> 17.04.2014 06:31:85616 ERROR pool-4-thread-1
> kafka.producer.async.DefaultEventHandler - Failed to collate messages by
> topic, partition due to: Failed to fetch topic metadata for topic: TOPIC99
>
> I am not able to get to the root cause of the issue.
> Appreciate your help.
>
> Regards,
> Yashika
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
________________________________
NOTE: This message may contain information that is confidential, proprietary,
privileged or otherwise protected by law. The message is intended solely for
the named addressee. If received in error, please destroy and notify the
sender. Any use of this email is prohibited when received in error. Impetus
does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors,
virus, interception or interference.
