Hi,
We are replacing the current data pipeline that sends messages to the tomcat
servlet that in turn writes to files and then pushes to HDFS to process.
Instead of replacing it entirely at first, we are sourcing the same messages
and send it to Kafka broker through its producers invoked by the same servlet.
We were trying to benchmark the insertion of 2B records under the following
configurations, but got an error and not sure why!!!!
And why is there a warning on zk.connectiontimeout.ms below as we do not
override it and shouldn’t it be default to 6000.
— 10 topics and 5 partitions per topic
--10 producers, each having eReporter
--each of the 10 producers will execute the same message files (about 100) 2m
times through eReporter
--10 brokers
--2 zookeepers
--each message having 2 transaction(size of message file = 725 bytes)
--Test start time : Fri Apr 4 05:44:01 PDT 2014 - it worked fine and records
were also getting populated in the topics as expected but
--At 6:23:27 got the following error :
The error in the eReporter catalina.out:
upanda >> topic ::imessage
6 23 27
Residue::3
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError:
kafka/producer/async/ProducerSendThread$$anonfun$run$1
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Caused by: java.lang.ClassNotFoundException:
kafka.producer.async.ProducerSendThread$$anonfun$run$1
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1678)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1523)
... 1 more
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError:
kafka/producer/async/ProducerSendThread$$anonfun$run$1
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError:
kafka/producer/async/ProducerSendThread$$anonfun$run$1
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError:
kafka/producer/async/ProducerSendThread$$anonfun$run$1
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError:
kafka/producer/async/ProducerSendThread$$anonfun$run$1
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError:
kafka/producer/async/ProducerSendThread$$anonfun$run$1
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
Exception in thread "ProducerSendThread-upanda" java.lang.NoClassDefFoundError:
kafka/producer/async/ProducerSendThread$$anonfun$run$1
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46)
The message in the eReporter log file :
2014-04-04 06:22:55,417 [pool-1-thread-10] INFO VerifiableProperties:68 -
Verifying properties
2014-04-04 06:22:55,417 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property batch.num.messages is overridden to 500
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property client.id is overridden to upanda
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property message.send.max.retries is overridden to 3
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property metadata.broker.list is overridden to nk11p16ic-erptapp005:3001
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property partitioner.class is overridden to kafka.producer.DefaultPartitioner
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property producer.type is overridden to async
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property queue.buffering.max.messages is overridden to 10000
2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property queue.buffering.max.ms is overridden to 5000
2014-04-04 06:22:55,419 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property queue.enqueue.timeout.ms is overridden to -1
2014-04-04 06:22:55,419 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property request.required.acks is overridden to 1
2014-04-04 06:22:55,419 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property retry.backoff.ms is overridden to 100
2014-04-04 06:22:55,419 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property serializer.class is overridden to kafka.serializer.StringEncoder
2014-04-04 06:22:55,419 [pool-1-thread-10] INFO VerifiableProperties:68 -
Property topic.metadata.refresh.interval.ms is overridden to 600000
2014-04-04 06:22:55,419 [pool-1-thread-10] WARN VerifiableProperties:83 -
Property zk.connectiontimeout.ms is not valid
2014-04-04 06:22:56,048 [ProducerSendThread-upanda] INFO ClientUtils$:68 -
Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 with
correlation id 0 for 1 topic(s) Set(imessage)
2014-04-04 06:22:56,775 [ProducerSendThread-upanda] INFO ClientUtils$:68 -
Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 with
correlation id 0 for 1 topic(s) Set(imessage)
2014-04-04 06:22:57,504 [ProducerSendThread-upanda] INFO ClientUtils$:68 -
Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 with
correlation id 0 for 1 topic(s) Set(imessage)
2014-04-04 06:22:58,234 [ProducerSendThread-upanda] INFO ClientUtils$:68 -
Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 with
correlation id 0 for 1 topic(s) Set(imessage)
2014-04-04 06:22:58,980 [ProducerSendThread-upanda] INFO ClientUtils$:68 -
Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 with
correlation id 0 for 1 topic(s) Set(imessage)
2014-04-04 06:22:59,693 [ProducerSendThread-upanda] INFO ClientUtils$:68 -
Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 with
correlation id 0 for 1 topic(s) Set(imessage)
2014-04-04 06:23:00,422 [ProducerSendThread-upanda] INFO ClientUtils$:68 -
Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 with
correlation id 0 for 1 topic(s) Set(imessage)
Thanks,
Maung
