Do you have the right version of scala jar? Thanks,
Jun On Fri, Apr 4, 2014 at 11:34 AM, Maung Than <[email protected]> wrote: > Hi, > > We are replacing the current data pipeline that sends messages to the > tomcat servlet that in turn writes to files and then pushes to HDFS to > process. > Instead of replacing it entirely at first, we are sourcing the same > messages and send it to Kafka broker through its producers invoked by the > same servlet. > > We were trying to benchmark the insertion of 2B records under the > following configurations, but got an error and not sure why!!!! > And why is there a warning on zk.connectiontimeout.ms below as we do not > override it and shouldn't it be default to 6000. > > -- 10 topics and 5 partitions per topic > --10 producers, each having eReporter > --each of the 10 producers will execute the same message files (about 100) > 2m times through eReporter > --10 brokers > --2 zookeepers > --each message having 2 transaction(size of message file = 725 bytes) > --Test start time : Fri Apr 4 05:44:01 PDT 2014 - it worked fine and > records were also getting populated in the topics as expected but > --At 6:23:27 got the following error : > > The error in the eReporter catalina.out: > > upanda >> topic ::imessage > 6 23 27 > Residue::3 > Exception in thread "ProducerSendThread-upanda" > java.lang.NoClassDefFoundError: > kafka/producer/async/ProducerSendThread$$anonfun$run$1 > at > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46) > Caused by: java.lang.ClassNotFoundException: > kafka.producer.async.ProducerSendThread$$anonfun$run$1 > at > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1678) > at > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1523) > ... 1 more > Exception in thread "ProducerSendThread-upanda" > java.lang.NoClassDefFoundError: > kafka/producer/async/ProducerSendThread$$anonfun$run$1 > at > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46) > Exception in thread "ProducerSendThread-upanda" > java.lang.NoClassDefFoundError: > kafka/producer/async/ProducerSendThread$$anonfun$run$1 > at > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46) > Exception in thread "ProducerSendThread-upanda" > java.lang.NoClassDefFoundError: > kafka/producer/async/ProducerSendThread$$anonfun$run$1 > at > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46) > Exception in thread "ProducerSendThread-upanda" > java.lang.NoClassDefFoundError: > kafka/producer/async/ProducerSendThread$$anonfun$run$1 > at > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46) > Exception in thread "ProducerSendThread-upanda" > java.lang.NoClassDefFoundError: > kafka/producer/async/ProducerSendThread$$anonfun$run$1 > at > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46) > Exception in thread "ProducerSendThread-upanda" > java.lang.NoClassDefFoundError: > kafka/producer/async/ProducerSendThread$$anonfun$run$1 > at > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:46) > > The message in the eReporter log file : > > 2014-04-04 06:22:55,417 [pool-1-thread-10] INFO VerifiableProperties:68 - > Verifying properties > 2014-04-04 06:22:55,417 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property batch.num.messages is overridden to 500 > 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property client.id is overridden to upanda > 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property message.send.max.retries is overridden to 3 > 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property metadata.broker.list is overridden to nk11p16ic-erptapp005:3001 > 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property partitioner.class is overridden to > kafka.producer.DefaultPartitioner > 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property producer.type is overridden to async > 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property queue.buffering.max.messages is overridden to 10000 > 2014-04-04 06:22:55,418 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property queue.buffering.max.ms is overridden to 5000 > 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property queue.enqueue.timeout.ms is overridden to -1 > 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property request.required.acks is overridden to 1 > 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property retry.backoff.ms is overridden to 100 > 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property serializer.class is overridden to kafka.serializer.StringEncoder > 2014-04-04 06:22:55,419 [pool-1-thread-10] INFO VerifiableProperties:68 - > Property topic.metadata.refresh.interval.ms is overridden to 600000 > 2014-04-04 06:22:55,419 [pool-1-thread-10] WARN VerifiableProperties:83 - > Property zk.connectiontimeout.ms is not valid > 2014-04-04 06:22:56,048 [ProducerSendThread-upanda] INFO ClientUtils$:68 > - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 > with correlation id 0 for 1 topic(s) Set(imessage) > 2014-04-04 06:22:56,775 [ProducerSendThread-upanda] INFO ClientUtils$:68 > - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 > with correlation id 0 for 1 topic(s) Set(imessage) > 2014-04-04 06:22:57,504 [ProducerSendThread-upanda] INFO ClientUtils$:68 > - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 > with correlation id 0 for 1 topic(s) Set(imessage) > 2014-04-04 06:22:58,234 [ProducerSendThread-upanda] INFO ClientUtils$:68 > - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 > with correlation id 0 for 1 topic(s) Set(imessage) > 2014-04-04 06:22:58,980 [ProducerSendThread-upanda] INFO ClientUtils$:68 > - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 > with correlation id 0 for 1 topic(s) Set(imessage) > 2014-04-04 06:22:59,693 [ProducerSendThread-upanda] INFO ClientUtils$:68 > - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 > with correlation id 0 for 1 topic(s) Set(imessage) > 2014-04-04 06:23:00,422 [ProducerSendThread-upanda] INFO ClientUtils$:68 > - Fetching metadata from broker id:0,host:nk11p16ic-erptapp005,port:3001 > with correlation id 0 for 1 topic(s) Set(imessage) > > Thanks, > Maung
