The stack trace you showed shouldn't be going beyond the isDebugEnabled()
check if your log4j properties is configured at a higher log level.
Regarding your producer dropping messages, have you checked the produce
metrics ("kafka.network":name={Produce-TotalTimeMs",type="RequestMetrics")
on the broker to see if the bottleneck is on the server or not?On Sun, Dec 14, 2014 at 9:31 PM, Rajiv Kurian <ra...@signalfuse.com> wrote: > > I'll try it and let you guys know. Is there anything that comes to mind > from these log messages though? Why would there be so many log messages? > Would you suggest doing something else to find out why things are working > so poorly? I am worried about making the risky transition to the beta > producer client and finding out that things don't improve at all. Another > thing that was peculiar was that only one of my producers (out of 3, each > on a separate host) were dropping so many messages. > > On Sun, Dec 14, 2014 at 4:58 PM, Neha Narkhede <n...@confluent.io> wrote: > > > Thanks for reporting the issue, Rajiv. Since we are actively phasing out > > the old client, it will be very helpful to know what the behavior on the > > new client is. > > > > > > On Fri, Dec 12, 2014 at 8:12 PM, Rajiv Kurian <ra...@signalfuse.com> > > wrote: > > > > > > I am using the kafka java api async client (the one that wraps the > Scala > > > client). It's dropping quite a bit of data due to the queue being full, > > and > > > the process ends up taking a lot of cpu too. > > > > > > I am posting to a topic with 1024 partitions (across 3 brokers) - maybe > > the > > > high number of brokers is one of the issues. Profiling with YourKit > > showed > > > 37% of my CPU being spent on > > kafka.producer.async.ProducerSendThread.run(). > > > So seems like the producer is not able to keep up with my application > and > > > starts dropping. When I expand this waterfall in YourKit, I see that > 23% > > > total (not out of the 37%) is being spent on logging! Something like > > this: > > > > > > > > > > > > kafka.producer.BrokerPartitionInfo$$anonfun$getBrokerPartitionInfo$2.apply(PartitionMetadata) > > > ->kafka.producer.BrokerPartitionInfo.debug(Function0) > > > ->kafka.utils.Logging$class.debug(Logging, Function0) > > > ->org.apache.log4j.Category.isDebugEnabled() > > > ->... (a bunch of other things that finally break down into) > > > ->LoggerContext.java:252 > > > > > > > > > ch.qos.logback.classic.spi.TurboFilterList.getTurboFilterChainDecision(Marker, > > > Logger, Level, String, Object[], Throwable) > > > > > > I am not sure what's going on here. When I look at my process log, none > > of > > > these messages are actually logged (probably because of the log level). > > > Further I don't see anything very suspicious on the broker logs. They > are > > > at 60-70% cpu. > > > > > > I am planning to try the new Java beta producer client, but I am afraid > > > something deeper is going on here, that might not be solved by > switching > > to > > > the newer client. > > > > > > > > > -- > > Thanks, > > Neha > > > -- Thanks, Neha
