I have added the configurations suggested, now my log4j appender file looks as follows
log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender log4j.appender.KAFKA.BrokerList=iakafka301p.dev.ch3.s.com:9092, iakafka302p.dev.ch3.s.com:9092,iakafka303p.dev.ch3.s.com:9092 log4j.appender.KAFKA.Topic=dev-1.0_audit log4j.appender.KAFKA.requiredNumAcks=0 log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout log4j.appender.KAFKA.layout.ConversionPattern=%m-%d log4j.appender.KAFKA.syncSend=false but still I could not see messages on queue without sleep time. I have enabled the log4 debug and following is the debug trace 2015-07-01 10:49:59,238 [main] INFO clients.producer.ProducerConfig - ProducerConfig values: value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer block.on.buffer.full = true retry.backoff.ms = 100 buffer.memory = 33554432 batch.size = 16384 metrics.sample.window.ms = 30000 metadata.max.age.ms = 300000 receive.buffer.bytes = 32768 timeout.ms = 30000 max.in.flight.requests.per.connection = 5 bootstrap.servers = [iakafka301p.dev.ch3.s.com:9092, iakafka302p.dev.ch3.s.com:9092, iakafka303p.dev.ch3.s.com:9092] metric.reporters = [] client.id = compression.type = none retries = 0 max.request.size = 1048576 send.buffer.bytes = 131072 acks = 1 reconnect.backoff.ms = 10 linger.ms = 0 metrics.num.samples = 2 metadata.fetch.timeout.ms = 60000 2015-07-01 10:49:59,515 [main] INFO kafka.utils.VerifiableProperties - Verifying properties 2015-07-01 10:49:59,550 [main] INFO kafka.utils.VerifiableProperties - Property metadata.broker.list is overridden to iakafka301p.dev.ch3.s.com:9092,iakafka302p.dev.ch3.s.com:9092, iakafka303p.dev.ch3.s.com:9092 2015-07-01 10:49:59,550 [main] INFO kafka.utils.VerifiableProperties - Property request.required.acks is overridden to 1 2015-07-01 10:49:59,550 [main] INFO kafka.utils.VerifiableProperties - Property serializer.class is overridden to kafka.serializer.StringEncoder I could not find much documentation also. Can anyone help me to find the proper documentation/code base for this issue. On Thu, Jun 18, 2015 at 6:14 AM, Manikumar Reddy <ku...@nmsworks.co.in> wrote: > You can enable producer debug log and verify. In 0.8.2.0, you can set > compressionType > , requiredNumAcks, syncSend producer config properties to log4j.xml. Trunk > build can take additional retries property . > > > Manikumar > > On Thu, Jun 18, 2015 at 1:14 AM, Madhavi Sreerangam < > madhavi.sreeran...@gmail.com> wrote: > > > I have configured my log4j with Kafka Appender.(Kafka version 0.8.2.0) > > Following are entries from my log4j file > > > > log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender > > log4j.appender.KAFKA.BrokerList=iakafka301p.dev.ch3.s.com:9092, > > iakafka302p.dev.ch3.s.com:9092,iakafka303p.dev.ch3.s.com:9092 > > log4j.appender.KAFKA.Topic=dev-1.0_audit > > log4j.appender.KAFKA.Serializer=kafka.test.AppenderStringSerializer > > log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout > > log4j.appender.KAFKA.layout.ConversionPattern=%m-%d > > > > Kafka is configured with 3 servers, 3 partitions and 3 replicas. > > I have created a test method to publish the messages to kafka topic as > > follows > > > > private void testKAFKAlog(int noOfMessages){ > > for(int i=0; i < noOfMessages; i++){ > > KAFKA_LOG.info("Test Message: " + i); > > } > > } > > I could not see any messages published into the topic. Then I have > modified > > the test method to introduce some wait between the requests as follows > > > > private void testKAFKAlog(int noOfMessages){ > > for(int i=0; i < noOfMessages; i++){ > > try { > > Thread.sleep(10); > > } catch (InterruptedException e) { > > e.printStackTrace(); > > } > > KAFKA_LOG.info("Test Message: " + i); > > } > > } > > > > Then all the messages started publishing. I did this exercise couple of > > times with and without sleep between the requests. Messages got published > > only when there is sleep in between the requests. > > Does any one help me here, what is wrong with the configurations I am > > using. (I can't afford 10ms wait for each message, as my application logs > > few Million messages for each run). > > Is there any way that I can override the default ProducerConfig for log4j > > kafka appender. > > >