[
https://issues.apache.org/jira/browse/KAFKA-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14228124#comment-14228124
]
devendra tagare commented on KAFKA-1789:
----------------------------------------
Hi,
The actual enqueue is taking more than 20ms.The message queue is not full in
these cases since there are not QueueFullExcetions.
we have set queue.buffering.max.messages is the default 10000 and
batch.num.messages = 100.
Can increasing the topic.metadata.refresh.interval.ms help if the broker is
slow?Atleast the broker would be polled fewer times for a metadata refresh.
Presently its set to topic.metadata.refresh.interval.ms = 30000.
~Dev
> Issue with Async producer
> -------------------------
>
> Key: KAFKA-1789
> URL: https://issues.apache.org/jira/browse/KAFKA-1789
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: devendra tagare
> Priority: Critical
>
> Hi,
> We are using an async producer to send data to a kafka cluster.The event rate
> at peak is around 250 events/second of size 25KB each.
> In the producer code base we have added specific debug statements to capture
> the time taken to create a producer,create a keyed message with a byte
> payload & send the message.
> We have added the below properties to the producerConfig
> queue.enqueue.timeout.ms=20
> send.buffer.bytes=1024000
> topic.metadata.refresh.interval.ms=30000
> Based on the documentation, producer.send() queues the message on the async
> producer's queue.
> So, ideally if the queue is full then the enqueue operation should result in
> an kafka.common.QueueFullException in 20 ms.
> The logs indicate that the enqueue operation is taking more than 20ms (takes
> around 250ms) without throwing any exceptions.
> Is there any other property that could conflict with queue.enqueue.timeout.ms
> which is causing this behavior ?
> Or is it possible that the queue is not full & yet the producer.send() call
> is still taking around 200ms under peak load ?
> Also, could you suggest any other alternatives so that we can either enforce
> a timeout or throw an exception in-case the async producer is taking more
> than a specified amount of time.
> Regards,
> Dev
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
