[
https://issues.apache.org/jira/browse/KAFKA-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176626#comment-15176626
]
Jiangjie Qin commented on KAFKA-3236:
-------------------------------------
[~tgraves] I remember there were some discussions about the requirement as you
described. i.e. you want to block to first get the metadata but don't want to
block afterwards. Unfortunately I forgot what exact the conclusion was from
that discussion. For your use case, would the following solution work?
1. set {{max.block.ms = 0}}
2. Let the metadata discovery thread call partitionsFor() and catch the timeout
exception in a while loop until it gets the metadata.
3. let the actual producing thread start produce.
Given your metadata discovery thread has to call partitionsFor() on different
producers, I assume you probably don't want it to be blocked on one of the
producer either, right?
I'm also wondering that if blocking on metadata refreshing during sending is
rare and you are OK with that occasional short blocking behavior, do you expect
buffer full to be frequent and you are not OK with that?
> Honor Producer Configuration "block.on.buffer.full"
> ---------------------------------------------------
>
> Key: KAFKA-3236
> URL: https://issues.apache.org/jira/browse/KAFKA-3236
> Project: Kafka
> Issue Type: Improvement
> Components: producer
> Affects Versions: 0.9.0.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
>
> In Kafka-0.9, "max.block.ms" is used to control how long the following
> methods will block.
> KafkaProducer.send() when
> * Buffer is full
> * Metadata is unavailable
> KafkaProducer.partitionsFor() when
> * Metadata is unavailable
> However when "block.on.buffer.full" is set to false, "max.block.ms" is in
> effect whenever a buffer is requested/allocated from the Producer BufferPool.
> Instead it should throw a BufferExhaustedException without waiting for
> "max.block.ms"
> This is particulary useful if a producer application does not wish to block
> at all on KafkaProducer.send() . We avoid waiting on KafkaProducer.send()
> when metadata is unavailable by invoking send() only if the producer instance
> has fetched the metadata for the topic in a different thread using the same
> producer instance. However "max.block.ms" is still required to specify a
> timeout for bootstrapping the metadata fetch.
> We should resolve this limitation by decoupling "max.block.ms" and
> "block.on.buffer.full".
> * "max.block.ms" will be used exclusively for fetching metadata when
> "block.on.buffer.full" = false (in pure non-blocking mode )
> * "max.block.ms" will be applicable to both fetching metadata as well as
> buffer allocation when "block.on.buffer.full = true
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
