[
https://issues.apache.org/jira/browse/NIFI-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14987181#comment-14987181
]
Andre commented on NIFI-1097:
-----------------------------
Joe,
As much as I would like NIFI-1095 to be implemented as soon as possible (I am
trying to get my head around Java so I can contribute that code) I am not sure
the behaviour present should be a *critical* concern (as long it is correctly
documented).
The reason I mentioned that is because the implementation of Kafka's async
calls seem to cursed with message delivery to the void, for example:
Shopify's Sarama (GoLang) deal with failure by spitting the message back to an
error channel (similar to what I imagine the new JAVA API is doing), but even
if you can get the details of a failed message delivery, I wonder how
challenging would be for NiFi to rely on Kafka's Async Producer for reliable
delivery when it documentation is quite clear in that space for blocking and
errors still exist around its buffering mechanism:
{quote}
This buffer has a hard limit on it's size, which is controlled by the
configuration total.memory.bytes. If send() is called faster than the I/O
thread can transfer data to the brokers the buffer will eventually run out of
space. The default behavior in this case is to block the send call until the
I/O thread catches up and more buffer space is available. However in cases
where non-blocking usage is desired the setting block.on.buffer.full=false will
cause the producer to instead throw an exception when buffer memory is
exhausted.
{quote}
I would rather draw a line and document that when considering reliable delivery
vs. speed on should chose:
If reliability is important, use Synchronous and let NIFI control buffering,
backpressure, etc. Performance will be impacted and that should be dealt by
either merging messages prior to send or via higher parallelism.
If speed is important, go nuts and use the Asynchronous producer, message
losses may occur.
Having said that, I'm all pro for moving to the new API. :-)
> PutKafka should be changed to use the new 0.8.2 Producer API
> ------------------------------------------------------------
>
> Key: NIFI-1097
> URL: https://issues.apache.org/jira/browse/NIFI-1097
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Critical
> Fix For: 0.4.0
>
>
> Currently, PutKafka uses the old Producer API, which provides several
> limitations. The most critical of these limitations is that the asynchronous
> send does not provide an indication of its success. As a result, when
> PutKafka is configured to send asynchronously, it will always route the
> FlowFIle to 'success' even if it does not send the data to Kafka.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
