[
https://issues.apache.org/jira/browse/NIFI-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17817180#comment-17817180
]
Pierre Villard commented on NIFI-12783:
---------------------------------------
Some observations regarding this discussion.
* dropping flow files is a rather unusual use case to be honest, this is
something that not a lot of users are doing AFAIK
* routing the failure relationship back to the processor is not really a good
practice anymore, the retry should be directly set on the relationship (this
has been added recently in the framework and can be set directly on the
relationships tab of the processor's configuration view)
* Kafka processors are used very extensively by the community so any change
should preserve the current behavior. We may consider a breaking change only
for NiFi 2.x at this point but I'm not sure I'd be ok with it
Some processors like PutKudu are specifying a property "Failure strategy" where
behavior on failure can be specified: rolling back versus routing to failure.
IMO the best approach would be to expose a similar property with a default of
"rolling back / yielding" to preserve current behavior.
One of the use cases where rolling back is absolutely key is when ordering must
be preserved which is a very valid use case when sending data to Kafka (CDC use
cases for example).
Thoughts?
> Kafka Producer Processors do not route in failure queue on timeout
> ------------------------------------------------------------------
>
> Key: NIFI-12783
> URL: https://issues.apache.org/jira/browse/NIFI-12783
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.23.2
> Reporter: Denis Jakupovic
> Priority: Major
>
> Hi,
> the Kafka producer processors do not route the FlowFiles on a timeout e.g.
> into the failure connection. They are yielded in the incomming connection.
> You can see the behaviour here e.g.:
> [https://stackoverflow.com/questions/71460008/apache-nifi-publishkafka-timeout-exception]
> I think this is a design flaw. I have a use case where messages should be
> dropped after a specific configurable time. If the messages are yielded in
> the incomming queue they are always published when the kafka broker are
> available again. I know I can set the expiration time in secs or mins in the
> incomming queue but it is not dynamically configurable because no attributes
> are allowed.
> Best
> Denis
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
