[
https://issues.apache.org/jira/browse/NIFI-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17817253#comment-17817253
]
Denis Jakupovic commented on NIFI-12783:
----------------------------------------
Hi [~pvillard]
yes yielding is a necessity for a lot of people. The current behavior should be
preserved as an option in the processor e.g. failure strategy. Currently there
is rollback and route to failure in the kafka processors but they are not
working as expected on connectivity issues I guess.
However what I meant with a standard approach in NiFi: The relationships are
seen as a reaction feature on specific events - do that on retry and do that on
failure etc. The expected behavior on a timeout should be route in failure, so
retry (penalize and/or yield), drop or route into another processor or even
retry again with the RetryFlowFile processor.
If Kafka has a downtime for several hours the queues are back pressured. With
the current behavior the FF are stuck in the inbound queue and there is no
reaction possible e.g. store in s3 or even in another kafka cluster. There a
lot of use cases where data needs to be transmitted immediately where ordering
is not important.
Best
Denis
> Kafka Producer Processors do not route in failure queue on timeout
> ------------------------------------------------------------------
>
> Key: NIFI-12783
> URL: https://issues.apache.org/jira/browse/NIFI-12783
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.23.2
> Reporter: Denis Jakupovic
> Priority: Major
>
> Hi,
> the Kafka producer processors do not route the FlowFiles on a timeout e.g.
> into the failure connection. They are yielded in the incomming connection.
> You can see the behaviour here e.g.:
> [https://stackoverflow.com/questions/71460008/apache-nifi-publishkafka-timeout-exception]
> I think this is a design flaw. I have a use case where messages should be
> dropped after a specific configurable time. If the messages are yielded in
> the incomming queue they are always published when the kafka broker are
> available again. I know I can set the expiration time in secs or mins in the
> incomming queue but it is not dynamically configurable because no attributes
> are allowed.
> Best
> Denis
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
