[
https://issues.apache.org/jira/browse/NIFI-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-9239:
-----------------------------
Fix Version/s: 1.15.0
Status: Patch Available (was: Open)
> Create processor to run Stateless dataflow, enabling Kafka's exactly once
> semantics
> -----------------------------------------------------------------------------------
>
> Key: NIFI-9239
> URL: https://issues.apache.org/jira/browse/NIFI-9239
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions, NiFi Stateless
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Fix For: 1.15.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There have been many requests for the ability to create a dataflow in NiFi
> that makes use of Kafka's "Exactly Once Semantics" (EOS). While there are
> benefits to being able to do so, the requirements that Kafka puts forth don't
> really work well with NiFi's architecture.
> However, it would make a lot of sense to run a NiFi dataflow using Stateless
> NiFi, in a manner that could support these EOS.
> To do so, we would need to update the consume & publish processors in order
> to support the exactly once semantics. The Kafka Consumer would need to be
> capable of not committing its offsets, and the publisher would need to
> understand that that was the case and acknowledge the offsets as part of its
> commit.
> This would require that all messages for the transaction be sent to
> PublishKafka(Record) as a single group, but that is possible with Batch
> Output mode of Process Groups.
> While this is possible, it then leaves a concern about the ease of running a
> Stateless flow with NiFi. While it can be run from command-line, we should
> also build a Processor that will be capable of fetching a dataflow (from
> file, registry, etc.) and running that flow as a Processor within NiFi. This
> offers many additional advantages also, such as the ability to perform a file
> listing in NiFi, which is persisted, and then processing it with stateless.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
