[
https://issues.apache.org/jira/browse/NIFI-7631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17156359#comment-17156359
]
Andy LoPresto commented on NIFI-7631:
-------------------------------------
We welcome feature development proposals, but I am slightly confused by some of
the comments here. The content, flowfile, and provenance repositories are
designed to be internal data structures for NiFi with no external interaction,
and NiFi's "data management responsibilities" are inherent to its function as a
flow manager. Without certain guarantees about the persistence/transience and
availability of the data in these repositories, NiFi would hardly function,
much less be able to route data between different components in the flow graph
in a performant and reliable manner. For message persistence, NiFi currently
consumes from and produces to systems like Pulsar, Kafka, etc., but this
interaction is an external communication through the flow rather than the
internal mechanics of the framework.
Have you read the [Apache NiFi
In-Depth|https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html]
document, which goes into further detail around the implementation and design
principles of these repositories and how they interact with the NiFi system?
> Create a nifi.content.repository.implementation for Apache Pulsar
> ------------------------------------------------------------------
>
> Key: NIFI-7631
> URL: https://issues.apache.org/jira/browse/NIFI-7631
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
> Reporter: Ryan LaMothe
> Priority: Major
>
> I would like to begin the development of a new
> nifi.content.repository.implementation for Apache Pulsar. In our modern,
> cloud-based streaming message environments, we are using Apache Pulsar for
> all of our persistent message/data and stream management. Apache NiFi
> currently supports only local disk (non-volatile) and in-memory (volatile)
> content repository implementations. This means that Apache NiFi currently
> performs double duty as both a workflow management environment and a
> message/data management system, as there are no remote message/data
> management content repository implementations available.
> The proposed new feature development would create a new content repository
> implementation designed around a streaming message/data architecture, in
> essence replacing the concept of a "NiFi local queue" with an "Apache Pulsar
> remote queue", allowing Apache Pulsar to remotely and independently manage
> messages/data on the behalf of NiFi. This would also support NiFi as a pure
> workflow management environment, decoupling it from its data management
> responsibilities.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
