[
https://issues.apache.org/jira/browse/NIFI-7631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan LaMothe updated NIFI-7631:
-------------------------------
Description:
I would like to begin the development of a new
nifi.content.repository.implementation that utilizes a shared file system and a
new nifi.flowfile.repository.implementation that utilizes Apache Pulsar. In our
modern, cloud-based streaming message environments, we are using high
performance shared file systems for persistent file/object storage and Apache
Pulsar for message management. Apache NiFi currently supports only local disk
(non-volatile) and in-memory (volatile) repository implementations. This means
that Apache NiFi currently performs double duty as both a workflow management
environment and a message/data management system, as there are no shared data
management or remote message management repository implementations available.
The proposed new feature development would create a new content repository
implementation designed around a shared file system and a streaming message
architecture around a distributed message bus. In essence replacing the concept
of "NiFi local queues" with "shared file system storage and Apache Pulsar
remote queues", allowing Apache Pulsar to remotely and independently manage
message state on the behalf of NiFi. In addition, this would also support NiFi
as a pure workflow management environment, decoupling it from its data
management responsibilities.
was:
I would like to begin the development of a new
nifi.content.repository.implementation for Apache Pulsar. In our modern,
cloud-based streaming message environments, we are using Apache Pulsar for all
of our persistent message/data and stream management. Apache NiFi currently
supports only local disk (non-volatile) and in-memory (volatile) content
repository implementations. This means that Apache NiFi currently performs
double duty as both a workflow management environment and a message/data
management system, as there are no remote message/data management content
repository implementations available.
The proposed new feature development would create a new content repository
implementation designed around a streaming message/data architecture, in
essence replacing the concept of a "NiFi local queue" with an "Apache Pulsar
remote queue", allowing Apache Pulsar to remotely and independently manage
messages/data on the behalf of NiFi. This would also support NiFi as a pure
workflow management environment, decoupling it from its data management
responsibilities.
> Create nifi.content.repository.implementation for SharedFileSystem and
> nifi.flowfile.repository.implementation for Apache Pulsar
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-7631
> URL: https://issues.apache.org/jira/browse/NIFI-7631
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
> Reporter: Ryan LaMothe
> Priority: Major
>
> I would like to begin the development of a new
> nifi.content.repository.implementation that utilizes a shared file system and
> a new nifi.flowfile.repository.implementation that utilizes Apache Pulsar. In
> our modern, cloud-based streaming message environments, we are using high
> performance shared file systems for persistent file/object storage and Apache
> Pulsar for message management. Apache NiFi currently supports only local disk
> (non-volatile) and in-memory (volatile) repository implementations. This
> means that Apache NiFi currently performs double duty as both a workflow
> management environment and a message/data management system, as there are no
> shared data management or remote message management repository
> implementations available.
> The proposed new feature development would create a new content repository
> implementation designed around a shared file system and a streaming message
> architecture around a distributed message bus. In essence replacing the
> concept of "NiFi local queues" with "shared file system storage and Apache
> Pulsar remote queues", allowing Apache Pulsar to remotely and independently
> manage message state on the behalf of NiFi. In addition, this would also
> support NiFi as a pure workflow management environment, decoupling it from
> its data management responsibilities.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
