[
https://issues.apache.org/jira/browse/NIFI-7631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan LaMothe updated NIFI-7631:
-------------------------------
Description:
I would like to begin the development of a new
nifi.content.repository.implementation that utilizes a shared filesystem and a
new nifi.flowfile.repository.implementation that utilizes Apache Pulsar. In our
modern, cloud-based streaming message environments, we are using high
performance shared filesystems for persistent file/object storage and Apache
Pulsar for message management. Apache NiFi currently supports only local disk
(non-volatile) and in-memory (volatile) repository implementations. This means
that Apache NiFi currently performs double duty as both a workflow management
environment and a message/data management system, as there are no shared data
management or remote message management repository implementations available.
The proposed new feature development would create a new content repository
implementation designed around a shared filesystem and a streaming message
architecture around a distributed message bus. In essence, replacing the
concept of "NiFi local queues" with "shared filesystem storage and remote
flowfile queues". This work will support NiFi as a pure workflow management
environment, decoupling it from its current data and message management
responsibilities.
was:
I would like to begin the development of a new
nifi.content.repository.implementation that utilizes a shared file system and a
new nifi.flowfile.repository.implementation that utilizes Apache Pulsar. In our
modern, cloud-based streaming message environments, we are using high
performance shared file systems for persistent file/object storage and Apache
Pulsar for message management. Apache NiFi currently supports only local disk
(non-volatile) and in-memory (volatile) repository implementations. This means
that Apache NiFi currently performs double duty as both a workflow management
environment and a message/data management system, as there are no shared data
management or remote message management repository implementations available.
The proposed new feature development would create a new content repository
implementation designed around a shared file system and a streaming message
architecture around a distributed message bus. In essence replacing the concept
of "NiFi local queues" with "shared file system storage and Apache Pulsar
remote queues", allowing Apache Pulsar to remotely and independently manage
message state on the behalf of NiFi. In addition, this would also support NiFi
as a pure workflow management environment, decoupling it from its data
management responsibilities.
> Create nifi.content.repository.implementation for SharedFileSystem and
> nifi.flowfile.repository.implementation for Apache Pulsar
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-7631
> URL: https://issues.apache.org/jira/browse/NIFI-7631
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
> Reporter: Ryan LaMothe
> Priority: Major
>
> I would like to begin the development of a new
> nifi.content.repository.implementation that utilizes a shared filesystem and
> a new nifi.flowfile.repository.implementation that utilizes Apache Pulsar. In
> our modern, cloud-based streaming message environments, we are using high
> performance shared filesystems for persistent file/object storage and Apache
> Pulsar for message management. Apache NiFi currently supports only local disk
> (non-volatile) and in-memory (volatile) repository implementations. This
> means that Apache NiFi currently performs double duty as both a workflow
> management environment and a message/data management system, as there are no
> shared data management or remote message management repository
> implementations available.
> The proposed new feature development would create a new content repository
> implementation designed around a shared filesystem and a streaming message
> architecture around a distributed message bus. In essence, replacing the
> concept of "NiFi local queues" with "shared filesystem storage and remote
> flowfile queues". This work will support NiFi as a pure workflow management
> environment, decoupling it from its current data and message management
> responsibilities.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
