Sam Hjelmfelt created NIFI-5922:
-----------------------------------
Summary: NiFi-Fn, an alternative runtime for NiFi flows
Key: NIFI-5922
URL: https://issues.apache.org/jira/browse/NIFI-5922
Project: Apache NiFi
Issue Type: Improvement
Reporter: Sam Hjelmfelt
NiFi-Fn is a library for running NiFi flows as stateless functions. It provides
similar delivery guarantees as NiFi without the need for on-disk repositories
by waiting to confirm receipt ofincoming data until it has been written to the
destination. This is similar to Storm’s acking mechanism and Spark’s interface
for committing Kafka offsets, except that in NiFi-Fn, this is completely
handled by the framework while still supporting all NiFi processors and
controller services natively without change.This results in the ability to run
NiFi flows as ephemeral, stateless functions and should be able to rival
MirrorMaker, Distcp, and Scoop for performance,efficiency, and scalability
while leveraging the vast library of NiFi processors and the NiFi UI for
building custom flows.
By leveraging container engines (e.g.YARN, Kubernetes), long-running NiFi-Fn
flows can be deployed that take full advantage of the platform’s scale and
multi-tenancy features. By leveraging Function as a Service engines (FaaS)
(e.g. AWS Lambda, Apache OpenWhisk), NiFi-Fn flows can be attached to event
sources (or just cron) for event-driven data movement where flows only run when
triggered and pricing is measured at the 100ms granularity. By combining the
two, large-scale batch processing could also be performed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
