Mark Payne created NIP-11:
-----------------------------
Summary: Introduce Connector extension point
Key: NIP-11
URL: https://issues.apache.org/jira/browse/NIP-11
Project: NiFi Improvement Proposal
Issue Type: New Feature
Reporter: Mark Payne
h1. Motivation
The Apache NiFi ecosystem currently caters primarily to Data Engineers and
Software Engineers, who possess the technical expertise required to navigate
its learning curve. While NiFi offers powerful capabilities for creating and
reusing versioned data flows, these flows often necessitate substantial
modifications, particularly when adapting to diverse source/destination system
authentication strategies. A consistent demand has emerged from less technical
users for a more "self-service" solution, alongside requests for automated use
cases where users can adjust a few properties without undertaking significant
flow alterations. To address these challenges and broaden NiFi's accessibility,
this proposal introduces a new "Connector" concept.
h1. Scope
The introduction of the Connector abstraction represents a significant
undertaking that will necessitate extensive modifications across the NiFi
architecture. This initiative will require substantial additions to the
existing NiFi API and Framework. Furthermore, it will necessitate updates to
the flow.json definition and the various Versioned object models to accommodate
the new abstraction. Additionally, the user interface will require updates to
provide a cohesive and intuitive experience for configuring and managing
Connectors.
h1. Description
This proposal aims to introduce a new "Connector" abstraction and extension
point within Apache NiFi. A Connector will encapsulate a complete NiFi flow,
represented as a Process Group. This new entity will be responsible for the
lifecycle management of its constituent components, including their starting
and stopping. Crucially, a Connector will also be responsible for dynamically
updating its encapsulated flow based on the configuration it exposes to the
user. This dynamic update capability will encompass modifying Processor and
Controller Service properties, and potentially even adding or removing
components within the flow.
Unlike a Processor, which typically presents a flat list of configurable
properties, a Connector will feature a structured collection of PropertyGroups.
This organizational approach will enable developers to logically separate
different areas of configuration, such as distinct sections for "Source
configuration," "Destination configuration," "Transformations," and
"Enrichments." This structured approach enhances clarity and ease of use for
users.
Use of Custom UIs, or even more tailored reusable UI components could be used
in order to provide a clearer, more usable configuration for selecting source
entities, as well. For example, a UI might offer the ability to choose which
remote resources should be polled for data whereas we typically rely on domain
specific languages such as glob patterns or regular expressions. While this is
feasible at a Processor level, it can be made more reusable and consistent
among components with Connectors, and that information can be used to more
intelligently configure the flow for the user.
Furthermore, the Connector abstraction will provide a robust foundation for
more safely upgrading data flows from one version to another. This is achieved
by enabling the execution of custom code to gracefully migrate configurations
and flows. This capability extends beyond individual component updates,
potentially allowing for complex operations such as migrating state between
different components, which is not feasible at the individual component level.
Finally, the Connector concept lays the groundwork for building future
capabilities that are inherently meaningful at the flow level, as opposed to
the single component (Processor or Controller Service) level. Examples of such
capabilities include comprehensive validation of an entire flow, gathering
holistic metrics that provide a more complete operational picture, or providing
metrics specifically tailored for intelligent scaling decisions.
h1. Expected API
The NiFi API will be impacted significantly. While it’s difficult to entirely
suss out the full API, an initial implementation can be found at
[https://github.com/markap14/nifi-api/tree/connectors/src/main/java/org/apache/nifi/components/connector]
Given the scope, the API will be noted as experimental and may change in a
non-backward compatible manner between minor versions until the API has been
more fully validated through developing concrete implementations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
