[
https://issues.apache.org/jira/browse/NIP-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18017134#comment-18017134
]
Matt Burgess commented on NIP-11:
---------------------------------
For full flexibility can we consider Source Connectors and Target Connectors
vs. End to End? It would unlock metadata-driven architectures of NiFi's prime
tenant of universal data distribution. I say this knowing our history of having
ConvertXtoY processors then implementing the Record-based components, as well
as Lookup controller services that can route to the "real" CSs that perform the
desired operation. Between these three approaches we could still offer more
flexible flow design without growing the XtoY matrix, and keep the overall
flows super-simple but also super-flexible. Thoughts?
> Introduce Connector extension point
> -----------------------------------
>
> Key: NIP-11
> URL: https://issues.apache.org/jira/browse/NIP-11
> Project: NiFi Improvement Proposal
> Issue Type: New Feature
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> h1. Motivation
> The Apache NiFi ecosystem currently caters primarily to Data Engineers and
> Software Engineers, who possess the technical expertise required to navigate
> its learning curve. While NiFi offers powerful capabilities for creating and
> reusing versioned data flows, these flows often necessitate substantial
> modifications, particularly when adapting to diverse source/destination
> system authentication strategies. A consistent demand has emerged from less
> technical users for a more "self-service" solution, alongside requests for
> automated use cases where users can adjust a few properties without
> undertaking significant flow alterations. To address these challenges and
> broaden NiFi's accessibility, this proposal introduces a new "Connector"
> concept.
> h1. Scope
> The introduction of the Connector abstraction represents a significant
> undertaking that will necessitate extensive modifications across the NiFi
> architecture. This initiative will require substantial additions to the
> existing NiFi API and Framework. Furthermore, it will necessitate updates to
> the flow.json definition and the various Versioned object models to
> accommodate the new abstraction. Additionally, the user interface will
> require updates to provide a cohesive and intuitive experience for
> configuring and managing Connectors.
> h1. Description
> This proposal aims to introduce a new "Connector" abstraction and extension
> point within Apache NiFi. A Connector will encapsulate a complete NiFi flow,
> represented as a Process Group. This new entity will be responsible for the
> lifecycle management of its constituent components, including their starting
> and stopping. Crucially, a Connector will also be responsible for dynamically
> updating its encapsulated flow based on the configuration it exposes to the
> user. This dynamic update capability will encompass modifying Processor and
> Controller Service properties, and potentially even adding or removing
> components within the flow.
>
> Unlike a Processor, which typically presents a flat list of configurable
> properties, a Connector will feature a structured collection of
> PropertyGroups. This organizational approach will enable developers to
> logically separate different areas of configuration, such as distinct
> sections for "Source configuration," "Destination configuration,"
> "Transformations," and "Enrichments." This structured approach enhances
> clarity and ease of use for users.
>
> Use of Custom UIs, or even more tailored reusable UI components could be used
> in order to provide a clearer, more usable configuration for selecting source
> entities, as well. For example, a UI might offer the ability to choose which
> remote resources should be polled for data whereas we typically rely on
> domain specific languages such as glob patterns or regular expressions. While
> this is feasible at a Processor level, it can be made more reusable and
> consistent among components with Connectors, and that information can be used
> to more intelligently configure the flow for the user.
>
> Furthermore, the Connector abstraction will provide a robust foundation for
> more safely upgrading data flows from one version to another. This is
> achieved by enabling the execution of custom code to gracefully migrate
> configurations and flows. This capability extends beyond individual component
> updates, potentially allowing for complex operations such as migrating state
> between different components, which is not feasible at the individual
> component level.
>
> Finally, the Connector concept lays the groundwork for building future
> capabilities that are inherently meaningful at the flow level, as opposed to
> the single component (Processor or Controller Service) level. Examples of
> such capabilities include comprehensive validation of an entire flow,
> gathering holistic metrics that provide a more complete operational picture,
> or providing metrics specifically tailored for intelligent scaling decisions.
>
> h1. Expected API
> The NiFi API will be impacted significantly. While it’s difficult to entirely
> suss out the full API, an initial implementation can be found at
> [https://github.com/markap14/nifi-api/tree/connectors/src/main/java/org/apache/nifi/components/connector]
>
> Given the scope, the API will be noted as experimental and may change in a
> non-backward compatible manner between minor versions until the API has been
> more fully validated through developing concrete implementations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
