[
https://issues.apache.org/jira/browse/NIP-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18031268#comment-18031268
]
Kevin Doran commented on NIP-14:
--------------------------------
Thanks David. I'm fine with leaving SCTP out of the party for now, as my
understanding (admittedly limited) is that it is mostly used in telecom
networking hardware, where NiFi is unlikely to be deployed.
I agree Application Protocols cannot use a Java Enum. I actually have them as a
just string constants in my experimental branch, but on second glance, I think
that it is more confusing than helpful to developers. If we decide to go
forward with this proposal, I will remove that class in favor of just using
Strings and docs and examples for developers on choosing application protocol
values.
> Support Dynamic Discovery of Data Ingress Ports
> -----------------------------------------------
>
> Key: NIP-14
> URL: https://issues.apache.org/jira/browse/NIP-14
> Project: NiFi Improvement Proposal
> Issue Type: Improvement
> Reporter: Kevin Doran
> Assignee: Kevin Doran
> Priority: High
>
> *Motivation*
> Increasingly, NiFi clusters are being deployed into environments where
> network traffic is managed through a non-NiFi component, such as a gateway,
> ingress controller, load balancer, or reverse proxy. These include cloud and
> containerized deployment environments. NiFi infrastructure is sometimes
> managed via infrastructure as code (IaC) frameworks as part of a larger
> system.
> Additionally, many NiFi operators desire to deploy flow definitions
> programmatically via automated deployment pipelines. Flow versioning and
> promotion of flow definitions from non-production to production NiFi clusters
> has become a best practice in the community.
> The combination of these factors leads to situations where it would be
> advantageous if data ingress ports defined as part of a flow definition (eg,
> a ListenHTTP processor) could be dynamically discovered by infrastructure and
> deployment software in order to automatically configure managed networking
> components responsible for managing ingress traffic to NiFi clusters.
> *Scope and Description*
> There are currently 9 processors and 1 controller service included in the
> NiFi source code distribution that define Listen Ports:
> - HandleHTTPRequest
> - ListenFTP
> - ListenHTTP
> - ListenOTLP
> - ListenSyslog
> - ListenTCP
> - ListenTrapSNMP
> - ListenUDP
> - ListenUDPRecord
> - JettyWebSocketServer (CS)
> (Note, that ListenSlack uses the Listen* naming convention, but this
> processor actually initiates a two-way TCP connection with a Slack workspace
> in order to start receiving events. It does not create a server process to
> accept inbound connections.)
> Of course, it is possible that community members have (or could) develop
> custom NiFi extensions that create Listen Ports. Apache NiFi components
> usually serve as example, reference implementations in these cases. The goal
> of this feature is to introduce standard interfaces for declaring data
> ingress ports in components as well as framework mechanisms for a standard
> discovery process of such ports, making it possible to support Apache NiFi
> provided components as well as external, third-party extensions.
> All of the example components listed above are ConfigurableComponents that
> define a Port property the establishes the numbered port that is bound to in
> order to listen to payloads and connections from external clients. A new
> Property Descriptor field in nifi-api would be a natural place to annotate
> that a property defines a Listen Port. This would allow discovery of Listen
> Ports both at runtime by the framework, as well as in flow definitions, which
> is very advantageous as it allows determining network ingress requirements
> based solely on the static flow definition before it has even been deployed
> into a NiFi runtime environment.
> In addition to additions to Property Descriptors in NiFi API described above,
> the NiFi Framework and NiFi REST API would be modified to dynamically
> discover configurable components (ie, processors and controller services)
> containing Listen Ports and list those components along with their current
> configuration programmatically.
> When looking at how a Listen Port should be defined, the most important
> aspect is the layer 4 transport protocol, as that is usually the most
> relevant information required to automatically configure external networking
> components such as gateways, ingress controllers, load balancers, and
> proxies. Of secondary importance is the layer 7 application protocol, if any.
> Looking at existing projects that have solved similar problems, we can take
> Kubernetes as an example. Kubernetes allows services, which can be arbitrary
> containers running any process, to declare exports ports. For service ports,
> the Kubernetes network data model just allows for declaring port number and
> [transport
> protocol|https://kubernetes.io/docs/reference/networking/service-protocols/]
> using well-defined enum values. Optionally, [application
> protocols|https://kubernetes.io/docs/concepts/services-networking/service/#application-protocol]
> can be provided as a hint, use freeform strings, and when available can be
> used by the provider for richer support of the app protocol. This data model
> serves as a good guide for NiFi to model a similar situation, just replacing
> services with NiFi extensions.
> A proposed, draft update to the NiFi API to introduce to concept of Listen
> Ports is available here:
> [https://github.com/kevdoran/nifi-api/tree/listen-ports]
> *Compatibility*
> Largely, this is a backwards compatible change.
> The proposed approach is entirely additive and optional: Once implemented and
> released, extension components can opt-in to declaring Listen Ports that they
> create. The burden to do so is minimal for component authors; in most cases,
> a few lines of code. Once done, flow registries and operational tools built
> atop them can dynamically discover flows that require data ingress rules, and
> the NiFi framework can dynamically discover components that provide data
> ingress ports and their port configuration to make it available via the REST
> API for external components. These offer incentives for component authors to
> opt-in to this feature, without any breaking changes that forces them to
> update their components to continue working on newer NiFi versions.
> There is one minor breaking change to the properties for the ListenSyslog
> processor. Syslog is an application protocol that can work over TCP or UDP.
> The current ListenSyslog processor allows specifying the port to listen on as
> a second property for specifying the transport protocol to accept (TCP or
> UDP). The proposed design for the NiFi API Property Descriptor would
> declaring a static transport protocol associated with a Listen Port. This
> allows knowing the transport protocol based on a flow definition without
> knowing what the runtime configuration will be, which greatly simplifies
> rules NiFi operators may want to codify such as if a flow definition is
> compatible with a target NiFi Runtime (eg, an operator may by policy block
> all inbound UDP traffic for security reasons.)
> The proposed solution to this is replacing the ListenSyslog processor Port
> and Protocol properties with TCP Port and UDP Port properties that are
> mutually exclusive (only one is allowed to be configured at a time). The
> migrate properties feature that was introduced for in-place flow version
> changes will allow us to migrate flows from the old configuration to the new
> configuration automatically for users, and migration guidance for the first
> NiFi release to include the modified processor can cover the remaining cases.
> *Verification*
> The Verification process for this feature would include:
> - Unit tests for components that define Listen Ports to make sure that they
> implement the new interfaces, and then when configured, the correct Listen
> Port definition is discoverable (ie, the return value for new interfaces
> matches the expected values based on the component configuration)
> - Integration tests that verify when multiple Listen components exist in a
> NiFi Cluster, the Listen Ports they create are correctly discoverable via the
> NiFi REST API.
> - Instructions for peer-reviewers to manually verify the feature
> implementation.
> - Documentation updates, primarily to the NiFi Developer Guide, to add
> instructions and guidance for implementing new listen components in a manner
> that is compatible with this new feature.
> *Alternatives*
> The following alternatives were considered:
> # {_}No changes to NiFi{_}; instead put the responsibility of discovering
> Listen Ports solely on external components, such as deployment scripts and
> infrastructure management logic. For example, external code could just "know"
> (ie, using hardcoded logic) that ListenHTTP defines a data ingress port via a
> property that accepts HTTP requests and look for instances of that known
> Processor type. Alternatively, we could go with a convention-based approach
> such as "processors that have a type name starting with Listen* and a Port
> property." Both of these are very brittle, lack discoverability by extension
> component authors, and do not account for the large community of NiFi
> developers that may use different conventions than those used for Apache NiFi
> components. For these reasons, this alternative was deemed insufficient.
> # {_}A larger change that also tries to unify the new feature port discovery
> to include the various framework-level ingress ports{_}, such as the remote
> input port used for site-to-site protocol or cluster communication ports. For
> example, maybe a new framework-level concept of a NiFi Gateway allows
> operators and flow authors to put all network ingress rules in one place.
> While this may have some advantages, and could be considered in the future,
> it was ultimately deemed too large a change at this time. It likely would
> include breaking changes to configuration files and and APIs, and therefore
> would be more reasonable to implement as a NiFi 3.0 / major version change,
> should the need ever arise. Additionally, framework-level ingress ports are
> defined via a different process (usually in nifi.properties) and once set do
> not change often; therefore, they are already much more management by
> something like IaC logic and less of a problem compared to ports that are
> part of flow definitions. This means that including them in the scope of this
> feature offers much less value despite greatly increasing the scope.
> The proposed scope and description offers the best benefit for minimal effort
> with practically no breaking changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
