[
https://issues.apache.org/jira/browse/NIP-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17910725#comment-17910725
]
Pierre Villard commented on NIP-4:
----------------------------------
{quote}We should consider this not being a public extension point. We have
various framework extensions which are not truly public extension points. I
envision a handful of these implementations but not dozens.
{quote}
Given the potential dependencies that would come with this (AWS, Azure, Google
Cloud, etc), I thought we would want to have the NAR approach with a new public
extension point. Besides, I believe that depending on user's environment,
specific needs might be required and it would make it easier for users to build
their own. I don't have a strong opinion about this though, it just feels
easier from a UX PoV (and aligned with parameter providers for example).
{quote}We should roll this out in a removable way as well meaning if we find
the implementation/api isn't quite right we can undo/remove rather than getting
locked-in for an entire major line.
{quote}
Can't we mark API changes as experimental to give us the option to remove those
at any point? Similar to what we did with some features of NiFi 1.
{quote}We will need to consider how to ensure/resolve that any/all nodes can
see/download a given nar from an endpoint.
{quote}
All of this complexity is managed by the NAR manager that has already been
introduced in NiFi 2. Assuming any node can access the endpoint where the
NAR(s) to download is/are located, then it would work just fine. I do
anticipate that some proxy configuration would likely be required in the
provided implementations (especially for accessing nexus like repos).
> Extension Registry Client
> -------------------------
>
> Key: NIP-4
> URL: https://issues.apache.org/jira/browse/NIP-4
> Project: NiFi Improvement Proposal
> Issue Type: New Feature
> Reporter: Pierre Villard
> Priority: Major
>
> *Motivation and description*
> The goal of this NIP is to describe the addition of a new extension point:
> extension registry client. The goal of an extension registry client is to
> have a "pull" mechanism for NiFi to pull extensions (NAR files) into NiFi so
> that the components can be used in NiFi for designing and running flows.
> Main features :
> * the extension registry clients would check, at a specified frequency, for
> available NARs in external locations. The list of available components would
> be retrieved by looking at the manifest files exposed by the external
> locations.
> * when listing components in NiFi (processors, controller services, etc),
> the list would be a merge of what is presently available on the NiFi nodes as
> well as what is theoretically available through the extension registry
> clients. If a user starts using a component that is not locally available,
> NiFi would download the required NAR(s) to have the component in NiFi. This
> would work similarly to how it works with Python components where things are
> loaded in the background. If things cannot be loaded, the component would be
> ghosted.
> * when importing a flow definition (flow definition upload, flow registry
> clients, etc), the same behaviour would apply for components that are defined
> in the flow: if the instantiated flow references a bundle that is not locally
> available but available remotely, we would download the required NARs
> Extension registry clients will be defined at controller level similarly to
> what is available today with registry clients, reporting tasks, parameter
> providers, etc. The same level of permissions will be required and it is
> expected that only NiFi admin would configure such a client against trusted
> external locations.
> *Advantages*
> * by doing this, it would be possible to ship a NiFi runtime with a minimal
> size and only download the NARs that are required for running the flow. That
> would make auto scaling more efficient and considerably reduce the container
> image size.
> * this would provide a great option to raise awareness about components that
> are maintained by the community but not made available in the convenience
> binary.
> * this would provide a nice opportunity for third parties to share specific
> components with the community while not pushing the responsibility of
> maintaining those on the community.
> *Scope*
> A large number of changes are expected but nothing that would not be backward
> compatible. This NIP will likely need to be updated for exhaustivity while
> implementing the overall capability. It is hard to anticipate the full extent
> of the changes before actual POC implementation.
> In nifi-api, the expected changes are:
> * new classes around ExtensionRegistryClient
> * additional ComponentType
> * the Bundle object will have a boolean isRemote to determine if a bundle is
> locally available or remotely available for NiFi
> The new APIs for Extension Registry Clients would be:
> * InputStream getNARFile(ExtensionRegistryClientConfigurationContext
> context, Bundle bundle) throws ExtensionRegistryException, IOException;
> * InputStream getManifestFile(ExtensionRegistryClientConfigurationContext
> context, Bundle bundle) throws ExtensionRegistryException, IOException;
> * Set<Bundle> listBundles(ExtensionRegistryClientConfigurationContext
> context) throws ExtensionRegistryException, IOException;
> As part of this change, we would mark as deprecated the existing
> ExternalResourceProvider concept and all of its implementations. All of this
> would be removed in NiFi 3.
> *Not in Scope*
> * Support for NAR including Python components is not in scope at this stage.
> This will require some discussions regarding the extensions manifest for such
> components.
> * For the initial version, we do not support the case where NiFi would be
> initially provisioned with a flow.json.gz containing extension registry
> clients as well as components in the flow definition that should be fetched
> from remote locations. Support can later be added by this case by initially
> ghosting the components and after the initial startup of NiFi, check the
> ghost components and trigger the download of remote extensions where
> applicable.
> *Verification*
> Implementing a couple of extension providers will help verifying the proper
> implementation of the feature. In particular some interesting options that
> are considered:
> * Local file system implementation
> * Nexus-like implementation
> * AWS S3 implementation
> Anything that is not Nexus-like with well defined APIs will need to adhere to
> a specific convention allowing for discoverability of the components as well
> as retrieval of the manifest files.
> In order to provide proper behaviour, it is assumed and required for the
> external stores to have the manifest files published along the NAR files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
