[
https://issues.apache.org/jira/browse/NIP-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17910385#comment-17910385
]
Joe Witt commented on NIP-4:
----------------------------
[~pvillard] Certainly an important topic so thanks for kicking this off.
We should consider this not being a public extension point. We have various
framework extensions which are not truly public extension points. I envision a
handful of these implementations but not dozens.
We should roll this out in a removable way as well meaning if we find the
implementation/api isn't quite right we can undo/remove rather than getting
locked-in for an entire major line.
We will need to consider how to ensure/resolve that any/all nodes can
see/download a given nar from an endpoint.
> Extension Registry Client
> -------------------------
>
> Key: NIP-4
> URL: https://issues.apache.org/jira/browse/NIP-4
> Project: NiFi Improvement Proposal
> Issue Type: New Feature
> Reporter: Pierre Villard
> Priority: Major
>
> *Motivation and description*
> The goal of this NIP is to describe the addition of a new extension point:
> extension registry client. The goal of an extension registry client is to
> have a "pull" mechanism for NiFi to pull extensions (NAR files) into NiFi so
> that the components can be used in NiFi for designing and running flows.
> Main features :
> * the extension registry clients would check, at a specified frequency, for
> available NARs in external locations. The list of available components would
> be retrieved by looking at the manifest files exposed by the external
> locations.
> * when listing components in NiFi (processors, controller services, etc),
> the list would be a merge of what is presently available on the NiFi nodes as
> well as what is theoretically available through the extension registry
> clients. If a user starts using a component that is not locally available,
> NiFi would download the required NAR(s) to have the component in NiFi. This
> would work similarly to how it works with Python components where things are
> loaded in the background. If things cannot be loaded, the component would be
> ghosted.
> * when importing a flow definition (flow definition upload, flow registry
> clients, etc), the same behaviour would apply for components that are defined
> in the flow: if the instantiated flow references a bundle that is not locally
> available but available remotely, we would download the required NARs
> Extension registry clients will be defined at controller level similarly to
> what is available today with registry clients, reporting tasks, parameter
> providers, etc. The same level of permissions will be required and it is
> expected that only NiFi admin would configure such a client against trusted
> external locations.
> *Advantages*
> * by doing this, it would be possible to ship a NiFi runtime with a minimal
> size and only download the NARs that are required for running the flow. That
> would make auto scaling more efficient and considerably reduce the container
> image size.
> * this would provide a great option to raise awareness about components that
> are maintained by the community but not made available in the convenience
> binary.
> * this would provide a nice opportunity for third parties to share specific
> components with the community while not pushing the responsibility of
> maintaining those on the community.
> *Scope*
> A large number of changes are expected but nothing that would not be backward
> compatible. This NIP will likely need to be updated for exhaustivity while
> implementing the overall capability. It is hard to anticipate the full extent
> of the changes before actual POC implementation.
> In nifi-api, the expected changes are:
> * new classes around ExtensionRegistryClient
> * additional ComponentType
> * the Bundle object will have a boolean isRemote to determine if a bundle is
> locally available or remotely available for NiFi
> The new APIs for Extension Registry Clients would be:
> * InputStream getNARFile(ExtensionRegistryClientConfigurationContext
> context, Bundle bundle) throws ExtensionRegistryException, IOException;
> * InputStream getManifestFile(ExtensionRegistryClientConfigurationContext
> context, Bundle bundle) throws ExtensionRegistryException, IOException;
> * Set<Bundle> listBundles(ExtensionRegistryClientConfigurationContext
> context) throws ExtensionRegistryException, IOException;
> As part of this change, we would mark as deprecated the existing
> ExternalResourceProvider concept and all of its implementations. All of this
> would be removed in NiFi 3.
> *Not in Scope*
> * Support for NAR including Python components is not in scope at this stage.
> This will require some discussions regarding the extensions manifest for such
> components.
> * For the initial version, we do not support the case where NiFi would be
> initially provisioned with a flow.json.gz containing extension registry
> clients as well as components in the flow definition that should be fetched
> from remote locations. Support can later be added by this case by initially
> ghosting the components and after the initial startup of NiFi, check the
> ghost components and trigger the download of remote extensions where
> applicable.
> *Verification*
> Implementing a couple of extension providers will help verifying the proper
> implementation of the feature. In particular some interesting options that
> are considered:
> * Local file system implementation
> * Nexus-like implementation
> * AWS S3 implementation
> Anything that is not Nexus-like with well defined APIs will need to adhere to
> a specific convention allowing for discoverability of the components as well
> as retrieval of the manifest files.
> In order to provide proper behaviour, it is assumed and required for the
> external stores to have the manifest files published along the NAR files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
