Pierre Villard created NIP-4:
--------------------------------
Summary: Extension Registry Client
Key: NIP-4
URL: https://issues.apache.org/jira/browse/NIP-4
Project: NiFi Improvement Proposal
Issue Type: New Feature
Reporter: Pierre Villard
*Motivation and description*
The goal of this NIP is to describe the addition of a new extension point:
extension registry client. The goal of an extension registry client is to have
a "pull" mechanism for NiFi to pull extensions (NAR files) into NiFi so that
the components can be used in NiFi for designing and running flows.
Main features :
* the extension registry clients would check, at a specified frequency, for
available NARs in external locations. The list of available components would be
retrieved by looking at the manifest files exposed by the external locations.
* when listing components in NiFi (processors, controller services, etc), the
list would be a merge of what is presently available on the NiFi nodes as well
as what is theoretically available through the extension registry clients. If a
user starts using a component that is not locally available, NiFi would
download the required NAR(s) to have the component in NiFi. This would work
similarly to how it works with Python components where things are loaded in the
background. If things cannot be loaded, the component would be ghosted.
* when importing a flow definition (flow definition upload, flow registry
clients, etc), the same behaviour would apply for components that are defined
in the flow: if the instantiated flow references a bundle that is not locally
available but available remotely, we would download the required NARs
Extension registry clients will be defined at controller level similarly to
what is available today with registry clients, reporting tasks, parameter
providers, etc. The same level of permissions will be required and it is
expected that only NiFi admin would configure such a client against trusted
external locations.
*Advantages*
* by doing this, it would be possible to ship a NiFi runtime with a minimal
size and only download the NARs that are required for running the flow. That
would make auto scaling more efficient and considerably reduce the container
image size.
* this would provide a great option to raise awareness about components that
are maintained by the community but not made available in the convenience
binary.
* this would provide a nice opportunity for third parties to share specific
components with the community while not pushing the responsibility of
maintaining those on the community.
*Scope*
A large number of changes are expected but nothing that would not be backward
compatible. This NIP will likely need to be updated for exhaustivity while
implementing the overall capability. It is hard to anticipate the full extent
of the changes before actual POC implementation.
In nifi-api, the expected changes are:
* new classes around ExtensionRegistryClient
* additional ComponentType
* the Bundle object will have a boolean isRemote to determine if a bundle is
locally available or remotely available for NiFi
The new APIs for Extension Registry Clients would be:
* InputStream getNARFile(ExtensionRegistryClientConfigurationContext context,
Bundle bundle) throws ExtensionRegistryException, IOException;
* InputStream getManifestFile(ExtensionRegistryClientConfigurationContext
context, Bundle bundle) throws ExtensionRegistryException, IOException;
* Set<Bundle> listBundles(ExtensionRegistryClientConfigurationContext context)
throws ExtensionRegistryException, IOException;
As part of this change, we would mark as deprecated the existing
ExternalResourceProvider concept and all of its implementations. All of this
would be removed in NiFi 3.
*Verification*
Implementing a couple of extension providers will help verifying the proper
implementation of the feature. In particular some interesting options that are
considered:
* Local file system implementation
* Nexus-like implementation
* AWS S3 implementation
Anything that is not Nexus-like with well defined APIs will need to adhere to a
specific convention allowing for discoverability of the components as well as
retrieval of the manifest files.
In order to provide proper behaviour, it is assumed and required for the
external stores to have the manifest files published along the NAR files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
