Joe, I envision the Registry being able to provide a subset of NARs required for a specific NiFi instance. The user may have a relatively small set of NARs required for a NiFi used for basic routing/distribution, and a different more extensive set of NARs required for a more robust NiFi instance which performs various forms of processing/transformations. The grouping I am describing would be a way to select multiple NARs required for a specific NiFi instance.
Expanding the scenario a little farther, suppose an integration/test environment defines the group. Then, the production environment can use the group definition to pull (or ensure it possesses) only the relevant NARs necessary. -Mark On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <[email protected]> wrote: > Mark > > Can you describe your use case from the user perspective both for the > entity that would upload the items and demarcate them as a group as > well as the user that would consume those bundles? > > I ask because the point here is that nars are themselves a 'group' in > that they are a logical/contained grouping of extensions. These can > have relationships to other nars as we know. And flows are designed > against specific components that come from certain nars for which we > know the precise coordinates. When importing flows that depend on > these the system will be able to automatically acquire all that it > needs. > > Thanks > On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <[email protected]> wrote: > > > > I would like to see a "group" capability in the Registry for NAR bundles. > > Then, users can create their own customized grouping of relevant NARs. > > Managing bundles one at a time will become tedious. > > > > Thanks, > > Mark > > > > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <[email protected]> wrote: > > > > > Sivaprasanna - yes absolutely. That is the core point I think of > > > Bryan's first bullet under the 'NiFi' section. > > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna < > [email protected]> > > > wrote: > > > > > > > > One quick question. With the extension registry, my understanding is > that > > > > we would try to reduce the overall NiFi size by offloading certain > > > existing > > > > NAR bundles to the extension registry. So are we expecting the > extension > > > > registry to also come up with the ability/mechanism that lets an > user to > > > > download these bundles ? > > > > > > > > - > > > > Sivaprasanna > > > > > > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <[email protected]> > wrote: > > > > > > > > > Bryan > > > > > > > > > > Very exciting to see this under way!!! We desperately have to get > our > > > > > core nifi build size way down and make it far easier for people to > > > > > contribute new processors. We have a long line of extensions that > > > > > could be really useful/valuable and this will help unlock that > while > > > > > improving the user experience tremendously. > > > > > > > > > > For the doc/concerns noted above we should also add/mention that > the > > > > > relationship between nars (dependencies between nars for > > > > > apis/controller services/parent nars/etc..) we want to have a way > to > > > > > manage/show that or a user experience for it. Maybe not a day-1 > thing > > > > > but important to call out. > > > > > > > > > > Thanks! > > > > > Joe > > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <[email protected]> > wrote: > > > > > > > > > > > > All, > > > > > > > > > > > > We've needed the elusive extension registry for quite some time > now, > > > > > > and with NiFi Registry I think we are in a good place to make > some > > > > > > progress in this area. > > > > > > > > > > > > I've started looking into adding "extension bundles" to NiFi > Registry > > > > > > as the next type of versioned item, along side the existing > versioned > > > > > > flows, and I wanted to take a minute to outline how that approach > > > > > > could work before getting too far into it. > > > > > > > > > > > > Also, I'd like to focus this discussion on the design and > > > > > > functionality of the extension registry, and not on how the > community > > > > > > is going to use it. Topics like hosting a central registry, > changing > > > > > > the build process, restructuring the git repo, releasing NARs, > etc, > > > > > > are all important topics, but first we need an extension registry > > > > > > before we can do any of that :) > > > > > > > > > > > > Here is a high-level description of what needs to be done... > > > > > > > > > > > > NiFi Registry > > > > > > > > > > > > - Add a new type of item called an extension bundle, where each > > > bundle > > > > > > can contain one ore extensions or APIs > > > > > > > > > > > > - Support bundles for traditional NiFi (aka NARs) and also > bundles > > > for > > > > > > MiNiFi CPP > > > > > > > > > > > > - Ability to upload the binary artifact for a bundle and extract > the > > > > > > metadata about the bundle, and metadata about the extensions > > > contained > > > > > > in the bundle (more on this later) > > > > > > > > > > > > - Provide a pluggable storage provider for saving the content of > each > > > > > > extension bundle so that we can have different implementations > like > > > > > > local fileysystem, S3, and other object stores > > > > > > > > > > > > - Provide a REST API for listing and retrieving available > bundles, > > > > > > integrate this into the registry Java client and NiFi CLI > > > > > > > > > > > > NAR Maven Plugin > > > > > > > > > > > > - Generate a descriptor for each component in the NAR which will > > > > > > provide information like the description, tags, restricted or > not, > > > > > > etc. > > > > > > > > > > > > - These descriptors will be parsed by NiFi Registry when a NAR is > > > > > > being uploaded so that NiFi Registry will know about the > extensions > > > > > > contained with in the NAR > > > > > > > > > > > > NiFi > > > > > > > > > > > > - Provide some type of extension manager experience where users > can > > > > > > search, browse, & install bundles that are available in any of > the > > > > > > registry clients that have been defined > > > > > > > > > > > > - Introduce a new security policy to control which users are > allowed > > > > > > to access the extension manager > > > > > > > > > > > > - Installing a bundle should load the NAR and make the extensions > > > > > > available leveraging the recent work done to auto-load new NARs > > > > > > > > > > > > - Importing versioned flows from registry should provide an easy > way > > > > > > to install bundles that are required by the flow, but missing > from > > > the > > > > > > local NiFi instance > > > > > > > > > > > > > > > > > > If anyone has any thoughts or concerns about this approach, > please > > > let > > > > > me know. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Bryan > > > > > > > > >
