From a security perspective, I’d like to see attention paid to the following topics:
* verifiable checksums over extensions/bundles to ensure the code being deployed from a remote system is trusted * fuzzy hashing (complementary) to provide “change factor” on new version of the same extension/bundle * cryptographic signatures from a trusted entity on extensions/bundles to allow for verification even over untrusted transmission * mechanisms to verify the available signatures automatically within NiFi rather than requiring external/manual verification * the same mutual authentication via TLS we provide for other services * the ability to configure multiple registries to allow for redundancy, load balancing, mitigation of denial of service, etc. Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Nov 14, 2018, at 06:55, Andrew Grande <[email protected]> wrote: > > I would like to see a clear separation of blob and metadata storage. Most > often you'd see some object storage being already distributed and > replicated, let's think about an easy way to backup or migrate the metadata > between registry instances. > > Andrew > >> On Tue, Nov 13, 2018, 11:32 AM Michael Moser <[email protected]> wrote: >> >> I have thought about this in the past, too. Here's a scenario where I >> could never really lay down an approach I was happy with. >> >> Consider that a NiFi user searches the NiFi registry and finds the PutMongo >> processor. Registry knows the PutMongo processor is in the >> nifi-mongodb-nar, and through its Nar-Dependency-Id that it has a >> dependency on a controller service interface in >> nifi-mongodb-client-service-api-nar. Great, the user can then download and >> install those two nars. How would we then suggest that the user also needs >> a MongoDBClientService controller service implementation, such as that in >> the nifi-mongodb-services-nar? >> >> I'm not looking for an answer now, of course, but I just wanted to feed the >> discussion. >> >> Thanks, >> -- Mike >> >> >>> On Tue, Nov 13, 2018 at 1:34 PM Bryan Bende <[email protected]> wrote: >>> >>> Mark, >>> >>> I think there are a couple of ways that could be solved, but >>> ultimately it would be up to how the users choose to setup/manage the >>> registry, or registries. >>> >>> The NiFi Registry security model is based around permissions to >>> buckets (read/write), and all versioned items belong to a bucket. So >>> you could think of each bucket as a mini extension repository for >>> which you can control access to specific users and groups, so there >>> could be a bucket of extensions for each of the NiFi instances in your >>> example. There can also be multiple registry instances registered with >>> a given NiFi so extensions can be pulled from multiple registries. >>> >>> The extensions an instance needs is based on the flows it is running, >>> the flows already have specific bundle coordinates for every component >>> in the flow. You can think of it similar to Maven where you declared a >>> dependency on library foo and the build goes out and gets it for you, >>> in this case it is a flow that declares a dependency on a bundle. >>> >>> Mike, >>> >>> Bundles would need to be uploaded to NiFi Registry (to a specific >>> bucket) as part of some TBD release process. At a minimum I was >>> envisioning NiFi CLI commands that can be pointed to a file or a >>> directory and upload the given bundles to registry. There could be >>> other options as well, possibly through a Maven plugin to release >>> directly into registry, or possibly to have a type of extension in >>> NiFi Registry that actually points to an external location, i.e. all >>> the NARs that end up in Maven central could somehow be imported into >>> NiFi Registry, but with pointers back to the content which actually >>> comes from Maven central. Lot of things to figure out here. >>> >>> -Bryan >>>> On Tue, Nov 13, 2018 at 1:16 PM Joe Witt <[email protected]> wrote: >>>> >>>> Group selection based on tag names for bundles could probably do that. >>>> Meaning it could be a sorting/filtering mechanism in the NiFi/Registry >>>> interface perhaps. Will be good to consider that UX as that >>>> progresses. >>>> >>>> As far as the different environments NiFi instances would certainly be >>>> able to load only referenced Nars for versioned flows so you'll get >>>> the optimal set (at runtime) automatically. Very powerful. >>>> On Tue, Nov 13, 2018 at 1:12 PM Mark Bean <[email protected]> >> wrote: >>>>> >>>>> Joe, >>>>> >>>>> I envision the Registry being able to provide a subset of NARs >>> required for >>>>> a specific NiFi instance. The user may have a relatively small set of >>> NARs >>>>> required for a NiFi used for basic routing/distribution, and a >>> different >>>>> more extensive set of NARs required for a more robust NiFi instance >>> which >>>>> performs various forms of processing/transformations. The grouping I >> am >>>>> describing would be a way to select multiple NARs required for a >>> specific >>>>> NiFi instance. >>>>> >>>>> Expanding the scenario a little farther, suppose an integration/test >>>>> environment defines the group. Then, the production environment can >>> use the >>>>> group definition to pull (or ensure it possesses) only the relevant >>> NARs >>>>> necessary. >>>>> >>>>> -Mark >>>>> >>>>>> On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <[email protected]> wrote: >>>>>> >>>>>> Mark >>>>>> >>>>>> Can you describe your use case from the user perspective both for >> the >>>>>> entity that would upload the items and demarcate them as a group as >>>>>> well as the user that would consume those bundles? >>>>>> >>>>>> I ask because the point here is that nars are themselves a 'group' >> in >>>>>> that they are a logical/contained grouping of extensions. These >> can >>>>>> have relationships to other nars as we know. And flows are >> designed >>>>>> against specific components that come from certain nars for which >> we >>>>>> know the precise coordinates. When importing flows that depend on >>>>>> these the system will be able to automatically acquire all that it >>>>>> needs. >>>>>> >>>>>> Thanks >>>>>> On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <[email protected]> >>> wrote: >>>>>>> >>>>>>> I would like to see a "group" capability in the Registry for NAR >>> bundles. >>>>>>> Then, users can create their own customized grouping of relevant >>> NARs. >>>>>>> Managing bundles one at a time will become tedious. >>>>>>> >>>>>>> Thanks, >>>>>>> Mark >>>>>>> >>>>>>> On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <[email protected]> >>> wrote: >>>>>>> >>>>>>>> Sivaprasanna - yes absolutely. That is the core point I think >> of >>>>>>>> Bryan's first bullet under the 'NiFi' section. >>>>>>>> On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna < >>>>>> [email protected]> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> One quick question. With the extension registry, my >>> understanding is >>>>>> that >>>>>>>>> we would try to reduce the overall NiFi size by offloading >>> certain >>>>>>>> existing >>>>>>>>> NAR bundles to the extension registry. So are we expecting >> the >>>>>> extension >>>>>>>>> registry to also come up with the ability/mechanism that lets >>> an >>>>>> user to >>>>>>>>> download these bundles ? >>>>>>>>> >>>>>>>>> - >>>>>>>>> Sivaprasanna >>>>>>>>> >>>>>>>>> On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt < >> [email protected]> >>>>>> wrote: >>>>>>>>> >>>>>>>>>> Bryan >>>>>>>>>> >>>>>>>>>> Very exciting to see this under way!!! We desperately have >>> to get >>>>>> our >>>>>>>>>> core nifi build size way down and make it far easier for >>> people to >>>>>>>>>> contribute new processors. We have a long line of >>> extensions that >>>>>>>>>> could be really useful/valuable and this will help unlock >>> that >>>>>> while >>>>>>>>>> improving the user experience tremendously. >>>>>>>>>> >>>>>>>>>> For the doc/concerns noted above we should also add/mention >>> that >>>>>> the >>>>>>>>>> relationship between nars (dependencies between nars for >>>>>>>>>> apis/controller services/parent nars/etc..) we want to have >>> a way >>>>>> to >>>>>>>>>> manage/show that or a user experience for it. Maybe not a >>> day-1 >>>>>> thing >>>>>>>>>> but important to call out. >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> Joe >>>>>>>>>> On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende < >>> [email protected]> >>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> All, >>>>>>>>>>> >>>>>>>>>>> We've needed the elusive extension registry for quite >> some >>> time >>>>>> now, >>>>>>>>>>> and with NiFi Registry I think we are in a good place to >>> make >>>>>> some >>>>>>>>>>> progress in this area. >>>>>>>>>>> >>>>>>>>>>> I've started looking into adding "extension bundles" to >>> NiFi >>>>>> Registry >>>>>>>>>>> as the next type of versioned item, along side the >> existing >>>>>> versioned >>>>>>>>>>> flows, and I wanted to take a minute to outline how that >>> approach >>>>>>>>>>> could work before getting too far into it. >>>>>>>>>>> >>>>>>>>>>> Also, I'd like to focus this discussion on the design and >>>>>>>>>>> functionality of the extension registry, and not on how >> the >>>>>> community >>>>>>>>>>> is going to use it. Topics like hosting a central >> registry, >>>>>> changing >>>>>>>>>>> the build process, restructuring the git repo, releasing >>> NARs, >>>>>> etc, >>>>>>>>>>> are all important topics, but first we need an extension >>> registry >>>>>>>>>>> before we can do any of that :) >>>>>>>>>>> >>>>>>>>>>> Here is a high-level description of what needs to be >>> done... >>>>>>>>>>> >>>>>>>>>>> NiFi Registry >>>>>>>>>>> >>>>>>>>>>> - Add a new type of item called an extension bundle, >> where >>> each >>>>>>>> bundle >>>>>>>>>>> can contain one ore extensions or APIs >>>>>>>>>>> >>>>>>>>>>> - Support bundles for traditional NiFi (aka NARs) and >> also >>>>>> bundles >>>>>>>> for >>>>>>>>>>> MiNiFi CPP >>>>>>>>>>> >>>>>>>>>>> - Ability to upload the binary artifact for a bundle and >>> extract >>>>>> the >>>>>>>>>>> metadata about the bundle, and metadata about the >>> extensions >>>>>>>> contained >>>>>>>>>>> in the bundle (more on this later) >>>>>>>>>>> >>>>>>>>>>> - Provide a pluggable storage provider for saving the >>> content of >>>>>> each >>>>>>>>>>> extension bundle so that we can have different >>> implementations >>>>>> like >>>>>>>>>>> local fileysystem, S3, and other object stores >>>>>>>>>>> >>>>>>>>>>> - Provide a REST API for listing and retrieving available >>>>>> bundles, >>>>>>>>>>> integrate this into the registry Java client and NiFi CLI >>>>>>>>>>> >>>>>>>>>>> NAR Maven Plugin >>>>>>>>>>> >>>>>>>>>>> - Generate a descriptor for each component in the NAR >>> which will >>>>>>>>>>> provide information like the description, tags, >> restricted >>> or >>>>>> not, >>>>>>>>>>> etc. >>>>>>>>>>> >>>>>>>>>>> - These descriptors will be parsed by NiFi Registry when >> a >>> NAR is >>>>>>>>>>> being uploaded so that NiFi Registry will know about the >>>>>> extensions >>>>>>>>>>> contained with in the NAR >>>>>>>>>>> >>>>>>>>>>> NiFi >>>>>>>>>>> >>>>>>>>>>> - Provide some type of extension manager experience where >>> users >>>>>> can >>>>>>>>>>> search, browse, & install bundles that are available in >>> any of >>>>>> the >>>>>>>>>>> registry clients that have been defined >>>>>>>>>>> >>>>>>>>>>> - Introduce a new security policy to control which users >>> are >>>>>> allowed >>>>>>>>>>> to access the extension manager >>>>>>>>>>> >>>>>>>>>>> - Installing a bundle should load the NAR and make the >>> extensions >>>>>>>>>>> available leveraging the recent work done to auto-load >> new >>> NARs >>>>>>>>>>> >>>>>>>>>>> - Importing versioned flows from registry should provide >>> an easy >>>>>> way >>>>>>>>>>> to install bundles that are required by the flow, but >>> missing >>>>>> from >>>>>>>> the >>>>>>>>>>> local NiFi instance >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> If anyone has any thoughts or concerns about this >> approach, >>>>>> please >>>>>>>> let >>>>>>>>>> me know. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Bryan >>>>>>>>>> >>>>>>>> >>>>>> >>> >>
