Likewise - thanks! Will be good to see this come back into core so they can grow together. This along with the work happening for stateless nifi will offer a lot of choices for users to use the most appropriate engine for their use case while building flows against consistent flow design/language and massive set of extensions!
On Mon, Nov 16, 2020 at 10:41 AM Pierre Villard <[email protected]> wrote: > +1 to all of this - I'm really looking forward to seeing more of this! > > Le lun. 16 nov. 2020 à 18:39, Kevin Doran <[email protected]> a écrit : > > > Thanks for your attention and effort on this, Matt. I look forward to the > > discuss threads, but at a high level, this sounds like a good and > important > > step for MiNiFi Java and C2. > > > > Cheers, > > Kevin > > > > > On Nov 16, 2020, at 12:28, Matt Burgess <[email protected]> wrote: > > > > > > All, > > > > > > As you know, MiNiFi Java is a separate project and codebase than NiFi, > > > it's my understanding that this was done because MiNiFi Java was a > > > fledgling effort while NiFi was an established project, and trying to > > > do rapid development and releases on MiNiFi would cause some > > > interference with NiFi features, releases, etc. > > > > > > Since then, MiNiFi Java has advanced in its own right, but lately > > > development and release cadence has slowed, and AFAICT much of the > > > effort is to bring NiFi capabilities into MiNiFi, exposing previously > > > ignored NiFi properties in MiNiFi's config.yml, etc. > > > > > > I've begun work on getting MiNiFi Java back into the NiFi codebase, > > > under its own area with its own assembly, bootstrap, etc. This is > > > being tracked in Jira under MINIFI-422 [1]. > > > > > > The idea is to be able to build a MiNiFi assembly from NiFi and MiNiFi > > > components, thereby giving MiNiFi capabilities from NiFi without > > > having to port them to a separate project. Kerberos support is a > > > shining example of this, as are other features available via NiFi's > > > various Context objects (where MiNiFi currently has them stubbed out). > > > > > > The first step was refactoring some NiFi NAR structure in order to > > > allow for a "headless" NiFi [2][3], the idea being that we could reuse > > > some existing framework components/NARs without being tied to the UI, > > > associated webapps, and Jetty itself. > > > > > > I will create separate [DISCUSS] threads for each of the following, > > > but here are some things I'm looking at going forward: > > > > > > 1) Bring MiNiFi bootstrap and external command-and-control (C2) code > > > over to a MiNiFi area in the NiFi project structure. Also create a > > > MiNiFi assembly in that area for the purposes of generating release > > > artifacts when ready. > > > > > > 2) Bring over and update the C2 Server reference implementation, for > > > testing and to give some way to externally update config/flows in > > > MiNiFi. This may mean we don't need the change ingestors anymore, but > > > I will send a separate DISCUSS thread to see who's using them, if > > > they're important to keep, etc. > > > > > > 3) Discuss whether to bring over the config.yml mechanism for > > > defining/updating configuration and flow. If we eventually want to > > > share more between MiNiFi and NiFi, should we return to using > > > nifi.properties and flow.xml.gz rather than maintaining (and updating) > > > the code to expose nifi properties as human-readable YAML strings, to > > > include all the schemas and converters? This too will be a separate > > > [DISCUSS] thread. > > > > > > 4) Once all is determined and implemented, look at ways to share any > > > components between NiFi and MiNiFi. This might include refactoring the > > > C2 stuff into module(s) for use by NiFi [4], adding/enabling profiles > > > and/or assemblies to build various products (headless NiFi with C2, > > > MiNiFi with Elasticsearch NARs, e.g.) > > > > > > I plan on starting on 1 soon, but am glad to discuss any of these > > > things or any other topics you'd like to bring up (for 2 and 3 let's > > > have the discussion in the forthcoming DISCUSS threads) > > > > > > Regards, > > > Matt > > > > > > [1] https://issues.apache.org/jira/browse/MINIFI-422 > > > [2] https://issues.apache.org/jira/browse/NIFI-7592 > > > [3] https://issues.apache.org/jira/browse/MINIFI-485 > > > [4] https://issues.apache.org/jira/browse/NIFI-6813 > > > > >
