Thanks for your attention and effort on this, Matt. I look forward to the discuss threads, but at a high level, this sounds like a good and important step for MiNiFi Java and C2.
Cheers, Kevin > On Nov 16, 2020, at 12:28, Matt Burgess <[email protected]> wrote: > > All, > > As you know, MiNiFi Java is a separate project and codebase than NiFi, > it's my understanding that this was done because MiNiFi Java was a > fledgling effort while NiFi was an established project, and trying to > do rapid development and releases on MiNiFi would cause some > interference with NiFi features, releases, etc. > > Since then, MiNiFi Java has advanced in its own right, but lately > development and release cadence has slowed, and AFAICT much of the > effort is to bring NiFi capabilities into MiNiFi, exposing previously > ignored NiFi properties in MiNiFi's config.yml, etc. > > I've begun work on getting MiNiFi Java back into the NiFi codebase, > under its own area with its own assembly, bootstrap, etc. This is > being tracked in Jira under MINIFI-422 [1]. > > The idea is to be able to build a MiNiFi assembly from NiFi and MiNiFi > components, thereby giving MiNiFi capabilities from NiFi without > having to port them to a separate project. Kerberos support is a > shining example of this, as are other features available via NiFi's > various Context objects (where MiNiFi currently has them stubbed out). > > The first step was refactoring some NiFi NAR structure in order to > allow for a "headless" NiFi [2][3], the idea being that we could reuse > some existing framework components/NARs without being tied to the UI, > associated webapps, and Jetty itself. > > I will create separate [DISCUSS] threads for each of the following, > but here are some things I'm looking at going forward: > > 1) Bring MiNiFi bootstrap and external command-and-control (C2) code > over to a MiNiFi area in the NiFi project structure. Also create a > MiNiFi assembly in that area for the purposes of generating release > artifacts when ready. > > 2) Bring over and update the C2 Server reference implementation, for > testing and to give some way to externally update config/flows in > MiNiFi. This may mean we don't need the change ingestors anymore, but > I will send a separate DISCUSS thread to see who's using them, if > they're important to keep, etc. > > 3) Discuss whether to bring over the config.yml mechanism for > defining/updating configuration and flow. If we eventually want to > share more between MiNiFi and NiFi, should we return to using > nifi.properties and flow.xml.gz rather than maintaining (and updating) > the code to expose nifi properties as human-readable YAML strings, to > include all the schemas and converters? This too will be a separate > [DISCUSS] thread. > > 4) Once all is determined and implemented, look at ways to share any > components between NiFi and MiNiFi. This might include refactoring the > C2 stuff into module(s) for use by NiFi [4], adding/enabling profiles > and/or assemblies to build various products (headless NiFi with C2, > MiNiFi with Elasticsearch NARs, e.g.) > > I plan on starting on 1 soon, but am glad to discuss any of these > things or any other topics you'd like to bring up (for 2 and 3 let's > have the discussion in the forthcoming DISCUSS threads) > > Regards, > Matt > > [1] https://issues.apache.org/jira/browse/MINIFI-422 > [2] https://issues.apache.org/jira/browse/NIFI-7592 > [3] https://issues.apache.org/jira/browse/MINIFI-485 > [4] https://issues.apache.org/jira/browse/NIFI-6813
