Some thoughts - so on the roadmap Joe posted a couple days ago, HA was a big focus (in 1.X). NiFi becomes a bigger better "hub", but likely those capabilities don't scale down quite as well. Thanks for kicking off this discussion, Joe, I think now is the time think about what we need to do about "spokes", especially when devices are getting smaller and the software is getting bigger.
In terms of how this fits into the project - this feels like a similar evolution that hadoop had with YARN - basically rethinking parts of the project to address shortcomings in the architecture/ecosystem. With different implementations (and presumably an spec for others to implement their own MiNiFi (would that be called a YourNiFi?)) that can be plugged in, it seems especially YARN-like. Also, on a lighter, and half-joking note, I prefer TiNiFi - portmanteau of Tiny and NiFi! "Tie-knee-fye". On Sat, Jan 9, 2016 at 7:29 PM, Joe Witt <[email protected]> wrote: > NiFi Community, > > I'd like to initiate discussion around a proposal to create our first > sub-project of NiFi. A possible name for it is "MiNiFi" a sort of > play on Mini-NiFi. > > The idea is to provide a complementary data collection agent to NiFi's > current approach of dataflow management. As noted in our ASF TLP > resolution NiFi is to provide "an automated and durable data broker > between systems providing interactive command and control and detailed > chain of custody for data." MiNiFi would be consistent with that > scope with a specific focus on the first-mile challenge so common in > dataflow. > > Specific goals of MiNiFi would be to provide a small, lightweight, > centrally managed agent that natively generates data provenance and > seamlessly integrates with NiFi for follow-on dataflow management and > maintenance of the chain of custody provided by the powerful data > provenance features of NiFi. > > MiNiFi should be designed to operate directly on or adjacent to the > source sensor, system, server generating the events as a resource > sensitive tenant. There are numerous agent models in existence today > but they do not offer the command and control or provenance that is so > important to the philosophy and scope of NiFi. > > These agents would necessarily have a different interactive command > and control model than NiFi as you'd not expect consistent behavior, > capability, or accessibility of all instances of the agents at any > given time. > > Multiple implementations of MiNiFi are envisioned including those that > operate on the JVM and those that do not. > > As the discussion advances we can put together wiki pages, concept > diagrams, and requirements to help better articulate how this might > evolve. We should also discuss the mechanics of how this might work > in terms of infrastructure, code repository, and more. > > Thanks > Joe >
