Hey folks, With the release of Registry, I’ve been contemplating leveraging it and the current codebase of NiFi to facilitate ease of use for MiNiFi flow design. One of the areas I would like to form a concerted effort around is that of the Command and Control (C2) functionality originally presented when the MiNiFi effort was started and further expounded upon with a feature proposal [1]. In that proposal, while the names are dated, we have components that fulfill some of the core building blocks toward the overall vision of C2.
For those not intimately familiar, MiNiFi is primarily configured via a YAML configuration file. [2] The initial idea was that flows would be simple a source, maybe a transform or two, and then transmission. The notion was that hand creation of one of these files would not be overly onerous. It became apparent from questions, however, that users were doing more complex flows and this YAML ranged from tedious to difficult for those situations. This precipitated the best approach currently available, utilizing a NiFi instance as a way to generate a template. This template was then exported, and using our provided toolkit to get YAML out that is deployable with your instance. Better, but far from ideal. A couple approaches came to mind to further narrow this gap of user expectation and realities of our configuration. (1) In our current state, we have heavily relied on convenience of component overlap. Generally speaking, we relied on the notion that NiFi’s bundled components are a superset of those found in MiNiFi implementations. We could have continued that, but this would not necessarily free the user of the file handling and transformation processes. Additionally, this comes with caveats, among them that given the ways in which C++ allows us different accesses, there are some interesting processors that work better for that more native approach and might not ever justify having an equivalent in Java. (2) Providing a first class experience in NiFi for MiNiFi flow designing making use of the existing UI components as a base and a new design mode specifically for providing a clean user experience, letting users operate in a different context but a familiar environment. Certainly preferred, but how this fit was not clear. As a crash course in MiNiFi efforts, there have been some key contributions that I believe when more strongly united can generate a very complete authoring and flow update process. Some of these items are: - C2 API Protocol specification and implementation in C++ [3] - C2 Server with the notion of serving different flows [4] backed by varying implementations, one of which was via REST to access NiFi Templates directly [5] - Warm Redeploy in MiNiFi Java [6] One of the first questions to come to mind was, “Is within NiFi the right place?” I think there are several reasons that this is reasonable. Seeing some of the discussion around the consolidation of common UI components presented by Scott Aslan, I think sharing the core of the UI elements between the traditional NiFi and MiNiFi flow designs is a logical extension. While those flows in MiNiFi are not real-time changes we are dealing with the same design elements in creating directed graphs. Additionally, while MiNiFi instances will likely not ever have the same breadth of extensions supported, there is nothing inherently prohibitive of a more complex graph comprised of larger numbers of edges and/or vertices. In my view, the mechanics are the same but just being performed in a different contexts. In the case of MiNiFi, this is effectively an “offline” context. By "offline," I mean we are not strictly bound to loaded components such as that in the current NiFi context consisting of items from unpacked NARs. How can we minimize the impact to NiFi? Given that this is a new capability, I don’t know that we necessarily need to interfere with the existing APIs and could possibly also make this an optional feature. From this standpoint, we could develop new endpoints for this specific context. Some of this work would rely on external services and/or definitions to drive the different context. For example, without NARs acting as the source of available components, something like the C2 server could provide extensions for MiNiFi instances it is aware of to help drive the experience. This data would be a skeletal view of a component definition/interface without any of the inherent code backing it needed. How do we maximize the utility of these efforts? With the idea about powering MiNiFi flow design with specific sets of components, I could see this extending to even NiFi itself. Consider the use case where something like I wish to make use of additional artifacts along the lines of [7] and an extension of the current Registry implementation. As a simple example, with an understanding of what the universe of components is available to my system and not just those currently loaded, it would be possible for users to design for other instances. With these hooks in place, there are a few permutations of this relative concept that enable us to carry out some of the “nice to haves” we haven’t quite been able to address. How does this all come together for MiNiFi? I have created a work of sheer artistic mastery on the Command and Control proposal to illustrate a rough sequence and architecture. [8] This diagram shows from entering MiNiFi flow design through to an instance updating to the new flow. Overall, Registry was a tremendous step forward for the NiFi ecosystem. I think in our current state, we have reached a convergence of capabilities that allow us to build higher level abstractions for dataflow management. I think the approach of (2) is a great first foray into that realm that provides an immediate solution for a gap in our MiNiFi efforts. Additionally, this approach lays the foundation for additional applications possibly inclusive of core NiFi. I look forward to your thoughts and commentary. Certainly many details to think and reason about, but I see some incredible potential with the work we've done thus far both near and long term. Thanks for your time! --aldrin
