Aldrin,
Thanks a lot for taking the time to write up such a detailed and well thought
out proposal.
I really like the idea of publishing flows to NiFi Registry that then get
deployed to MiNiFi agents via the C2 Server. I don't have much to add there,
and I think that you've identified one of the big missing parts to make that
experience user-friendly -- how to design and author the flows in the first
place.
The MiNiFi Flow Designer would fill that need in a really elegant way with a
low learning curve for NiFi users. Having the ability to design and author
flows in an environment other than the "runtime" environment one is targeting
would be a huge step forward. Not just for MiNiFi, but multi-environment
deployments of NiFi, where one might be developing a flow in an environment
other than where they intend to run it. This would make the SDLC options for
NiFi flows even more flexible, which is a big part of the goal of the new NiFi
Registry.
It certainly presents some challenges, but I definitely think the upside for
MiNiFi flow authoring would be huge.
Thanks,
Kevin
On 2/28/18, 07:53, "Aldrin Piri" <aldrinp...@gmail.com> wrote:
It appears I accidentally left off the actual links. Here they are in all
their glory:
[1]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
[2]
https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
[3] https://issues.apache.org/jira/browse/MINIFICPP-36
[4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
[5]
https://github.com/apache/nifi-minifi/tree/master/minifi-c2#configuration-providers
[6]
https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#automatic-warm-redeploy
[7]
https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
[8]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails
On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <aldrinp...@gmail.com> wrote:
> Hey folks,
>
> With the release of Registry, I’ve been contemplating leveraging it and
> the current codebase of NiFi to facilitate ease of use for MiNiFi flow
> design. One of the areas I would like to form a concerted effort around
is
> that of the Command and Control (C2) functionality originally presented
> when the MiNiFi effort was started and further expounded upon with a
> feature proposal [1]. In that proposal, while the names are dated, we
have
> components that fulfill some of the core building blocks toward the
overall
> vision of C2.
>
> For those not intimately familiar, MiNiFi is primarily configured via a
> YAML configuration file. [2] The initial idea was that flows would be
> simple a source, maybe a transform or two, and then transmission. The
> notion was that hand creation of one of these files would not be overly
> onerous. It became apparent from questions, however, that users were
doing
> more complex flows and this YAML ranged from tedious to difficult for
those
> situations. This precipitated the best approach currently available,
> utilizing a NiFi instance as a way to generate a template. This template
> was then exported, and using our provided toolkit to get YAML out that is
> deployable with your instance. Better, but far from ideal.
>
> A couple approaches came to mind to further narrow this gap of user
> expectation and realities of our configuration.
>
> (1) In our current state, we have heavily relied on convenience of
> component overlap. Generally speaking, we relied on the notion that
NiFi’s
> bundled components are a superset of those found in MiNiFi
> implementations. We could have continued that, but this would not
> necessarily free the user of the file handling and transformation
> processes. Additionally, this comes with caveats, among them that given
> the ways in which C++ allows us different accesses, there are some
> interesting processors that work better for that more native approach and
> might not ever justify having an equivalent in Java.
>
> (2) Providing a first class experience in NiFi for MiNiFi flow designing
> making use of the existing UI components as a base and a new design mode
> specifically for providing a clean user experience, letting users operate
> in a different context but a familiar environment. Certainly preferred,
> but how this fit was not clear.
>
> As a crash course in MiNiFi efforts, there have been some key
> contributions that I believe when more strongly united can generate a very
> complete authoring and flow update process. Some of these items are:
>
> - C2 API Protocol specification and implementation in C++ [3]
> - C2 Server with the notion of serving different flows [4] backed by
> varying implementations, one of which was via REST to access NiFi
Templates
> directly [5]
> - Warm Redeploy in MiNiFi Java [6]
>
>
> One of the first questions to come to mind was, “Is within NiFi the right
> place?”
>
> I think there are several reasons that this is reasonable. Seeing some of
> the discussion around the consolidation of common UI components presented
> by Scott Aslan, I think sharing the core of the UI elements between the
> traditional NiFi and MiNiFi flow designs is a logical extension. While
> those flows in MiNiFi are not real-time changes we are dealing with the
> same design elements in creating directed graphs. Additionally, while
> MiNiFi instances will likely not ever have the same breadth of extensions
> supported, there is nothing inherently prohibitive of a more complex graph
> comprised of larger numbers of edges and/or vertices. In my view, the
> mechanics are the same but just being performed in a different contexts.
> In the case of MiNiFi, this is effectively an “offline” context. By
> "offline," I mean we are not strictly bound to loaded components such as
> that in the current NiFi context consisting of items from unpacked NARs.
>
> How can we minimize the impact to NiFi?
>
> Given that this is a new capability, I don’t know that we necessarily need
> to interfere with the existing APIs and could possibly also make this an
> optional feature. From this standpoint, we could develop new endpoints
for
> this specific context. Some of this work would rely on external services
> and/or definitions to drive the different context. For example, without
> NARs acting as the source of available components, something like the C2
> server could provide extensions for MiNiFi instances it is aware of to
help
> drive the experience. This data would be a skeletal view of a component
> definition/interface without any of the inherent code backing it needed.
>
> How do we maximize the utility of these efforts?
>
> With the idea about powering MiNiFi flow design with specific sets of
> components, I could see this extending to even NiFi itself. Consider the
> use case where something like I wish to make use of additional artifacts
> along the lines of [7] and an extension of the current Registry
> implementation. As a simple example, with an understanding of what the
> universe of components is available to my system and not just those
> currently loaded, it would be possible for users to design for other
> instances. With these hooks in place, there are a few permutations of
this
> relative concept that enable us to carry out some of the “nice to haves”
we
> haven’t quite been able to address.
>
> How does this all come together for MiNiFi?
>
> I have created a work of sheer artistic mastery on the Command and Control
> proposal to illustrate a rough sequence and architecture. [8] This
diagram
> shows from entering MiNiFi flow design through to an instance updating to
> the new flow.
>
>
> Overall, Registry was a tremendous step forward for the NiFi ecosystem. I
> think in our current state, we have reached a convergence of capabilities
> that allow us to build higher level abstractions for dataflow management.
> I think the approach of (2) is a great first foray into that realm that
> provides an immediate solution for a gap in our MiNiFi efforts.
> Additionally, this approach lays the foundation for additional
applications
> possibly inclusive of core NiFi.
>
> I look forward to your thoughts and commentary. Certainly many details to
> think and reason about, but I see some incredible potential with the work
> we've done thus far both near and long term.
>
> Thanks for your time!
> --aldrin
>
>
