AndyC/Aldrin, The UI/UX design system sub project currently under discussion will be based off of the work completed during the NiFi Registry and will not contain any UI/UX functionality from NiFi. The long term goal would be to upgrade angular in the NiFi UI in order to leverage this new design system.
-Scotty On Thu, Mar 1, 2018 at 1:45 PM, Kevin Doran <kdo...@apache.org> wrote: > Aldrin, > > Thanks a lot for taking the time to write up such a detailed and well > thought out proposal. > > I really like the idea of publishing flows to NiFi Registry that then get > deployed to MiNiFi agents via the C2 Server. I don't have much to add > there, and I think that you've identified one of the big missing parts to > make that experience user-friendly -- how to design and author the flows in > the first place. > > The MiNiFi Flow Designer would fill that need in a really elegant way with > a low learning curve for NiFi users. Having the ability to design and > author flows in an environment other than the "runtime" environment one is > targeting would be a huge step forward. Not just for MiNiFi, but > multi-environment deployments of NiFi, where one might be developing a flow > in an environment other than where they intend to run it. This would make > the SDLC options for NiFi flows even more flexible, which is a big part of > the goal of the new NiFi Registry. > > It certainly presents some challenges, but I definitely think the upside > for MiNiFi flow authoring would be huge. > > Thanks, > Kevin > > On 2/28/18, 07:53, "Aldrin Piri" <aldrinp...@gmail.com> wrote: > > It appears I accidentally left off the actual links. Here they are in > all > their glory: > > [1] > https://cwiki.apache.org/confluence/display/MINIFI/ > MiNiFi+Command+and+Control > [2] > https://github.com/apache/nifi-minifi/blob/master/ > minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file > [3] https://issues.apache.org/jira/browse/MINIFICPP-36 > [4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2 > [5] > https://github.com/apache/nifi-minifi/tree/master/ > minifi-c2#configuration-providers > [6] > https://github.com/apache/nifi-minifi/blob/master/ > minifi-docs/src/main/markdown/System_Admin_Guide.md# > automatic-warm-redeploy > [7] > https://cwiki.apache.org/confluence/display/NIFI/ > Extension+Repositories+%28aka+Extension+Registry%29+for+ > Dynamically-loaded+Extensions > [8] > https://cwiki.apache.org/confluence/display/MINIFI/ > MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails > > > On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <aldrinp...@gmail.com> > wrote: > > > Hey folks, > > > > With the release of Registry, I’ve been contemplating leveraging it > and > > the current codebase of NiFi to facilitate ease of use for MiNiFi > flow > > design. One of the areas I would like to form a concerted effort > around is > > that of the Command and Control (C2) functionality originally > presented > > when the MiNiFi effort was started and further expounded upon with a > > feature proposal [1]. In that proposal, while the names are dated, > we have > > components that fulfill some of the core building blocks toward the > overall > > vision of C2. > > > > For those not intimately familiar, MiNiFi is primarily configured > via a > > YAML configuration file. [2] The initial idea was that flows would > be > > simple a source, maybe a transform or two, and then transmission. > The > > notion was that hand creation of one of these files would not be > overly > > onerous. It became apparent from questions, however, that users > were doing > > more complex flows and this YAML ranged from tedious to difficult > for those > > situations. This precipitated the best approach currently available, > > utilizing a NiFi instance as a way to generate a template. This > template > > was then exported, and using our provided toolkit to get YAML out > that is > > deployable with your instance. Better, but far from ideal. > > > > A couple approaches came to mind to further narrow this gap of user > > expectation and realities of our configuration. > > > > (1) In our current state, we have heavily relied on convenience of > > component overlap. Generally speaking, we relied on the notion that > NiFi’s > > bundled components are a superset of those found in MiNiFi > > implementations. We could have continued that, but this would not > > necessarily free the user of the file handling and transformation > > processes. Additionally, this comes with caveats, among them that > given > > the ways in which C++ allows us different accesses, there are some > > interesting processors that work better for that more native > approach and > > might not ever justify having an equivalent in Java. > > > > (2) Providing a first class experience in NiFi for MiNiFi flow > designing > > making use of the existing UI components as a base and a new design > mode > > specifically for providing a clean user experience, letting users > operate > > in a different context but a familiar environment. Certainly > preferred, > > but how this fit was not clear. > > > > As a crash course in MiNiFi efforts, there have been some key > > contributions that I believe when more strongly united can generate > a very > > complete authoring and flow update process. Some of these items are: > > > > - C2 API Protocol specification and implementation in C++ [3] > > - C2 Server with the notion of serving different flows [4] backed > by > > varying implementations, one of which was via REST to access NiFi > Templates > > directly [5] > > - Warm Redeploy in MiNiFi Java [6] > > > > > > One of the first questions to come to mind was, “Is within NiFi the > right > > place?” > > > > I think there are several reasons that this is reasonable. Seeing > some of > > the discussion around the consolidation of common UI components > presented > > by Scott Aslan, I think sharing the core of the UI elements between > the > > traditional NiFi and MiNiFi flow designs is a logical extension. > While > > those flows in MiNiFi are not real-time changes we are dealing with > the > > same design elements in creating directed graphs. Additionally, > while > > MiNiFi instances will likely not ever have the same breadth of > extensions > > supported, there is nothing inherently prohibitive of a more complex > graph > > comprised of larger numbers of edges and/or vertices. In my view, > the > > mechanics are the same but just being performed in a different > contexts. > > In the case of MiNiFi, this is effectively an “offline” context. By > > "offline," I mean we are not strictly bound to loaded components > such as > > that in the current NiFi context consisting of items from unpacked > NARs. > > > > How can we minimize the impact to NiFi? > > > > Given that this is a new capability, I don’t know that we > necessarily need > > to interfere with the existing APIs and could possibly also make > this an > > optional feature. From this standpoint, we could develop new > endpoints for > > this specific context. Some of this work would rely on external > services > > and/or definitions to drive the different context. For example, > without > > NARs acting as the source of available components, something like > the C2 > > server could provide extensions for MiNiFi instances it is aware of > to help > > drive the experience. This data would be a skeletal view of a > component > > definition/interface without any of the inherent code backing it > needed. > > > > How do we maximize the utility of these efforts? > > > > With the idea about powering MiNiFi flow design with specific sets of > > components, I could see this extending to even NiFi itself. > Consider the > > use case where something like I wish to make use of additional > artifacts > > along the lines of [7] and an extension of the current Registry > > implementation. As a simple example, with an understanding of what > the > > universe of components is available to my system and not just those > > currently loaded, it would be possible for users to design for other > > instances. With these hooks in place, there are a few permutations > of this > > relative concept that enable us to carry out some of the “nice to > haves” we > > haven’t quite been able to address. > > > > How does this all come together for MiNiFi? > > > > I have created a work of sheer artistic mastery on the Command and > Control > > proposal to illustrate a rough sequence and architecture. [8] This > diagram > > shows from entering MiNiFi flow design through to an instance > updating to > > the new flow. > > > > > > Overall, Registry was a tremendous step forward for the NiFi > ecosystem. I > > think in our current state, we have reached a convergence of > capabilities > > that allow us to build higher level abstractions for dataflow > management. > > I think the approach of (2) is a great first foray into that realm > that > > provides an immediate solution for a gap in our MiNiFi efforts. > > Additionally, this approach lays the foundation for additional > applications > > possibly inclusive of core NiFi. > > > > I look forward to your thoughts and commentary. Certainly many > details to > > think and reason about, but I see some incredible potential with the > work > > we've done thus far both near and long term. > > > > Thanks for your time! > > --aldrin > > > > > > > >
