Hey folks,
I have been thinking about establishing a game plan to start pursuing some of the tenets of MiNiFi Command and Control and captured these in a new Feature Proposal on the MiNiFi wiki[1]. With this page, I have created some initial design documentation to aid in articulating some of the cases and functionality we are looking to provide predicated on the discussions surrounding MiNiFi's inception. The initial thrust was that of command and control of MiNiFi instances and standing up additional infrastructure to establish bi-directional communications. There have been some initial community discussions as to what this would look like and the interactions that were expected. For the first effort, accomplishing the ability to have a mechanism to provide updated flows to allow instances to roll over to new flows would be a great value to aid in the management of dataflow. A simpler goal, but comes with a lot of nuance to consider. As is the case with systems detached from core infrastructure and networking, connectivity issues are a concern. When it comes to handling data that is collected and the processing after it has reached its destination, comprehension of the data and its path to this point are important. Provenance and attributes can play a big part of this, but as versions of processing flows may diverge being able to understand what is still of value is critical. It is with this, that the capability of being able to version flows is also of great importance. As a result, I have also added some additional context and background to the initial feature proposal of Configuration Management of Flows [2] to help support the tagging of MiNiFi data. Flow versioning is but one component of this overall CM process and NiFi user SDLC that is important for being able to reason about the data that is received from MiNiFi instances. However, there are many shared benefits that NiFi would also be able to utilize for various needs such as migration of flows between varying levels of production and creating richer context of provenance events in the face of evolving flows and processing. I would appreciate anyone interested in things MiNiFi C&C or has sought better ways to manage NiFi flows between environments to comment on some of the initial content at the linked proposals. Feedback or ideas to help guide the direction of these efforts will aid greatly as we move forward on establishing design. These proposals are likely to introduce new components and interfaces into what is evolving into a NiFi ecosystem of sorts. Being mindful of the use cases that the community has for some of the functionalities that support these goals would allow a more singular approach of the overall management of dataflow despite where it exists in its lifecycle. With features such as these getting incorporated with some of the other components mentioned, we will have the capabilities to do some really exciting and impressive management of data. Thanks! --aldrin [1] https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control [2] https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows#ConfigurationManagementofFlows-FlowVersioning
