Joe, In terms of sharing a flow configuration across a development team, do you recommend against version controlling the underlying flow.xml.gz file?
I realize it is very specific to the internals of NiFi, but if you need to replicate an exact flow across environments (developer machine, integration, production), and possibly automatically deploy a flow, I wasn't sure if templates solved that problem. Admittedly I don't know much about templates. The obvious downside is that the flow.xml.gz is not friendly for merges when two people modify it, so you have to be extremely careful that you are modifying the latest one. One approach is to make a rule that everyone on the Dev team always modify the flow on a shared NiFi instance, and copy the flow.xml back to version control from that instance. This way if two people modify it at the same time, both their changes should be captured. Thoughts? -Bryan On Thursday, February 19, 2015, Joe Witt <[email protected]> wrote: > Hello > > This exchange occurred today in the #nifi IRC chatroom. Wanted to > memorialize it here and respond. > > Big thanks to Josh Elser for manning the booth and helping out! > > The question: > +++++++++++++++++ > 11:57 < ilesteban> Hi guys. I started to play around with NiFi > recently. I'm planning to start using it for a project I'm currently > working on. > 11:57 < ilesteban> <ilesteban> I have some bpm background and I'm > trying to see what are the things than bpm and nifi have in common and > what are the things that they don't. > 11:57 < ilesteban> <ilesteban> One of the first things I noticed is > that, apparently, in Nifi there is no separation between the workflow > definition and the workflow execution environment. > 11:57 < ilesteban> <ilesteban> You define and execute your wrokflow in > the same environment. > 11:57 < ilesteban> <ilesteban> I like the idea, but I'm struggling to > see how this fit into a distributed team environment. > 11:57 < ilesteban> <ilesteban> Let's say I want to version a worflow > (in git, svn, or whatever) so I can share it among different team > members. I also want to keep versions from different milestones in my > project. > 11:57 < ilesteban> <ilesteban> How can I do that? > 11:57 < ilesteban> <ilesteban> Do I have to version the whole NiFi app? > 11:57 < ilesteban> <ilesteban> Is there any way to extract just the > workflow definition (and maybe configuration) in a more > versioning-friendly format? > 11:57 < ilesteban> <ilesteban> The way I used to do this with bpm > (jBPM in particular) was to export the process definition into XML and > then version that XML. Another team member could then get that XML, > import it into an editor and con > 12:03 < elserj> Last I knew, there wasn't good versioning built into a > workflow > 12:03 < elserj> and it wasn't serialized in such a way that was > conducive to sharing among users > 12:03 < elserj> i'm not sure if that's be changed > 12:03 < ilesteban> elserj, I'm thinking about templates > 12:03 < elserj> you'd probably be better off asking on > [email protected] though :) > 12:04 < ilesteban> maybe I can export a template and version it > 12:04 < ilesteban> elserj, I'll do that > 12:14 < elserj> sorry I'm not of much help. I don't do much with Nifi. > I just try to help out where I can :) > +++++++++++++++ > > Response: > > NiFi certainly does take a different approach to the more common > 'design and deploy' model. NiFI provides a real-time modification > capability that affords immediate feedback. This model is really > powerful in that it lets you immediately see the good or bad of > changes being made. Further you can safely copy the live flow of data > off for evaluation and testing without impacting the production flow. > The whole idea is to foster and encourage iteration which we believe > is best done in the fires of real data. > > But the point you bring up about being able to capture or save a > design and share it with others is great too. NiFi supports templates > for this exact reason. You can make real proven dataflows and then > generate templates of them, version them as you wish, and share them > with others. They can import them into their flows, improve them, and > share them back and so on. I think there is more we can do to improve > that but that will come with feedback and based on what folks are > trying to accomplish. > > Thanks > Joe >
