Bryan, I do think sharing the flow.xml.gz is problematic:
- It contains the sensitive properties (albeit encrypted). Templates do not include those values at all. So with templates you're not unnecessarily increasing exposure of those values. - It is all or nothing. Templates are 'pieces of the flow' and you can treat them as such, import them in that manner, etc.. - automatic/continuous deployment for testing/etc.. I think templates could also be used for this. All the invocations go through the REST api anyway so a client could be written to do the automatic deployment stuff Multiple devs editng a shared instance: - This is interesting. So the situation you describe is one where there is so much live-change activity going on between 2 or more people that the optimistic-locking model NiFi has is problematic? One thing I know we've sort of kicked around previously is that of isolating the scope of the 'flow lock' to a single process group. That way two people could work on sibling process groups with no worry about one getting the lock before the other as is the case now. If this is really a problem you see that is worth solving though that is really an exciting problem for us to have. Thanks Joe On Thu, Feb 19, 2015 at 11:13 PM, Bryan Bende <[email protected]> wrote: > Joe, > > In terms of sharing a flow configuration across a development team, do you > recommend against version controlling the underlying flow.xml.gz file? > > I realize it is very specific to the internals of NiFi, but if you need to > replicate an exact flow across environments (developer machine, > integration, production), and possibly automatically deploy a flow, I > wasn't sure if templates solved that problem. Admittedly I don't know much > about templates. > > The obvious downside is that the flow.xml.gz is not friendly for merges > when two people modify it, so you have to be extremely careful that you are > modifying the latest one. One approach is to make a rule that everyone on > the Dev team always modify the flow on a shared NiFi instance, and copy the > flow.xml back to version control from that instance. This way if two people > modify it at the same time, both their changes should be captured. > > Thoughts? > > -Bryan > > On Thursday, February 19, 2015, Joe Witt <[email protected]> wrote: > >> Hello >> >> This exchange occurred today in the #nifi IRC chatroom. Wanted to >> memorialize it here and respond. >> >> Big thanks to Josh Elser for manning the booth and helping out! >> >> The question: >> +++++++++++++++++ >> 11:57 < ilesteban> Hi guys. I started to play around with NiFi >> recently. I'm planning to start using it for a project I'm currently >> working on. >> 11:57 < ilesteban> <ilesteban> I have some bpm background and I'm >> trying to see what are the things than bpm and nifi have in common and >> what are the things that they don't. >> 11:57 < ilesteban> <ilesteban> One of the first things I noticed is >> that, apparently, in Nifi there is no separation between the workflow >> definition and the workflow execution environment. >> 11:57 < ilesteban> <ilesteban> You define and execute your wrokflow in >> the same environment. >> 11:57 < ilesteban> <ilesteban> I like the idea, but I'm struggling to >> see how this fit into a distributed team environment. >> 11:57 < ilesteban> <ilesteban> Let's say I want to version a worflow >> (in git, svn, or whatever) so I can share it among different team >> members. I also want to keep versions from different milestones in my >> project. >> 11:57 < ilesteban> <ilesteban> How can I do that? >> 11:57 < ilesteban> <ilesteban> Do I have to version the whole NiFi app? >> 11:57 < ilesteban> <ilesteban> Is there any way to extract just the >> workflow definition (and maybe configuration) in a more >> versioning-friendly format? >> 11:57 < ilesteban> <ilesteban> The way I used to do this with bpm >> (jBPM in particular) was to export the process definition into XML and >> then version that XML. Another team member could then get that XML, >> import it into an editor and con >> 12:03 < elserj> Last I knew, there wasn't good versioning built into a >> workflow >> 12:03 < elserj> and it wasn't serialized in such a way that was >> conducive to sharing among users >> 12:03 < elserj> i'm not sure if that's be changed >> 12:03 < ilesteban> elserj, I'm thinking about templates >> 12:03 < elserj> you'd probably be better off asking on >> [email protected] though :) >> 12:04 < ilesteban> maybe I can export a template and version it >> 12:04 < ilesteban> elserj, I'll do that >> 12:14 < elserj> sorry I'm not of much help. I don't do much with Nifi. >> I just try to help out where I can :) >> +++++++++++++++ >> >> Response: >> >> NiFi certainly does take a different approach to the more common >> 'design and deploy' model. NiFI provides a real-time modification >> capability that affords immediate feedback. This model is really >> powerful in that it lets you immediately see the good or bad of >> changes being made. Further you can safely copy the live flow of data >> off for evaluation and testing without impacting the production flow. >> The whole idea is to foster and encourage iteration which we believe >> is best done in the fires of real data. >> >> But the point you bring up about being able to capture or save a >> design and share it with others is great too. NiFi supports templates >> for this exact reason. You can make real proven dataflows and then >> generate templates of them, version them as you wish, and share them >> with others. They can import them into their flows, improve them, and >> share them back and so on. I think there is more we can do to improve >> that but that will come with feedback and based on what folks are >> trying to accomplish. >> >> Thanks >> Joe >>
