You can get a lot of mileage using the following suggestions: - There's nothing preventing you from storing the flow.xml.gz file into version control. While this file is binary and you won't see any diffs on it, your VCS (git, svn, etc.) won't have a problem storing it. If storing the flow.xml.gz as binary is a problem, then you could simply uncompress it to flow.xml first and version control that.
- You might likewise want to version control your entire conf directory. - For any processor properties which can be configured using NIFI's expression language, you can reference System properties and/or environment variables. This allows you to have variant configurations based on the deployment context. Things like hostnames or database connection properties could in theory be configured externally. - Your custom controller services can likewise be configured to utilize system properties or environment vars. Most standard controller services don't have this capability yet, but work is being done that might help in this area as well. I personally don't like using templates to solve your problem. Having flow snippets that need to be stored and applied between dev/int/prod doesn't make a lot of sense to me. Better in my mind is to have a single flow.xml.gz that you can carry between those tiers. NIFI should allow you to get most of the way there today, but if you run into problems, I'm sure you could affect additional changes to help with this issue. Adam On Tue, Apr 28, 2015 at 9:58 AM, Esteban Aliverti < [email protected]> wrote: > Joe, > thanks for your prompt response. > > Let me give you more background about what I'm currently doing: > We are currently a group of 3 developers working on a NiFi workflow. My two > colleagues are in charge of building the individual processors required for > the workflow we are trying to create. My role, even if I'm also a Processor > developer myself, is kind of the "integrator" of these processors. I have a > local NiFi running and I'm in charge of the "workflow" design. I use this > local instance as a playground for develop and initial testing. > Once I've updated my local workflow, I usually "export" it into another > instance that is currently being used by QA. This QA instance is only > modified after I validate that my local instance is stable enough. Given > that there are a number of other developers using this QA instance as > clients, the update process is not so easy. Before I introduce any change > in QA, I need to coordinate with the QA department first. This is one of > the biggest reasons I keep a local NiFi instance for development. I can do > whatever I want whenever I want to in my local instance but I need to be > very careful when I modify the QA instance. > > I have asked similar questions in this forum in the past and one of your > recommendations was to use the same workflow for both QA and DEV. You > suggested me to use an attribute in the FlowFiles as some kind of "flag" so > it can be routed to the DEV branch of the workflow. I see 2 problems with > this approach (maybe because of my own development workflow): > 1.- Sometimes I don't have connectivity to the server where the QA NiFi > instance is. I would like to be able to continue my development tasks even > if I'm offline. > 2.- My workflow is not stateless. I have a bunch of Controller Services > that are used through the workflow that intentionally keep state. If I use > the same NiFi instance for both QA and DEV, I will not only have to add a > new "flag" attribute in my workflows, but I will also has to implement some > kind of mechanism in my Controller Services as well to separate DEV and QA > data. I don't think this is a really good idea. > > Regards, > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > > Esteban Aliverti > - Blog @ http://ilesteban.wordpress.com > > On Tue, Apr 28, 2015 at 3:34 PM, Joe Witt <[email protected]> wrote: > > > Esteban, > > > > Thank you for providing so much detail to what you're trying to do. I > > want to walk through this further with you to better understand the > > perspective. But first I will say that the pain you refer to with > > removing a flow is something we will sort out. You should be able to > > simply right-click and delete so we will improve that until it is > > true. > > > > Can you explain why you want to build on one instance, then make a > > template, then go to another instance? I am wondering if this is a > > sort of dev/integration/production workflow. > > > > I think our general philosophy has been to enable developers to easily > > build new extensions which are effectively unit tested and then put > > those out on the instance where the data is flowing. We then optimize > > extensively around the operations/administrator experience. We've > > seen a consistent theme of need though to clarify how the developer > > experience can/should work. So happy to take more feedback on what > > you're trying to do and also 'why' you're trying to do it so we can > > help guide and so we can learn how to adjust as needed. > > > > Thanks > > Joe > > > > On Tue, Apr 28, 2015 at 8:49 AM, Esteban Aliverti > > <[email protected]> wrote: > > > Hi guys, > > > I was wondering what is the best way to replace a complete workflow in > > > NiFi. > > > My scenario is that I have multiple instances of NiFi running the same > > > workflow. At some point, when changes are introduced into one of the > > > instances (let's say a new processor is introduced) I would like to > > > replicate the changes to the other instances. For small changes, things > > can > > > be easily handled, but for big changes - introduction of a whole new > > > branch, for example - things are not so easy. > > > Given that the workflows I'm currently using are in dev stage, I don't > > mind > > > to lose any FlowFile that already exists in the NiFi instance I'm > > updating. > > > > > > The way I'm currently handling this is: > > > 1.- I create a new template from the NiFi instance where the changes > were > > > introduced. > > > 2.- I export the template as an xml file. > > > 3.- I import the template in the NiFi instance where I want the changes > > to > > > be applied. > > > 4.- I manually remove any processor from the NiFi instance where I want > > the > > > changes to be applied. (for complex workflows, this step is a real > > pita!). > > > 5.- I use the template I've previously imported in the NiFi instance > > where > > > I want the changes to be applied. > > > > > > My question is: what is the best way to achieve this? > > > Is there any way to completely delete the workflow that NiFi is > currently > > > executing? > > > > > > Regards, > > > > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > > > > > > Esteban Aliverti > > > - Blog @ http://ilesteban.wordpress.com > > >
