Hello I've had this thread flagged for a while now. The sorts of concerns raised in it have been fairly common patterns of requests from various groups. I believe they are fairly well accounted for in the present list of feature proposals as seen here [1].
Please take a look and advise if you think something is missing. Thanks Joe [1] https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals On Wed, Apr 29, 2015 at 9:46 AM, Esteban Aliverti <[email protected]> wrote: > Thanks Adam, > > I guess I was using templates because I didn't know about the > existence of flow.xml.gz > :). > Now, if I have 2 different instances of NiFi, and I want to override the > workflow in one of them with the workflow of the other, do I need to do > anything else than just overwrite flow.xml.gz? > > Regards, > > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > > Esteban Aliverti > - Blog @ http://ilesteban.wordpress.com > > On Wed, Apr 29, 2015 at 2:04 PM, Adam Taft <[email protected]> wrote: > >> You can get a lot of mileage using the following suggestions: >> >> - There's nothing preventing you from storing the flow.xml.gz file into >> version control. While this file is binary and you won't see any diffs on >> it, your VCS (git, svn, etc.) won't have a problem storing it. If storing >> the flow.xml.gz as binary is a problem, then you could simply uncompress it >> to flow.xml first and version control that. >> >> - You might likewise want to version control your entire conf directory. >> >> - For any processor properties which can be configured using NIFI's >> expression language, you can reference System properties and/or environment >> variables. This allows you to have variant configurations based on the >> deployment context. Things like hostnames or database connection properties >> could in theory be configured externally. >> >> - Your custom controller services can likewise be configured to utilize >> system properties or environment vars. Most standard controller services >> don't have this capability yet, but work is being done that might help in >> this area as well. >> >> I personally don't like using templates to solve your problem. Having flow >> snippets that need to be stored and applied between dev/int/prod doesn't >> make a lot of sense to me. Better in my mind is to have a single >> flow.xml.gz that you can carry between those tiers. NIFI should allow you >> to get most of the way there today, but if you run into problems, I'm sure >> you could affect additional changes to help with this issue. >> >> Adam >> >> >> >> On Tue, Apr 28, 2015 at 9:58 AM, Esteban Aliverti < >> [email protected]> wrote: >> >> > Joe, >> > thanks for your prompt response. >> > >> > Let me give you more background about what I'm currently doing: >> > We are currently a group of 3 developers working on a NiFi workflow. My >> two >> > colleagues are in charge of building the individual processors required >> for >> > the workflow we are trying to create. My role, even if I'm also a >> Processor >> > developer myself, is kind of the "integrator" of these processors. I >> have a >> > local NiFi running and I'm in charge of the "workflow" design. I use this >> > local instance as a playground for develop and initial testing. >> > Once I've updated my local workflow, I usually "export" it into another >> > instance that is currently being used by QA. This QA instance is only >> > modified after I validate that my local instance is stable enough. Given >> > that there are a number of other developers using this QA instance as >> > clients, the update process is not so easy. Before I introduce any change >> > in QA, I need to coordinate with the QA department first. This is one of >> > the biggest reasons I keep a local NiFi instance for development. I can >> do >> > whatever I want whenever I want to in my local instance but I need to be >> > very careful when I modify the QA instance. >> > >> > I have asked similar questions in this forum in the past and one of your >> > recommendations was to use the same workflow for both QA and DEV. You >> > suggested me to use an attribute in the FlowFiles as some kind of "flag" >> so >> > it can be routed to the DEV branch of the workflow. I see 2 problems with >> > this approach (maybe because of my own development workflow): >> > 1.- Sometimes I don't have connectivity to the server where the QA NiFi >> > instance is. I would like to be able to continue my development tasks >> even >> > if I'm offline. >> > 2.- My workflow is not stateless. I have a bunch of Controller Services >> > that are used through the workflow that intentionally keep state. If I >> use >> > the same NiFi instance for both QA and DEV, I will not only have to add a >> > new "flag" attribute in my workflows, but I will also has to implement >> some >> > kind of mechanism in my Controller Services as well to separate DEV and >> QA >> > data. I don't think this is a really good idea. >> > >> > Regards, >> > >> > >> > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX >> > >> > Esteban Aliverti >> > - Blog @ http://ilesteban.wordpress.com >> > >> > On Tue, Apr 28, 2015 at 3:34 PM, Joe Witt <[email protected]> wrote: >> > >> > > Esteban, >> > > >> > > Thank you for providing so much detail to what you're trying to do. I >> > > want to walk through this further with you to better understand the >> > > perspective. But first I will say that the pain you refer to with >> > > removing a flow is something we will sort out. You should be able to >> > > simply right-click and delete so we will improve that until it is >> > > true. >> > > >> > > Can you explain why you want to build on one instance, then make a >> > > template, then go to another instance? I am wondering if this is a >> > > sort of dev/integration/production workflow. >> > > >> > > I think our general philosophy has been to enable developers to easily >> > > build new extensions which are effectively unit tested and then put >> > > those out on the instance where the data is flowing. We then optimize >> > > extensively around the operations/administrator experience. We've >> > > seen a consistent theme of need though to clarify how the developer >> > > experience can/should work. So happy to take more feedback on what >> > > you're trying to do and also 'why' you're trying to do it so we can >> > > help guide and so we can learn how to adjust as needed. >> > > >> > > Thanks >> > > Joe >> > > >> > > On Tue, Apr 28, 2015 at 8:49 AM, Esteban Aliverti >> > > <[email protected]> wrote: >> > > > Hi guys, >> > > > I was wondering what is the best way to replace a complete workflow >> in >> > > > NiFi. >> > > > My scenario is that I have multiple instances of NiFi running the >> same >> > > > workflow. At some point, when changes are introduced into one of the >> > > > instances (let's say a new processor is introduced) I would like to >> > > > replicate the changes to the other instances. For small changes, >> things >> > > can >> > > > be easily handled, but for big changes - introduction of a whole new >> > > > branch, for example - things are not so easy. >> > > > Given that the workflows I'm currently using are in dev stage, I >> don't >> > > mind >> > > > to lose any FlowFile that already exists in the NiFi instance I'm >> > > updating. >> > > > >> > > > The way I'm currently handling this is: >> > > > 1.- I create a new template from the NiFi instance where the changes >> > were >> > > > introduced. >> > > > 2.- I export the template as an xml file. >> > > > 3.- I import the template in the NiFi instance where I want the >> changes >> > > to >> > > > be applied. >> > > > 4.- I manually remove any processor from the NiFi instance where I >> want >> > > the >> > > > changes to be applied. (for complex workflows, this step is a real >> > > pita!). >> > > > 5.- I use the template I've previously imported in the NiFi instance >> > > where >> > > > I want the changes to be applied. >> > > > >> > > > My question is: what is the best way to achieve this? >> > > > Is there any way to completely delete the workflow that NiFi is >> > currently >> > > > executing? >> > > > >> > > > Regards, >> > > > >> > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX >> > > > >> > > > Esteban Aliverti >> > > > - Blog @ http://ilesteban.wordpress.com >> > > >> > >>
