thanks Bryan and Kevin. I will be happy to open a jira - would it be a NiFi jira or NiFi registry?
I like the approach that Bryan suggested. I guess for now I will just color code the processors that need to be changed in production. P.S. I really, really like where NiFi is going...I've looked at StreamSets and Cask, but for my purposes, I was looking for a tool when I can process various tables without creating a flow per table. I was able to create a very simple flow in NiFi, that will handle 25 tables. My next project is to handle 600 tables in near real-time. I just could see how I would do that with StreamSets or Cask, when you have to create a pipeline per table. I was only being able to do something similar with Apache Airflow, but airflow cannot do things in near real-time. The concept of FlowFiles with attributes is a genius idea, and I am blown away with all the possibilities to extend the functionality of NiFi with custom processors and Groovy scripts. Awesome job, guys. On Thu, Mar 1, 2018 at 1:29 PM, Kevin Doran <kdo...@apache.org> wrote: > Hi Boris, > > Good point regarding concurrent tasks; thanks for sharing! > > This is a great candidate for something that one should be able to create > environment-specific values for, as Bryan suggests. I agree we should > create a NiFi JIRA to track this enhancement. > > Thanks, > Kevin > > On 3/1/18, 11:44, "Bryan Bende" <bbe...@gmail.com> wrote: > > Hello, > > Glad you are having success with NiFi + NiFi Registry! > > You brought up an interesting point about the concurrent tasks... > > I think we may want to consider making the concurrent tasks work > similar to variables, in that we capture the concurrent tasks that the > flow was developed with and would use it initially, but then if you > have modified this value in the target environment it would not > trigger a local change and would be retained across upgrades so that > you don't have to reset it. > > For now you could probably always leave the versioned flow with the > lower value of 2, then once you are in prod you bump it to 4 until the > next upgrade is available, you then revert the local changes, do the > upgrade, and put it back to 4, but its not ideal because it shows a > local change the entire time. > > I don't think there is much you can do differently right now, but I > think this is a valid case to create a JIRA for. > > Thanks, > > Bryan > > On Thu, Mar 1, 2018 at 11:29 AM, Boris Tyukin <bo...@boristyukin.com> > wrote: > > Hello NiFi community, > > > > started using NiFi recently and fell in love with it! We run 1.6 > NiFi alone > > with new NiFi registry and I am trying to figure out how to promote > NiFi > > flow, created in VM environment to our cluster. > > > > One of the things is "Concurrent Tasks" processor parameter. I bump > it to 2 > > or 4 for some processors in my flow, when I develop / test it in VM. > > > > Then we deploy this flow to a beefy cluster node (with 48 cores) and > want to > > change concurrency to let's say 8 or 10 or 12 for some processors. > > > > Then I work on a new version/make some changes in my VM, and need to > be more > > shy with concurrency so set it back to 2 or 4. > > > > Then the story repeats... > > > > Is there a better way than to manually set this parameter? I do not > believe > > I can use a variable there and have to type the actual number of > tasks. > > > > > > Thanks > > Boris > > > >
