There are 2 efforts, with somewhat different focus. You are already aware
of the community-driven nipyapi, but there's also an official module
Ientioned before, will be included with the 1.6 release.

Andrew

On Fri, Mar 2, 2018, 8:59 AM Boris Tyukin <bo...@boristyukin.com> wrote:

> Hi Andrew,
>
> thanks for the idea. I've been playing with nipyapi recently so might give
> this a try.
>
> Thanks
>
> On Thu, Mar 1, 2018 at 7:32 PM, Andrew Grande <apere...@gmail.com> wrote:
>
>> Boris,
>>
>> Here's an idea youncould explore _today_.
>>
>> Assume your dev and prod flows live in different bucket/registry
>> instance. Given that you are trying out NiFi 1.6, you should be able to
>> extract the versioned flow from DEV and process it to change the
>> concurrency level for PROD before committing it to the prod registry
>> instance. Any script which understands json would do. nifi-toolkit-cli will
>> take care of extracting and moving flow versions.
>>
>> It's not ideal (yes, would like concurrency to be a customizable flow
>> var), and it assumes an explicit process to promote between environments,
>> but technically it is possible already. The user experience can be improved
>> in the future.
>>
>> Andrew
>>
>>
>> On Thu, Mar 1, 2018, 1:52 PM Kevin Doran <kdo...@apache.org> wrote:
>>
>>> I think you could put it under either project. Ultimately, if we go with
>>> that approach, most (all?) of the logic/enhancement would be in the NiFi
>>> code base during save version / import flow / change version operations, so
>>> probably best to create it there.
>>>
>>>
>>>
>>> Glad you are finding NiFi useful.
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Kevin
>>>
>>>
>>>
>>> *From: *Boris Tyukin <bo...@boristyukin.com>
>>> *Reply-To: *<users@nifi.apache.org>
>>> *Date: *Thursday, March 1, 2018 at 13:44
>>> *To: *<users@nifi.apache.org>
>>> *Subject: *Re: setting processor concurrency based on the
>>> development/production environment
>>>
>>>
>>>
>>> thanks Bryan and Kevin. I will be happy to open a jira - would it be a
>>> NiFi jira or NiFi registry?
>>>
>>>
>>>
>>> I like the approach that Bryan suggested.
>>>
>>>
>>>
>>> I guess for now I will just color code the processors that need to be
>>> changed in production.
>>>
>>>
>>>
>>> P.S. I really, really like where NiFi is going...I've looked at
>>> StreamSets and Cask, but for my purposes, I was looking for a tool when I
>>> can process various tables without creating a flow per table. I was able to
>>> create a very simple flow in NiFi, that will handle 25 tables. My next
>>> project is to handle 600 tables in near real-time. I just could see how I
>>> would do that with StreamSets or Cask, when you have to create a pipeline
>>> per table. I was only being able to do something similar with Apache
>>> Airflow, but airflow cannot do things in near real-time. The concept of
>>> FlowFiles with attributes is a genius idea, and I am blown away with all
>>> the possibilities to extend the functionality of NiFi with custom
>>> processors and Groovy scripts. Awesome job, guys.
>>>
>>>
>>>
>>> On Thu, Mar 1, 2018 at 1:29 PM, Kevin Doran <kdo...@apache.org> wrote:
>>>
>>> Hi Boris,
>>>
>>> Good point regarding concurrent tasks; thanks for sharing!
>>>
>>> This is a great candidate for something that one should be able to
>>> create environment-specific values for, as Bryan suggests. I agree we
>>> should create a NiFi JIRA to track this enhancement.
>>>
>>> Thanks,
>>> Kevin
>>>
>>>
>>> ´╗┐On 3/1/18, 11:44, "Bryan Bende" <bbe...@gmail.com> wrote:
>>>
>>>     Hello,
>>>
>>>     Glad you are having success with NiFi + NiFi Registry!
>>>
>>>     You brought up an interesting point about the concurrent tasks...
>>>
>>>     I think we may want to consider making the concurrent tasks work
>>>     similar to variables, in that we capture the concurrent tasks that
>>> the
>>>     flow was developed with and would use it initially, but then if you
>>>     have modified this value in the target environment it would not
>>>     trigger a local change and would be retained across upgrades so that
>>>     you don't have to reset it.
>>>
>>>     For now you could probably always leave the versioned flow with the
>>>     lower value of 2, then once you are in prod you bump it to 4 until
>>> the
>>>     next upgrade is available, you then revert the local changes, do the
>>>     upgrade, and put it back to 4, but its not ideal because it shows a
>>>     local change the entire time.
>>>
>>>     I don't think there is much you can do differently right now, but I
>>>     think this is a valid case to create a JIRA for.
>>>
>>>     Thanks,
>>>
>>>     Bryan
>>>
>>>     On Thu, Mar 1, 2018 at 11:29 AM, Boris Tyukin <bo...@boristyukin.com>
>>> wrote:
>>>     > Hello NiFi community,
>>>     >
>>>     > started using NiFi recently and fell in love with it! We run 1.6
>>> NiFi alone
>>>     > with new NiFi registry and I am trying to figure out how to
>>> promote NiFi
>>>     > flow, created in VM environment to our cluster.
>>>     >
>>>     > One of the things is "Concurrent Tasks" processor parameter. I
>>> bump it to 2
>>>     > or 4 for some processors in my flow, when I develop / test it in
>>> VM.
>>>     >
>>>     > Then we deploy this flow to a beefy cluster node (with 48 cores)
>>> and want to
>>>     > change concurrency to let's say 8 or 10 or 12 for some processors.
>>>     >
>>>     > Then I work on a new version/make some changes in my VM, and need
>>> to be more
>>>     > shy with concurrency so set it back to 2 or 4.
>>>     >
>>>     > Then the story repeats...
>>>     >
>>>     > Is there a better way than to manually set this parameter? I do
>>> not believe
>>>     > I can use a variable there and have to type the actual number of
>>> tasks.
>>>     >
>>>     >
>>>     > Thanks
>>>     > Boris
>>>
>>>
>>>
>>>
>>
>

Reply via email to