Re: Re: Re: Re: 答复: Re: Is there a REST API to run a dataflow on demand?
Hi Luby,
James Wing's solution is the way I would go. You can typically keep that
second upstream source processor disabled. When you want to trigger the
flow dynamically, you can do that with a script that would enable the
processor, and push the data you want to run through the flow to that
processor, and then disable the processor again once the data has been sent
through the flow. You could even automate the disabling of that processor
with Wait/Notify processors that trigger a script to disable that second
upstream source processor.
NiFi has many small building blocks that can be used to facilitate more
complicated features. It's a toolbox. There might not be a t-square in
there at the start, for instance, but the parts are in there to make a
great t-square. :)
On Thu, Feb 22, 2018 at 9:22 PM Sivaprasanna
wrote:
> I believe even if it is where to be made available as part of NiFi REST
> API, I think it is internally going to call the start and stop APIs.
>
> On Fri, 23 Feb 2018 at 6:54 AM, wrote:
>
> > Yes, that is what I do currently.
> >
> > But I think it will be better if NiFi can support this feature natively.
> >
> >
> >
> > 发件人:
> > "Andrew Grande"
> > 收件人:
> > dev@nifi.apache.org
> > 日期:
> > 2018/02/23 09:07
> > 主题:
> > Re: Re: Re: 答复: Re: Is there a REST API to run a dataflow on demand?
> >
> >
> >
> > One could write a script and call it in 1 step. I don't believe there is
> > anything available OOTB.
> >
> > Andrew
> >
> > On Thu, Feb 22, 2018, 7:58 PM wrote:
> >
> > > Thanks a lot for your help.
> > >
> > > Yes. that is what I do to trigger a dataflow on demand.
> > > But I want to know if there is an API that I can do this in one step.
> > >
> > >
> > >
> > > 发件人:
> > > "Daniel Chaffelson"
> > > 收件人:
> > > dev@nifi.apache.org
> > > 日期:
> > > 2018/02/23 04:46
> > > 主题:
> > > Re: Re: 答复: Re: Is there a REST API to run a dataflow on demand?
> > >
> > >
> > >
> > > Hi Boying,
> > >
> > > I have been working on a NiFi Python Client SDK that might help you
> > here,
> > > as the goal is to be able to replicate everyday actions taken in the
> > NiFi
> > > GUI as well as extending it for CICD/SDLC work.
> > > For example with the following commands you would:
> > >
> > >1. get the reference object for a processor
> > >2. stop it if it is running
> > >3. change the scheduling period to 3s (or most other parameters)
> > >4. start it again
> > >
> > >
> > > import nipyapi
> > > processor_state_1 = nipyapi.canvas.get_processor('MyProcessor')
> > > nipyapi.canvas.schedule_processor(processor, scheduled=False)
> > > update = nipyapi.nifi.ProcessorConfigDTO(
> > > scheduling_period='3s'
> > > )
> > > processor_state_2 = nipyapi.canvas.update_processor(processor, update)
> > > nipyapi.canvas.schedule_processor(processor, scheduled=True)
> > >
> > > If you need a different set of steps then please let me know and
> perhaps
> > I
> > > can help.
> > > Those commands are currently in the master branch awaiting release:
> > > https://github.com/Chaffelson/nipyapi
> > >
> > > Thanks,
> > > Dan
> > >
> > > On Thu, Feb 22, 2018 at 7:41 AM wrote:
> > >
> > > > Thanks very much, I'll try your suggestions.
> > > >
> > > >
> > > >
> > > > 发件人:
> > > > James Wing
> > > > 收件人:
> > > > NiFi Dev List
> > > > 日期:
> > > > 2018/02/22 14:05
> > > > 主题:
> > > > Re: 答复: Re: Is there a REST API to run a dataflow on demand?
> > > >
> > > >
> > > >
> > > > The NiFi API can be used to start and stop processors or process
> > groups,
> > > > and this might solve your use case. But NiFi does not have an API to
> > > run
> > > > a
> > > > processor only once, immediately, separate from its configured
> > schedule.
> > > I
> > > > have solved similar problems in the past by creating two separate
> > > upstream
> > > > sources - one for scheduled operation, and one for ad-hoc operation.
> > > > GenerateFlowFile, GetFile, or similar processors can be used to
> inject
> > a
> > > > flowfile where you need to kick off the flow.
> > > >
> > > > Thanks,
> > > >
> > > > James
> > > >
> > > > On Wed, Feb 21, 2018 at 5:57 PM, wrote:
> > > >
> > > > > Thanks a lot.
> > > > >
> > > > > But I want to know if there is a REST API that triggers a dataflow
> > on
> > > > > demand?
> > > > > I don't find the API in the page.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 发件人:
> > > > > Charlie Meyer
> > > > > 收件人:
> > > > > dev@nifi.apache.org
> > > > > 日期:
> > > > > 2018/02/22 09:36
> > > > > 主题:
> > > > > Re: Is there a REST API to run a dataflow on demand?
> > > > >
> > > > >
> > > > >
> > > > > Yep, when you make the changes in the UI, open developer tools in
> > your
> > > > > browser and see what calls to the nifi api it is making then mimic
> > > those
> > > > > with code.
> > > > >
> > > > > The nifi team also kindly publishes
> > > > > https://nifi.apache.org/docs/nifi-docs/rest-api/index.html which
> > help
> > > a
> > > > > lot.
> > > > >
> > > > > Best of luck!
> > > > >
> > > > > -Char
Re: Re: Re: Re: 答复: Re: Is there a REST API to run a dataflow on demand?
I believe even if it is where to be made available as part of NiFi REST
API, I think it is internally going to call the start and stop APIs.
On Fri, 23 Feb 2018 at 6:54 AM, wrote:
> Yes, that is what I do currently.
>
> But I think it will be better if NiFi can support this feature natively.
>
>
>
> 发件人:
> "Andrew Grande"
> 收件人:
> dev@nifi.apache.org
> 日期:
> 2018/02/23 09:07
> 主题:
> Re: Re: Re: 答复: Re: Is there a REST API to run a dataflow on demand?
>
>
>
> One could write a script and call it in 1 step. I don't believe there is
> anything available OOTB.
>
> Andrew
>
> On Thu, Feb 22, 2018, 7:58 PM wrote:
>
> > Thanks a lot for your help.
> >
> > Yes. that is what I do to trigger a dataflow on demand.
> > But I want to know if there is an API that I can do this in one step.
> >
> >
> >
> > 发件人:
> > "Daniel Chaffelson"
> > 收件人:
> > dev@nifi.apache.org
> > 日期:
> > 2018/02/23 04:46
> > 主题:
> > Re: Re: 答复: Re: Is there a REST API to run a dataflow on demand?
> >
> >
> >
> > Hi Boying,
> >
> > I have been working on a NiFi Python Client SDK that might help you
> here,
> > as the goal is to be able to replicate everyday actions taken in the
> NiFi
> > GUI as well as extending it for CICD/SDLC work.
> > For example with the following commands you would:
> >
> >1. get the reference object for a processor
> >2. stop it if it is running
> >3. change the scheduling period to 3s (or most other parameters)
> >4. start it again
> >
> >
> > import nipyapi
> > processor_state_1 = nipyapi.canvas.get_processor('MyProcessor')
> > nipyapi.canvas.schedule_processor(processor, scheduled=False)
> > update = nipyapi.nifi.ProcessorConfigDTO(
> > scheduling_period='3s'
> > )
> > processor_state_2 = nipyapi.canvas.update_processor(processor, update)
> > nipyapi.canvas.schedule_processor(processor, scheduled=True)
> >
> > If you need a different set of steps then please let me know and perhaps
> I
> > can help.
> > Those commands are currently in the master branch awaiting release:
> > https://github.com/Chaffelson/nipyapi
> >
> > Thanks,
> > Dan
> >
> > On Thu, Feb 22, 2018 at 7:41 AM wrote:
> >
> > > Thanks very much, I'll try your suggestions.
> > >
> > >
> > >
> > > 发件人:
> > > James Wing
> > > 收件人:
> > > NiFi Dev List
> > > 日期:
> > > 2018/02/22 14:05
> > > 主题:
> > > Re: 答复: Re: Is there a REST API to run a dataflow on demand?
> > >
> > >
> > >
> > > The NiFi API can be used to start and stop processors or process
> groups,
> > > and this might solve your use case. But NiFi does not have an API to
> > run
> > > a
> > > processor only once, immediately, separate from its configured
> schedule.
> > I
> > > have solved similar problems in the past by creating two separate
> > upstream
> > > sources - one for scheduled operation, and one for ad-hoc operation.
> > > GenerateFlowFile, GetFile, or similar processors can be used to inject
> a
> > > flowfile where you need to kick off the flow.
> > >
> > > Thanks,
> > >
> > > James
> > >
> > > On Wed, Feb 21, 2018 at 5:57 PM, wrote:
> > >
> > > > Thanks a lot.
> > > >
> > > > But I want to know if there is a REST API that triggers a dataflow
> on
> > > > demand?
> > > > I don't find the API in the page.
> > > >
> > > >
> > > >
> > > >
> > > > 发件人:
> > > > Charlie Meyer
> > > > 收件人:
> > > > dev@nifi.apache.org
> > > > 日期:
> > > > 2018/02/22 09:36
> > > > 主题:
> > > > Re: Is there a REST API to run a dataflow on demand?
> > > >
> > > >
> > > >
> > > > Yep, when you make the changes in the UI, open developer tools in
> your
> > > > browser and see what calls to the nifi api it is making then mimic
> > those
> > > > with code.
> > > >
> > > > The nifi team also kindly publishes
> > > > https://nifi.apache.org/docs/nifi-docs/rest-api/index.html which
> help
> > a
> > > > lot.
> > > >
> > > > Best of luck!
> > > >
> > > > -Charlie
> > > >
> > > > On Wed, Feb 21, 2018 at 7:34 PM, wrote:
> > > >
> > > > > Hi, team,
> > > > >
> > > > > We set up several NiFi dataflows for data processing.
> > > > > These dataflows are configured to run once per day in the
> midnight.
> > > > >
> > > > > But sometimes, some dataflows are failed,I want to run the
> dataflow
> > > > again
> > > > > immediately after fixing the issue instead of waiting for running
> it
> > > in
> > > > > the midnight to
> > > > > make sure that the issue is really fixed.
> > > > >
> > > > > The only way I know to do this is to change the time of running
> the
> > > > > dataflow to the 5 mintutes from now for example
> > > > > and then change it back to midnight.
> > > > >
> > > > > It's a little inconvenient.
> > > > >
> > > > > Is there any REST API that I can use to trigger the dataflow on
> > demand
> > > > > i.e. without change the time back and forth?
> > > > >
> > > > > Thanks
> > > > >
> > > > > Boying
> > > > >
> > > > >
> > > > >
> > > > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保
> 存
> > 、
> > > 对
> > > > 外
> > > > > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件
> 告
> > 知
> > > 发
> > > > 件
> > > > > 人，并
