Wait only if flagged?

[Peter Wicks (pwicks)]

I have a database flow, which is a sequence of 4 processors. For database 
performance reasons I need to make sure only one file is in this section of the 
flow at a time. Not just one file per queue/processor, but for the whole 
section.

I feel like I should be able to use Wait/Notify to do this, but Wait/Notify 
seem to do the opposite. I want to allow a file to go into the flow unless 
there is a flag. If there is a flag I want the FlowFile to wait until the flag 
is cleared.

Thoughts?

Thanks,
 Peter

RE: [EXT] Re: Calling NIFI job from any Enterprise Scheduler

[Peter Wicks (pwicks)]

A lot of enterprise schedulers have an option to post to an HTTP endpoint. In 
the past I’ve used NiFi’s ListenHTTP processor to allow for remote triggering 
of a flow.

But really any of the Listen/Consume processors could potentially be used for 
remote triggering:


  *   ListenHTTP or ListenWebSocket (might be a good way to trigger from some 
kind of Javascript front end, less helpful form an enterprise scheduler)
  *   ConsumeJMSQueue/Topic
  *   ConsumeMQTT
  *   ListenTCP/ListenUDP

You could also use a file as a trigger by writing an empty file to a directory 
and then using FetchFile to trigger the flow. You could even use this as a way 
to pass in a configuration file to a flow… ☺ all sorts of options.

--Peter

From: Jeremy Dyer [mailto:jdy...@gmail.com]
Sent: Tuesday, November 14, 2017 02:24
To: users@nifi.apache.org
Subject: [EXT] Re: Calling NIFI job from any Enterprise Scheduler

Siva - I find the best way to trigger a NiFi workload from any sort of 
enterprise scheduler is to start the workflow you wish to trigger with a 
*ConsumeJMS processor and then have that enterprise scheduler fire the event to 
start the job to that configured endpoint. In this manner you can use your 
enterprise scheduler to invoke the flow without any major modifications.

- Jeremy Dyer

On Mon, Nov 13, 2017 at 1:20 PM, Sivakumar, S 
> wrote:
Hi Team,
I have requirement to call NIFI job from any Enterprise Scheduler (for ex: 
tidal Enterprise Scheduler). What is the way to call nifi job from a command 
line so that it can triggered from external program?

-Siva

Re: Splunk Processor - Re-play

[Sivakumar, S]

Hello,
Even I provide Earliest Time and Latest Time with “Provided” strategy, it is 
causing data duplicate in the system. The same number of records are repeated 
as many times, when the workflow runs.

My question is, even if specify the "Managed" time strategy, how to go back to 
certain time ranges (basically the re-pulling the same records which already 
pulled), by changing the stored value somewhere in the system where processor 
refers to that. Could be from persistent provenance repo. How to tweak the time 
range values from those repo?

-Siva

From: Bryan Bende 
Reply-To: "users@nifi.apache.org" 
Date: Monday, November 13, 2017 at 11:57 PM
To: "users@nifi.apache.org" 
Cc: "Bharani, Manish" 
Subject: Re: Splunk Processor - Re-play

Hello,
If you want to specify Earliest Time and Latest Time, then you need to change 
Time Range Strategy to 'Provided".
The "Managed" time ranger strategies are meant to let the processor calculate 
the time ranges for you on each execution and you can not specify time ranges 
when using those strategies.
-Bryan

On Mon, Nov 13, 2017 at 9:03 AM, Sivakumar, S 
> wrote:
Hi Folks,
I am newbie to nifi tool. I am using GetSplunk 1.4.0 processor to pull data 
from Splunk. Somewhat I have managed pulled the data for T-3, but I want to 
re-play and again want to pull data with some more additional transformation 
added in the splunk query.

I have below two problems

1. No data is pulled in the Flow control.
2. If I change the “Time Range Strategy “
a. Provided, the SAME data is pulled as many number of times 
till the Flow control runs,
b. Managed from Beginning, it is pulled huge volume of data.

Please advise me how to replay the flow control from where I want and continue 
from that point onwards


[cid:image001.png@01D35D20.91679E20]

-Siva

Re: Wait only if flagged?

[Matt Burgess]

Peter,

I haven't tried this, but my knee-jerk reaction is to switch the roles
of the "wait" and "success" relationships. Maybe you can send the
"wait" relationship downstream and route the "success" one back to
Wait. Then when the flag is "cleared", the flow files will start going
to the "success" relationship which is routed back to wait.

Regards,
Matt

On Mon, Nov 13, 2017 at 7:21 PM, Peter Wicks (pwicks)  wrote:
> I have a database flow, which is a sequence of 4 processors. For database
> performance reasons I need to make sure only one file is in this section of
> the flow at a time. Not just one file per queue/processor, but for the whole
> section.
>
>
>
> I feel like I should be able to use Wait/Notify to do this, but Wait/Notify
> seem to do the opposite. I want to allow a file to go into the flow unless
> there is a flag. If there is a flag I want the FlowFile to wait until the
> flag is cleared.
>
>
>
> Thoughts?
>
>
>
> Thanks,
>
>  Peter

RE: [EXT] Re: Wait only if flagged?

[Peter Wicks (pwicks)]

Matt,

I played around with your idea. I haven't been able to get it to work.

First flow file comes in, goes out the wait relationship. Now how do we stop 
the next FlowFile from going out the wait relationship and entering the flow? 
Well we'd have to use Notify along with some scheduling on Wait so multiple 
FlowFile's don't come through right away; so I put a Notify processor right 
after Wait. Now the second FlowFile enters, goes out the success relationship 
because there is a signal count, then immediately goes down wait since the 
signal is clear...

If this use case seems common enough, I was thinking that the solution might be 
to have add an option on Wait that lets you choose whether you want to "Wait 
for signal" or "Wait only if signaled". This would restore the relationships 
back to their proper uses. Then you'd use the Notify property 'Signal Counter 
Delta' mode of 0 at the end of the flow.  Flow would be something like Wait 
(Wait only if signaled)->Notify (Signal to 1)->... Processors ...->Notify 
(Signal to 0). Or perhaps as part of the feature, "Wait only if signaled" mode 
would do the notify step internally.

Would still like to find a clean/easy way to do this though.

Thanks,
 Peter

-Original Message-
From: Matt Burgess [mailto:mattyb...@apache.org] 
Sent: Tuesday, November 14, 2017 08:53
To: users@nifi.apache.org
Subject: [EXT] Re: Wait only if flagged?

Peter,

I haven't tried this, but my knee-jerk reaction is to switch the roles
of the "wait" and "success" relationships. Maybe you can send the
"wait" relationship downstream and route the "success" one back to
Wait. Then when the flag is "cleared", the flow files will start going
to the "success" relationship which is routed back to wait.

Regards,
Matt

On Mon, Nov 13, 2017 at 7:21 PM, Peter Wicks (pwicks)  wrote:
> I have a database flow, which is a sequence of 4 processors. For database
> performance reasons I need to make sure only one file is in this section of
> the flow at a time. Not just one file per queue/processor, but for the whole
> section.
>
>
>
> I feel like I should be able to use Wait/Notify to do this, but Wait/Notify
> seem to do the opposite. I want to allow a file to go into the flow unless
> there is a flag. If there is a flag I want the FlowFile to wait until the
> flag is cleared.
>
>
>
> Thoughts?
>
>
>
> Thanks,
>
>  Peter

Re: [EXT] Re: Wait only if flagged?

[Koji Kawamura]

Hi Peter,

I think I have done similar thing before. This flow template may be
useful in your case, too.
https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244

Thanks,
Koji

On Tue, Nov 14, 2017 at 12:58 PM, Peter Wicks (pwicks)
 wrote:
> Matt,
>
> I played around with your idea. I haven't been able to get it to work.
>
> First flow file comes in, goes out the wait relationship. Now how do we stop 
> the next FlowFile from going out the wait relationship and entering the flow? 
> Well we'd have to use Notify along with some scheduling on Wait so multiple 
> FlowFile's don't come through right away; so I put a Notify processor right 
> after Wait. Now the second FlowFile enters, goes out the success relationship 
> because there is a signal count, then immediately goes down wait since the 
> signal is clear...
>
> If this use case seems common enough, I was thinking that the solution might 
> be to have add an option on Wait that lets you choose whether you want to 
> "Wait for signal" or "Wait only if signaled". This would restore the 
> relationships back to their proper uses. Then you'd use the Notify property 
> 'Signal Counter Delta' mode of 0 at the end of the flow.  Flow would be 
> something like Wait (Wait only if signaled)->Notify (Signal to 1)->... 
> Processors ...->Notify (Signal to 0). Or perhaps as part of the feature, 
> "Wait only if signaled" mode would do the notify step internally.
>
> Would still like to find a clean/easy way to do this though.
>
> Thanks,
>  Peter
>
> -Original Message-
> From: Matt Burgess [mailto:mattyb...@apache.org]
> Sent: Tuesday, November 14, 2017 08:53
> To: users@nifi.apache.org
> Subject: [EXT] Re: Wait only if flagged?
>
> Peter,
>
> I haven't tried this, but my knee-jerk reaction is to switch the roles
> of the "wait" and "success" relationships. Maybe you can send the
> "wait" relationship downstream and route the "success" one back to
> Wait. Then when the flag is "cleared", the flow files will start going
> to the "success" relationship which is routed back to wait.
>
> Regards,
> Matt
>
> On Mon, Nov 13, 2017 at 7:21 PM, Peter Wicks (pwicks)  
> wrote:
>> I have a database flow, which is a sequence of 4 processors. For database
>> performance reasons I need to make sure only one file is in this section of
>> the flow at a time. Not just one file per queue/processor, but for the whole
>> section.
>>
>>
>>
>> I feel like I should be able to use Wait/Notify to do this, but Wait/Notify
>> seem to do the opposite. I want to allow a file to go into the flow unless
>> there is a flag. If there is a flag I want the FlowFile to wait until the
>> flag is cleared.
>>
>>
>>
>> Thoughts?
>>
>>
>>
>> Thanks,
>>
>>  Peter

Splunk Processor - Re-play

[Sivakumar, S]

Hi Folks,
I am newbie to nifi tool. I am using GetSplunk 1.4.0 processor to pull data 
from Splunk. Somewhat I have managed pulled the data for T-3, but I want to 
re-play and again want to pull data with some more additional transformation 
added in the splunk query.

I have below two problems

1. No data is pulled in the Flow control.
2. If I change the “Time Range Strategy “
a. Provided, the SAME data is pulled as many number of times 
till the Flow control runs,
b. Managed from Beginning, it is pulled huge volume of data.

Please advise me how to replay the flow control from where I want and continue 
from that point onwards


[cid:image001.png@01D35CB6.46544F10]

-Siva

GetSFTP error

[Yatsevitch, Mcgregor J]

Hi,
I'm trying to get a GetSFTP processor to work.
I've provided the private key path.
The error in nifi-app.log says:
Null.PointerException: null
at com.jcraft.jsch.KeyPairPKCS8.getPublicKeyBlob(keyPairPKCS8.java:199)
at com.jcraft.jsch.IdentityFile.getPublicKeyBlob(IdentityFile.java:199)

The file in the private key path is a pem file, created with an openssl pkcs12 
command.

Has anyone come across this? What do I have wrong?
Thanks

Re: Splunk Processor - Re-play

[Bryan Bende]

Hello,

If you want to specify Earliest Time and Latest Time, then you need to
change Time Range Strategy to 'Provided".

The "Managed" time ranger strategies are meant to let the processor
calculate the time ranges for you on each execution and you can not specify
time ranges when using those strategies.

-Bryan


On Mon, Nov 13, 2017 at 9:03 AM, Sivakumar, S 
wrote:

> Hi Folks,
>
> I am newbie to nifi tool. I am using GetSplunk 1.4.0 processor to pull
> data from Splunk. Somewhat I have managed pulled the data for T-3, but I
> want to re-play and again want to pull data with some more additional
> transformation added in the splunk query.
>
>
>
> I have below two problems
>
>
>
> 1. No data is pulled in the Flow control.
>
> 2. If I change the “Time Range Strategy “
>
> a. Provided, the SAME data is pulled as many number of
> times till the Flow control runs,
>
> b. Managed from Beginning, it is pulled huge volume of
> data.
>
>
>
> Please advise me how to replay the flow control from where I want and
> continue from that point onwards
>
>
>
>
>
>
>
> -Siva
>

Calling NIFI job from any Enterprise Scheduler

[Sivakumar, S]

Hi Team,
I have requirement to call NIFI job from any Enterprise Scheduler (for ex: 
tidal Enterprise Scheduler). What is the way to call nifi job from a command 
line so that it can triggered from external program?

-Siva

Re: Calling NIFI job from any Enterprise Scheduler

[Jeremy Dyer]

Siva - I find the best way to trigger a NiFi workload from any sort of
enterprise scheduler is to start the workflow you wish to trigger with a
*ConsumeJMS processor and then have that enterprise scheduler fire the
event to start the job to that configured endpoint. In this manner you can
use your enterprise scheduler to invoke the flow without any major
modifications.

- Jeremy Dyer

On Mon, Nov 13, 2017 at 1:20 PM, Sivakumar, S 
wrote:

> Hi Team,
>
> I have requirement to call NIFI job from any Enterprise Scheduler (for ex:
> tidal Enterprise Scheduler). What is the way to call nifi job from a
> command line so that it can triggered from external program?
>
>
>
> -Siva
>