Re: Only get file when a set exists.

[Martijn Dekkers]

Hi Koji, that looks excellent, many thanks!

I did consider wait/notify, but wasn't quite sure - not used wait/notify
before. Many thanks for putting the example together, I will give this a
try.

Martijn

On 28 May 2018 at 05:11, Koji Kawamura  wrote:

> Hi Martin,
>
> Alternative approach is using Wait/Notify processors.
> I have developed similar flow using those before, and it will work
> with your case I believe.
> A NiFi flow template is available here.
> https://gist.github.com/ijokarumawak/06b3b071eeb4d10d8a27507981422edd
>
> Hope this helps,
> Koji
>
>
> On Sun, May 27, 2018 at 11:48 PM, Andrew Grande 
> wrote:
> > Martijn,
> >
> > Here's an idea you could explore. Have the ListFile processor work as
> usual
> > and create a custom component (start with a scripting one to prototype)
> > grouping the filenames as needed. I don't know of the number of files in
> a
> > set is different every time, so trying to be more robust.
> >
> > Once you group and count the set, you can transfer the names to the
> success
> > relationship. Ignore otherwise and wait until the set is full.
> >
> > Andrew
> >
> >
> > On Sun, May 27, 2018, 7:29 AM Martijn Dekkers 
> > wrote:
> >>
> >> Hello all,
> >>
> >> I am trying to work out an issue with little success.
> >>
> >> I need to ingest files generated by some application. I can only ingest
> >> these files when a specific set exists. For example:
> >>
> >> file_123_456_ab.ex1
> >> file_123_456_cd.ex1
> >> file_123_456_ef.ex1
> >> file_123_456_gh.ex1
> >> file_123_456.ex2
> >>
> >> Only when a set like that exists should I pick them up into the Flow.
> The
> >> parts I am looking for to "group" would "ab.ex1", "cd.ex1", "ef.ex1",
> >> "gh.ex1", ".ex2".
> >>
> >> I tried to do this with some expression, but couldn't work it out.
> >>
> >> What would be the best way to achieve this?
> >>
> >> Many thanks!
>

Re: Only get file when a set exists.

[Koji Kawamura]

Hi Martin,

Alternative approach is using Wait/Notify processors.
I have developed similar flow using those before, and it will work
with your case I believe.
A NiFi flow template is available here.
https://gist.github.com/ijokarumawak/06b3b071eeb4d10d8a27507981422edd

Hope this helps,
Koji


On Sun, May 27, 2018 at 11:48 PM, Andrew Grande  wrote:
> Martijn,
>
> Here's an idea you could explore. Have the ListFile processor work as usual
> and create a custom component (start with a scripting one to prototype)
> grouping the filenames as needed. I don't know of the number of files in a
> set is different every time, so trying to be more robust.
>
> Once you group and count the set, you can transfer the names to the success
> relationship. Ignore otherwise and wait until the set is full.
>
> Andrew
>
>
> On Sun, May 27, 2018, 7:29 AM Martijn Dekkers 
> wrote:
>>
>> Hello all,
>>
>> I am trying to work out an issue with little success.
>>
>> I need to ingest files generated by some application. I can only ingest
>> these files when a specific set exists. For example:
>>
>> file_123_456_ab.ex1
>> file_123_456_cd.ex1
>> file_123_456_ef.ex1
>> file_123_456_gh.ex1
>> file_123_456.ex2
>>
>> Only when a set like that exists should I pick them up into the Flow. The
>> parts I am looking for to "group" would "ab.ex1", "cd.ex1", "ef.ex1",
>> "gh.ex1", ".ex2".
>>
>> I tried to do this with some expression, but couldn't work it out.
>>
>> What would be the best way to achieve this?
>>
>> Many thanks!

Re: Only get file when a set exists.

[Andrew Grande]

Martijn,

Here's an idea you could explore. Have the ListFile processor work as usual
and create a custom component (start with a scripting one to prototype)
grouping the filenames as needed. I don't know of the number of files in a
set is different every time, so trying to be more robust.

Once you group and count the set, you can transfer the names to the success
relationship. Ignore otherwise and wait until the set is full.

Andrew

On Sun, May 27, 2018, 7:29 AM Martijn Dekkers 
wrote:

> Hello all,
>
> I am trying to work out an issue with little success.
>
> I need to ingest files generated by some application. I can only ingest
> these files when a specific set exists. For example:
>
> file_123_456_ab.ex1
> file_123_456_cd.ex1
> file_123_456_ef.ex1
> file_123_456_gh.ex1
> file_123_456.ex2
>
> Only when a set like that exists should I pick them up into the Flow. The
> parts I am looking for to "group" would "ab.ex1", "cd.ex1", "ef.ex1",
> "gh.ex1", ".ex2".
>
> I tried to do this with some expression, but couldn't work it out.
>
> What would be the best way to achieve this?
>
> Many thanks!
>

Only get file when a set exists.

[Martijn Dekkers]

Hello all,

I am trying to work out an issue with little success.

I need to ingest files generated by some application. I can only ingest
these files when a specific set exists. For example:

file_123_456_ab.ex1
file_123_456_cd.ex1
file_123_456_ef.ex1
file_123_456_gh.ex1
file_123_456.ex2

Only when a set like that exists should I pick them up into the Flow. The
parts I am looking for to "group" would "ab.ex1", "cd.ex1", "ef.ex1",
"gh.ex1", ".ex2".

I tried to do this with some expression, but couldn't work it out.

What would be the best way to achieve this?

Many thanks!