Yes Dave, ours is streaming data . We do ETL on the live data before loading into target. We parse the json data, since the volume is high and don’t know how many events we get, I am looking to load the data by merging instead of doing individual insert. This will avoid more threads on database and does the bulk update.
Merge content works when I stop the prior professor so I was looking to implement wait strategy. Thanks, Asmath Sent from my iPhone > On Oct 26, 2020, at 10:33 PM, David Early <david.ea...@grokstream.com> wrote: > > > Can you expand on your use case? Do you need to wait for a specific amount > of time after receiving the first flow file? Why would a scheduled run every > 2 min not work? Is the issue that you need all related flowfiles merged > together? > > Dave > >> On Mon, Oct 26, 2020, 9:25 PM KhajaAsmath Mohammed <mdkhajaasm...@gmail.com> >> wrote: >> Thanks David, I have the same issue. Was never able to get it work with >> merge record directly . Since my flow has stream of data, I can’t implement >> cron in between the job. I might have to see if groovy script has any option >> to wait for specified amount of time before merge operation >> >> Sent from my iPhone >> >>>> On Oct 26, 2020, at 9:22 PM, David Early <david.ea...@grokstream.com> >>>> wrote: >>>> >>> >>> I have a case where I have a single stream of data items that need merged >>> into a single file. >>> >>> I do this by setting the number of bins in merge to 1 and using the cron >>> scheduler to run the merge every 15 min. I never got the bin age to work >>> the way I wanted. >>> >>> I set the number of flowfiles to include in the output to a value much >>> greater than the expected number of ff that will appear in the input queue. >>> >>> This creates a single output flowfile on a timed schedule. >>> >>> In my case, I follow this with a record based query that deduplicates the >>> data. >>> >>> Dave >>> >>>> On Sun, Oct 25, 2020, 12:00 PM KhajaAsmath Mohammed >>>> <mdkhajaasm...@gmail.com> wrote: >>>> Hi, >>>> >>>> I am looking for a use case to wait for processor/flow files for 2 minutes >>>> and later merge them using a merge processor. Is there a processor or >>>> script to achieve this? >>>> >>>> Thanks, >>>> Asmath
