Re: How to process files sequentially?
Hi Prabhu, Here is another example that counts number of files within a dir using ExecuteScript with Groovy. Although I don't have tested with Windows env, it should work. https://gist.github.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0#file-process-in-order-and-defrag-with-groovy-xml Thanks, Koji On Fri, May 19, 2017 at 8:13 PM, prabhu Mahendran wrote: > Sorry Koji, > > I have used NiFi in Windows So i can't able to use ExecuteStreamCommand for > it. > > It shows error in "System cannot find path Specified" > > Is any other way for doing it in windows? > > On Fri, May 19, 2017 at 12:50 PM, Koji Kawamura > wrote: >> >> Hi Prabhu, >> >> I just used MergeContent to confirm test result. In your case, I >> thought the goal is sending queries to SQL Server in order so I think >> you don't have to use MergeContent. >> >> Having said that, I came up with an idea. This is kind of a hack but >> using ExecuteStreamingCommand to analyze number of files in a dir and >> use MergeContent 'Defragment' mode, I was able to merge content >> without using static minimum number of files. >> >> Here is another template for that example: >> >> https://gist.githubusercontent.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0/raw/967a051177878a98f5ddb57653478c6091a7b23c/process-files-in-order-and-defrag.xml >> >> Thanks, >> Koji >> >> On Fri, May 19, 2017 at 6:11 PM, prabhu Mahendran >> wrote: >> > Hi Koji, >> > >> > Thanks for your mail. >> > >> > In your template i have one query regarding if number of files taken by >> > get >> > file is unknown then MergeContent Processor could not work right? >> > because >> > you have specify maximum number of bins to be 5. >> > >> > But in my case i am having dynamic number of file counts.In that case >> > merge >> > content will failed to merge. >> > >> > Please stop me if i'm understand anything wrong. >> > >> > How to give dynamic number of entries/bin for MergeContent due to >> > currently >> > there is no expression language supported? >> > >> > On Fri, May 19, 2017 at 10:34 AM, Koji Kawamura >> > wrote: >> >> >> >> Hi Prabhu, >> >> >> >> I think you can use EnforceOrder processor which is available since >> >> 1.2.0, without Wait/Notify processor. >> >> >> >> Here is a sample flow I tested how it can be used for use-cases like >> >> yours: >> >> https://gist.github.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0 >> >> >> >> Thanks, >> >> Koji >> >> >> >> On Fri, May 19, 2017 at 1:52 PM, prabhu Mahendran >> >> wrote: >> >> > I have approximately 1000 files in local drive.I need to move that >> >> > files >> >> > into SQL Server accordingly one after another. >> >> > >> >> > Since local drive having files like file1.csv,file2.csv,..upto >> >> > file1000.csv.I am sure that number of files in local drive may change >> >> > dynamically. >> >> > >> >> > I can able to created template for move that files into SQL >> >> > Server.But i >> >> > have to process the file2 when file 1 has been completely moved into >> >> > SQL >> >> > Server. >> >> > >> >> > Is this possible in NiFi without using Wait\Notify processor? >> >> > >> >> > can anyone please guide me to solve this? >> > >> > > >
Re: How to process files sequentially?
Sorry Koji, I have used NiFi in Windows So i can't able to use ExecuteStreamCommand for it. It shows error in "System cannot find path Specified" Is any other way for doing it in windows? On Fri, May 19, 2017 at 12:50 PM, Koji Kawamura wrote: > Hi Prabhu, > > I just used MergeContent to confirm test result. In your case, I > thought the goal is sending queries to SQL Server in order so I think > you don't have to use MergeContent. > > Having said that, I came up with an idea. This is kind of a hack but > using ExecuteStreamingCommand to analyze number of files in a dir and > use MergeContent 'Defragment' mode, I was able to merge content > without using static minimum number of files. > > Here is another template for that example: > https://gist.githubusercontent.com/ijokarumawak/ > 7e6158460cfcb0b5911acefbb455edf0/raw/967a051177878a98f5ddb57653478c > 6091a7b23c/process-files-in-order-and-defrag.xml > > Thanks, > Koji > > On Fri, May 19, 2017 at 6:11 PM, prabhu Mahendran > wrote: > > Hi Koji, > > > > Thanks for your mail. > > > > In your template i have one query regarding if number of files taken by > get > > file is unknown then MergeContent Processor could not work right? because > > you have specify maximum number of bins to be 5. > > > > But in my case i am having dynamic number of file counts.In that case > merge > > content will failed to merge. > > > > Please stop me if i'm understand anything wrong. > > > > How to give dynamic number of entries/bin for MergeContent due to > currently > > there is no expression language supported? > > > > On Fri, May 19, 2017 at 10:34 AM, Koji Kawamura > > wrote: > >> > >> Hi Prabhu, > >> > >> I think you can use EnforceOrder processor which is available since > >> 1.2.0, without Wait/Notify processor. > >> > >> Here is a sample flow I tested how it can be used for use-cases like > >> yours: > >> https://gist.github.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0 > >> > >> Thanks, > >> Koji > >> > >> On Fri, May 19, 2017 at 1:52 PM, prabhu Mahendran > >> wrote: > >> > I have approximately 1000 files in local drive.I need to move that > files > >> > into SQL Server accordingly one after another. > >> > > >> > Since local drive having files like file1.csv,file2.csv,..upto > >> > file1000.csv.I am sure that number of files in local drive may change > >> > dynamically. > >> > > >> > I can able to created template for move that files into SQL > Server.But i > >> > have to process the file2 when file 1 has been completely moved into > SQL > >> > Server. > >> > > >> > Is this possible in NiFi without using Wait\Notify processor? > >> > > >> > can anyone please guide me to solve this? > > > > >
Re: How to process files sequentially?
Hi Prabhu, I just used MergeContent to confirm test result. In your case, I thought the goal is sending queries to SQL Server in order so I think you don't have to use MergeContent. Having said that, I came up with an idea. This is kind of a hack but using ExecuteStreamingCommand to analyze number of files in a dir and use MergeContent 'Defragment' mode, I was able to merge content without using static minimum number of files. Here is another template for that example: https://gist.githubusercontent.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0/raw/967a051177878a98f5ddb57653478c6091a7b23c/process-files-in-order-and-defrag.xml Thanks, Koji On Fri, May 19, 2017 at 6:11 PM, prabhu Mahendran wrote: > Hi Koji, > > Thanks for your mail. > > In your template i have one query regarding if number of files taken by get > file is unknown then MergeContent Processor could not work right? because > you have specify maximum number of bins to be 5. > > But in my case i am having dynamic number of file counts.In that case merge > content will failed to merge. > > Please stop me if i'm understand anything wrong. > > How to give dynamic number of entries/bin for MergeContent due to currently > there is no expression language supported? > > On Fri, May 19, 2017 at 10:34 AM, Koji Kawamura > wrote: >> >> Hi Prabhu, >> >> I think you can use EnforceOrder processor which is available since >> 1.2.0, without Wait/Notify processor. >> >> Here is a sample flow I tested how it can be used for use-cases like >> yours: >> https://gist.github.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0 >> >> Thanks, >> Koji >> >> On Fri, May 19, 2017 at 1:52 PM, prabhu Mahendran >> wrote: >> > I have approximately 1000 files in local drive.I need to move that files >> > into SQL Server accordingly one after another. >> > >> > Since local drive having files like file1.csv,file2.csv,..upto >> > file1000.csv.I am sure that number of files in local drive may change >> > dynamically. >> > >> > I can able to created template for move that files into SQL Server.But i >> > have to process the file2 when file 1 has been completely moved into SQL >> > Server. >> > >> > Is this possible in NiFi without using Wait\Notify processor? >> > >> > can anyone please guide me to solve this? > >
Re: How to process files sequentially?
Hi Koji, Thanks for your mail. In your template i have one query regarding if number of files taken by get file is unknown then MergeContent Processor could not work right? because you have specify maximum number of bins to be 5. But in my case i am having dynamic number of file counts.In that case merge content will failed to merge. Please stop me if i'm understand anything wrong. How to give dynamic number of entries/bin for MergeContent due to currently there is no expression language supported? On Fri, May 19, 2017 at 10:34 AM, Koji Kawamura wrote: > Hi Prabhu, > > I think you can use EnforceOrder processor which is available since > 1.2.0, without Wait/Notify processor. > > Here is a sample flow I tested how it can be used for use-cases like yours: > https://gist.github.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0 > > Thanks, > Koji > > On Fri, May 19, 2017 at 1:52 PM, prabhu Mahendran > wrote: > > I have approximately 1000 files in local drive.I need to move that files > > into SQL Server accordingly one after another. > > > > Since local drive having files like file1.csv,file2.csv,..upto > > file1000.csv.I am sure that number of files in local drive may change > > dynamically. > > > > I can able to created template for move that files into SQL Server.But i > > have to process the file2 when file 1 has been completely moved into SQL > > Server. > > > > Is this possible in NiFi without using Wait\Notify processor? > > > > can anyone please guide me to solve this? >
Re: How to process files sequentially?
Hi Prabhu, I think you can use EnforceOrder processor which is available since 1.2.0, without Wait/Notify processor. Here is a sample flow I tested how it can be used for use-cases like yours: https://gist.github.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0 Thanks, Koji On Fri, May 19, 2017 at 1:52 PM, prabhu Mahendran wrote: > I have approximately 1000 files in local drive.I need to move that files > into SQL Server accordingly one after another. > > Since local drive having files like file1.csv,file2.csv,..upto > file1000.csv.I am sure that number of files in local drive may change > dynamically. > > I can able to created template for move that files into SQL Server.But i > have to process the file2 when file 1 has been completely moved into SQL > Server. > > Is this possible in NiFi without using Wait\Notify processor? > > can anyone please guide me to solve this?
