Hey Valencia, I don't believe that PutFile allows you to append to a file, because doing so is rife with problems if you encounter any kind of error (IOException, for example) or if NiFi restarts in between. Instead, you should take a look at MergeContent. You can set the "Merge Strategy" to "Defragment" in order to re-assemble the FlowFiles that were split apart via SplitText.
That being said, splitting the data apart, then using ExtractText, and merging back together can be quite expensive. If your data is JSON or CSV, then you should probably look into using the Record-Based Processors (PublishKafkaRecord, QueryRecord/PartitionRecord). This allows you to avoid ever splitting the data apart to begin with and as a result can perform dramatically better. Thanks -Mark On Feb 21, 2018, at 7:27 AM, Valencia Serrao <[email protected]<mailto:[email protected]>> wrote: Hi Mark, Yes! I could get all the required entries with the respective matched and unmatched segregated in different folders. Thanks a lot, Mark!! My next plan is to check and see how to append all the flowfiles with matched entries in one file. Regards, Valencia <graycol.gif>Valencia Serrao---02/16/2018 06:00:47 PM---Hi Mark, Thanks for looking into this. I am trying to put in the components you have suggested. I'll From: Valencia Serrao/Austin/Contr/IBM To: [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Date: 02/16/2018 06:00 PM Subject: Re: [Data Flow] File content not read completely ________________________________ Hi Mark, Thanks for looking into this. I am trying to put in the components you have suggested. I'll update. Regards, Valencia <graycol.gif>Mark Payne ---02/15/2018 07:09:32 PM---Valencia, The SplitText processor does not change the ‘filename’ attribute of the FlowFile. So you w From: Mark Payne <[email protected]<mailto:[email protected]>> To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: 02/15/2018 07:09 PM Subject: Re: [Data Flow] File content not read completely ________________________________ Valencia, The SplitText processor does not change the ‘filename’ attribute of the FlowFile. So you will end up with multiple FlowFiles having the same name. PutFile may well be overwriting the same file many times - or failing to to write the files do to filename conflicts. You can resolve this, if it’s your problem, by adding an UpdateAttribute to your flow just before PutFile and changing the filename to something unique like ${UUID()} or ${filename}.${nextInt()} Hope this helps! -Mark Sent from my iPhone On Feb 15, 2018, at 4:59 AM, Valencia Serrao <[email protected]<mailto:[email protected]>> wrote: Hi All, I've started hands-on with Nifi. Basic flows I was able to do without any issues. But currently I've tried adding more steps to the flow. Flow intent: Get a local file, split the text on new line, extract text based on regex, Put matched/unmatched data on respective kafka topics and finally write the kafka contents on the local targets set in PutFile. Current Flow steps: GetFile, SplitText, ExtractText, PutKafka -( 2 of them,one for matched and unmatched), and 2 PutFiles components. The issue I'm facing is that - after the flow execution I see only one entry in each of the 2 PutFile targets and rest of the content is not written to them even if the criteria is matched. I feel its not looping through the whole file or something like that. But I had read that Nifi flow is executed for all contents in source files. Maybe I've missed some config somewhere. It would be really helpful if anyone could help on this issue. Regards, Valencia
