Yaismel, My best guess is that to best accomplish this it would require custom coding to handling the merging logic. As described thus far I believe I understand the use case but I still have a lot of questions about frequency of arrival for each dataset, how to handle misses (where one side doesn't have a reference for a row in the other side), what to do with out of order arrival, etc..
Thanks joe On Fri, Nov 13, 2015 at 6:18 PM, Yaismel Miranda Pons <[email protected]> wrote: > Hi Joe, thanks for taking the time to answer. > > This is the scenario I'm trying to accomplish with nifi: I want to create a > simple dataflow for automating the process of ingesting CSV data found in > some datasets. The dataset could come from either an Http endpoint or just > be csv files in a directory and it has to be ingested every month. > I was able to implement this scenario with nifi when the data is just a > single CSV file, but I have some cases where can be 2 or more CSV files > related. I would like to know if there is an effective way in nifi to > combine these CSV files into a single one, based on specific criterias. Each > file contains around 15 million records. > > Thanks > Yaismel > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/Merging-csv-files-based-on-criterias-tp4711p4873.html > Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
