My team designed a NiFi flow to handle CSV files of size around 15GB. But later we realised that files can be upto 500 GB. I set the queue size limit to 25GB. This is a one time data load to S3. I'm converting each CSV file to parquet in NiFi using a convert record processor. What happens in these situations? Can NiFi be able to handle this kind of scenario?
FYI, my NiFi has 40 gigs of memory and 2TB of storage. Regards Satish
