Hi, The camel-csv component converts the file into a List,
Therefore your 20 MB file is taking up 40 MB as a String. The List will take up a little more than another 40 MB (so you will need 80 MB during the conversion). This lists lives throughout your split process... What you could do to reduce memory footprint: 1. Avoid the string conversion. If you need to read your streams multiple times, a steam cache might help you. 2. Use streaming for the csv conversion (set lazyLoad="true"). If that doesn't help, you should analyze the heap dump for the out of memory error to see where the memory is consumed (if you are using eclipse there is a Memory Analyzer plugin available). Best regards Stephan -----Original Message----- From: Michele [mailto:michele.mazzi...@finconsgroup.com] Sent: Mittwoch, 13. April 2016 14:34 To: users@camel.apache.org Subject: RE: Best Strategy to process a large number of rows in File Hi, In this case file size is 20.424KB and it contains about 40000 lines. I used the convertTo to avoid Rollback on rename file (GenericFileOperationFailedException - only Windows). The File is a cvs and so I defined CSV Data format like this (Transforms each line in a map key-value) <csv id="ReaderDataFormat" delimiter="{{in.file.delimiter}}" skipHeaderRecord="true" ignoreEmptyLines="true" useMaps="true"> <header>Key</header> <header>Date</header> <header>Time</header> <header>User1</header> <header>User2</header> <header>User3</header> <header>Status</header> <header>CertificateNumber</header> <header>CA_KEY_INDEX</header> <header>INJ_CARD_SN</header> <header>TERM_PN</header> <header>TERM_SN</header> <header>Certificate</header> </csv> I used CSV DF to handle CSV payloads. Active MQ configuration is based on PooledConnection and to limit the amount of memory (Per-Destination) I set: <policyEntry queue="IF_CBIKIT" producerFlowControl="true" optimizedDispatch="true" reduceMemoryFootprint="true" memoryLimit="10Mb" > </policyEntry> But, after about 1h the system goes in OutOfMemory. Thanks you so much for your help. Best Regards Michele -- View this message in context: http://camel.465427.n5.nabble.com/Best-Strategy-to-process-a-large-number-of-rows-in-File-tp5779856p5781003.html Sent from the Camel - Users mailing list archive at Nabble.com.
