[
https://issues.apache.org/activemq/browse/CAMEL-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=60412#action_60412
]
Claus Ibsen commented on CAMEL-2899:
------------------------------------
We could either
a)
Use lightweight record structure for the 160K list of files instead of turning
them into Exchange objects which carries extra load
b)
Alter so the maxMessagesPerPoll will be effective earlier. Currently it loads
the entire list, then you can filter, sort etc. And after all this it will cut
down the list.
If we apply the maxMessagesPerPoll earlier we can't really be able to sort the
entire list etc. And the filter could also cut out some from the max list.
Or in other words we will have to re-add files until we hit that limit if there
are more files.
In your case I don't think you care about sorting/filtering etc. You just want
to poll X number of files per go to avoid excessive memory overhead. So I think
we could add an option in the uri for solution b.
Anyone got a good name for such an option?
Any thoughts?
> out of heap space if remote FTP site has too many files to pick up
> ------------------------------------------------------------------
>
> Key: CAMEL-2899
> URL: https://issues.apache.org/activemq/browse/CAMEL-2899
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-ftp
> Affects Versions: 2.3.0
> Reporter: Karl Palsson
> Fix For: 2.4.0
>
>
> 2010-07-02 11:38:07,439 FATAL
> [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 10] -
> <Consumer Consumer[my_ftp_URI_here caused by: Java heap space>
> java.lang.OutOfMemoryError: Java heap space
> My remote FTP server has ~60k 100 byte files, and the camel endpoint consumer
> falls over and doesn't start again. I can use JMX to stop/start the
> consumer, (it still has status "started") and it will log in to the remote
> server again, but then fall over with the out of heap space.
> I can work around this by increasing the heap, or by moving some of the files
> aside, but I don't think camel should care how many files there are, or at
> least, I think it should deal with it more gracefully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
