Re: file component polling files
My general goal is to consume fast large quantity of files but while i was experimenting with file component in order to understood how polling works i tried this case. To poll every 30 secs one file but it seems to pick more than one file in those 30 secs and i was wondering if am doing something wrong. Yes i am unmarshaling at a point between file consumer and queue producer uri's. How is that affect the speed? What do you mean by getting off the file reader? Thanks -- View this message in context: http://camel.465427.n5.nabble.com/file-component-polling-files-tp5788570p5788846.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: file component polling files
I'm not quite sure what you are attempting to do here. You poll every 30 seconds and seem to only want to pick up 1 file but then say you want it to pick up a "large amount of files every poll". Are you trying to pick up just 1 file every 30 seconds or are you trying to read all the files when the poll fires? If you want to pick up everything it will do that normally. Are you unmarshaling these files at some point? Speeding things up can be accomplished via thread pools and getting them off the file reader can be accomplished by something as simple as a SEDA queue with a thread pool. But you also have to be aware of how much data you have in each file and make sure it doesn't exceed your memory settings. That usually means tremendously big files because folks usually overestimate how much data a file represents. "thinking that the file component will pick up a single file every 30 secs but it seems to consume the files like normal. Any idea why is it doing that or am i doing something wrong? My idea is to use the maxMessagesPerPoll to make the file component pick up a large amount of files every poll. Is that going to speed up the consumption?" On Fri, Oct 14, 2016 at 3:48 AM, redpower1989 wrote: > I tried maxMessagesPerPoll = 1 and delay 30 secs and it wasn't picking 1 > file > every 30 secs and i am wondering if i am setting something wrong on the > consumer. > > By the way i found a way to speed up the file processing and it is the use > of threads. Be careful that you need to find a balance for the poolsize > value. > http://camel.apache.org/async.html > > > > -- > View this message in context: http://camel.465427.n5.nabble. > com/file-component-polling-files-tp5788570p5788775.html > Sent from the Camel - Users mailing list archive at Nabble.com. >
Re: file component polling files
I tried maxMessagesPerPoll = 1 and delay 30 secs and it wasn't picking 1 file every 30 secs and i am wondering if i am setting something wrong on the consumer. By the way i found a way to speed up the file processing and it is the use of threads. Be careful that you need to find a balance for the poolsize value. http://camel.apache.org/async.html -- View this message in context: http://camel.465427.n5.nabble.com/file-component-polling-files-tp5788570p5788775.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: file component polling files
The delay will control the frequency at which it polls the directory. To limit the number of files each time, check out maxMessagesPerPoll and eagermaxmessagesperPoll http://camel.apache.org/file2.html -- View this message in context: http://camel.465427.n5.nabble.com/file-component-polling-files-tp5788570p5788599.html Sent from the Camel - Users mailing list archive at Nabble.com.
file component polling files
Hello, I have a route which consumes from a file component and the message end up to a activemq queue . I was testing my application to consume a large amount of files. The consumption's is slow and while i was trying to solve the problem i use the follow uri: file://inbox?move=backup/${date:now:MMdd}/${file:name}&delay=3&maxMessagesPerPoll=1" thinking that the file component will pick up a single file every 30 secs but it seems to consume the files like normal. Any idea why is it doing that or am i doing something wrong? My idea is to use the maxMessagesPerPoll to make the file component pick up a large amount of files every poll. Is that going to speed up the consumption? Thanks -- View this message in context: http://camel.465427.n5.nabble.com/file-component-polling-files-tp5788570.html Sent from the Camel - Users mailing list archive at Nabble.com.