Hi If possible then using done file names is IMHO a better strategy. Though that would require the other party to do this when it uploads to the FTP server.
On Mon, Jun 1, 2015 at 6:10 PM, Tom Duncalf <t...@tomduncalf.com> wrote: > Hi, > > I am building a Camel route to consume files uploaded by FTP and then upload > them elsewhere. It would seem that "changed" is the most suitable readLock > strategy for this, and I would like to set a fairly large > readLockCheckInterval and readLockTimeout (e.g. 20s and 40s) to help prevent > incomplete files from users with slow/intermittent connections being picked > up by the route prematurely. > > What I would like to achieve is to make it so that the File consumer is not > blocked from picking up more files for the duration of the readLockTimeout - > that is, currently, if file1 is uploaded at t=0 and file2 uploaded at t=5, > then the consumer picks up file1 at t=0 and is then blocked until at least > t=20 (assuming a 20s check interval), so file2 will not be picked up until > at least t=20 (and processed until at least t=40). Instead, I would like > there to be (for example) more than one consumer thread, so that when the > first consumer thread picks up file1 and is waiting until t=20, another > consumer thread can still pick up file2 at t=5 and wait until t=25. > > None of the concurrency options available seem to cover this scenario - I > understand that I can decouple the file being consumed and the subsequent > processing using, for example, a SEDA queue, or I can split the processing > into multiple threads after a file has been processed by the File consumer > using the Threads DSL, but I can't see how to make the consumer itself > consume files and therefore create messages in a multi-threaded manner. > > One workaround may be to spawn a new process for each FTP user and have it > consume from their home directory, but I would prefer to avoid this > additional complexity and it would not gracefully handle a situation where > one user uploads a large volume of files. One other option I can think of is > to start multiple instances of my route somehow (to create multiple > consumers), and then synchronise them with a shared repository for the > inProgressRepository, but I'm not sure if that would actually work. > > Any input would be great - even if it is basic as I am new to Camel! > > Thanks, > Tom > > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Parallel-processing-multiple-file-consumer-threads-with-readLock-changed-and-long-timeout-tp5767753.html > Sent from the Camel - Users mailing list archive at Nabble.com. -- Claus Ibsen ----------------- Red Hat, Inc. Email: cib...@redhat.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen hawtio: http://hawt.io/ fabric8: http://fabric8.io/
