Hi, I am building a Camel route to consume files uploaded by FTP and then upload them elsewhere. It would seem that "changed" is the most suitable readLock strategy for this, and I would like to set a fairly large readLockCheckInterval and readLockTimeout (e.g. 20s and 40s) to help prevent incomplete files from users with slow/intermittent connections being picked up by the route prematurely.
What I would like to achieve is to make it so that the File consumer is not blocked from picking up more files for the duration of the readLockTimeout - that is, currently, if file1 is uploaded at t=0 and file2 uploaded at t=5, then the consumer picks up file1 at t=0 and is then blocked until at least t=20 (assuming a 20s check interval), so file2 will not be picked up until at least t=20 (and processed until at least t=40). Instead, I would like there to be (for example) more than one consumer thread, so that when the first consumer thread picks up file1 and is waiting until t=20, another consumer thread can still pick up file2 at t=5 and wait until t=25. None of the concurrency options available seem to cover this scenario - I understand that I can decouple the file being consumed and the subsequent processing using, for example, a SEDA queue, or I can split the processing into multiple threads after a file has been processed by the File consumer using the Threads DSL, but I can't see how to make the consumer itself consume files and therefore create messages in a multi-threaded manner. One workaround may be to spawn a new process for each FTP user and have it consume from their home directory, but I would prefer to avoid this additional complexity and it would not gracefully handle a situation where one user uploads a large volume of files. One other option I can think of is to start multiple instances of my route somehow (to create multiple consumers), and then synchronise them with a shared repository for the inProgressRepository, but I'm not sure if that would actually work. Any input would be great - even if it is basic as I am new to Camel! Thanks, Tom -- View this message in context: http://camel.465427.n5.nabble.com/Parallel-processing-multiple-file-consumer-threads-with-readLock-changed-and-long-timeout-tp5767753.html Sent from the Camel - Users mailing list archive at Nabble.com.
