Thanks Claus - unfortunately, there is a requirement to support existing client workflows, which are usually not able to upload a "done" file without additional development work on their end, and it seems that ProFTPD is not able to do this automatically.
Are you aware of any way to prevent the file consumer blocking while waiting for the "changed" read lock or would another approach such as one Camel process per user directory be the recommended approach in this case? Thanks, Tom Tom Duncalf / Software Developer tomduncalf.com <http://www.tomduncalf.com> / @tomduncalf <http://twitter.com/tomduncalf> On 2 June 2015 at 08:12, Claus Ibsen-2 [via Camel] < ml-node+s465427n5767770...@n5.nabble.com> wrote: > Hi > > If possible then using done file names is IMHO a better strategy. > Though that would require the other party to do this when it uploads > to the FTP server. > > > > On Mon, Jun 1, 2015 at 6:10 PM, Tom Duncalf <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=5767770&i=0>> wrote: > > > Hi, > > > > I am building a Camel route to consume files uploaded by FTP and then > upload > > them elsewhere. It would seem that "changed" is the most suitable > readLock > > strategy for this, and I would like to set a fairly large > > readLockCheckInterval and readLockTimeout (e.g. 20s and 40s) to help > prevent > > incomplete files from users with slow/intermittent connections being > picked > > up by the route prematurely. > > > > What I would like to achieve is to make it so that the File consumer is > not > > blocked from picking up more files for the duration of the > readLockTimeout - > > that is, currently, if file1 is uploaded at t=0 and file2 uploaded at > t=5, > > then the consumer picks up file1 at t=0 and is then blocked until at > least > > t=20 (assuming a 20s check interval), so file2 will not be picked up > until > > at least t=20 (and processed until at least t=40). Instead, I would like > > there to be (for example) more than one consumer thread, so that when > the > > first consumer thread picks up file1 and is waiting until t=20, another > > consumer thread can still pick up file2 at t=5 and wait until t=25. > > > > None of the concurrency options available seem to cover this scenario - > I > > understand that I can decouple the file being consumed and the > subsequent > > processing using, for example, a SEDA queue, or I can split the > processing > > into multiple threads after a file has been processed by the File > consumer > > using the Threads DSL, but I can't see how to make the consumer itself > > consume files and therefore create messages in a multi-threaded manner. > > > > One workaround may be to spawn a new process for each FTP user and have > it > > consume from their home directory, but I would prefer to avoid this > > additional complexity and it would not gracefully handle a situation > where > > one user uploads a large volume of files. One other option I can think > of is > > to start multiple instances of my route somehow (to create multiple > > consumers), and then synchronise them with a shared repository for the > > inProgressRepository, but I'm not sure if that would actually work. > > > > Any input would be great - even if it is basic as I am new to Camel! > > > > Thanks, > > Tom > > > > > > > > > > -- > > View this message in context: > http://camel.465427.n5.nabble.com/Parallel-processing-multiple-file-consumer-threads-with-readLock-changed-and-long-timeout-tp5767753.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > > > > -- > Claus Ibsen > ----------------- > Red Hat, Inc. > Email: [hidden email] > <http:///user/SendEmail.jtp?type=node&node=5767770&i=1> > Twitter: davsclaus > Blog: http://davsclaus.com > Author of Camel in Action: http://www.manning.com/ibsen > hawtio: http://hawt.io/ > fabric8: http://fabric8.io/ > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://camel.465427.n5.nabble.com/Parallel-processing-multiple-file-consumer-threads-with-readLock-changed-and-long-timeout-tp5767753p5767770.html > To unsubscribe from Parallel processing/multiple file consumer threads > with readLock=changed and long timeout?, click here > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5767753&code=dG9tQHRvbWR1bmNhbGYuY29tfDU3Njc3NTN8MTAzMzMyNTM2> > . > NAML > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://camel.465427.n5.nabble.com/Parallel-processing-multiple-file-consumer-threads-with-readLock-changed-and-long-timeout-tp5767753p5767781.html Sent from the Camel - Users mailing list archive at Nabble.com.
