Hi Jakub,
I would just double check that the endpoints after the consumer from
SMB are asynchronous. I think the throughput you got when using
multiple consumers could be caused by the fact that you now process
the rest of the route in parallel.
Perhaps using a simple thread EIP[1] might help, something like:
from("smb:\\...").threads(10).to("...")
There's also a jcifs component over at camel-extras[2] that you might
want to check out.
zoran
[1]
https://github.com/apache/camel/blob/master/core/camel-core/src/main/docs/eips/threads-eip.adoc
[2]
https://github.com/camel-extra/camel-extra/tree/master/components/camel-jcifs/
On Thu, Apr 18, 2019 at 6:44 PM Jakub Bekier <jakub.beki...@gmail.com> wrote:
>
> I am trying to download a large amount of files from an smb:// endpoint and
> I would like the file consumption to be as fast as possible - meaning I
> would like to download and process as many files as possible, for example
> polling for 2000 files instead of whatever the default polling value in
> Camel is. To achieve this, I was looking for a way to increase the incoming
> batch size or add multi-threaded/parallel consumption on a single folder.
>
> So far I have tried the following, but I noticed the download rate stayed
> the same as with one thread:
>
> from("smb:\\someFolder?&maxMessagesPerPoll=1000")...
>
> from("smb:\\someFolder?&maxMessagesPerPoll=-1")...
>
> from("smb:\\someFolder").threads(20)...
>
> Even with the threads option, despite seeing that the thread IDs vary, the
> download rate was the same. I suppose Camel simply assings a thread pool so
> that a different thread will pick up the incoming file each time, but the
> polling amount is the same.
>
> I also looked up the concurrentConsumers flag, but from what I gather it is
> not supported for files, rather for ActiveMQ.
>
> The only efficiency fix I managed to apply was consuming from multiple
> smb:// endpoints. The throughput rate increased rapidly when I did that. I
> am wondering if there is a way to achieve the same result as with multiple
> folders but using a single folder only?
--
Zoran Regvart
