Hi Souciance,
Thanks for reply.
Yes the behaviour is same . I have tried on other SFTP's as well. The issue
which we are facing is on client environment and with the below tests we are
able to reproduce this issue in local environment as well. Which proves it is
not related to SFTP server.
Also there was 1 file which took around 3 min when target directory contain
40,000 file. I had put around 22 files , So the total time it took to transfer
22 files is around 22 * 3 min = 66 min. The behaviour is exponential the
number of files you increase on target directory. The camel route will take
more time to transfer the file to target directory.
I have enabled camel logger in DEBUG . I try to debug , what I observed is it
is reading that is taking time. PFB logger.
INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Sending IOP response file : SX30RPMAJAR1702483756.txt to ICOMS FTP directory
DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.
190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&
tempPrefix=Q&username=sigmauser] Exchange[SX30RPMAJAR1702483756.txt]
DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.GenericFileConverter> Read file
/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJA
1702483756.txt(no charset)
Thanks
Ayush
-----Original Message-----
From: souciance [mailto:souciance.eqdam.ras...@gmail.com]
Sent: Wednesday, April 5, 2017 12:29 PM
To: users@camel.apache.org
Subject: Re: Performance issue in Camel SFTP component
Well, have you transferred the same amount of files with some other FTP client
just to compare how much "slower" the ftp component is? 40,000 files in 3
minutes doesn't sounds ok.
On Wed, Apr 5, 2017 at 7:53 AM, Ayush Dixit [via Camel] <
ml-node+s465427n5796945...@n5.nabble.com> wrote:
> Hi,
>
> Can someone help me regarding the performance issue I am facing in
> camel SFTP component ? I have gone through the documentation but did
> not find anything that can improve performance.
>
> Thanks
> Ayush
>
> -----Original Message-----
> From: Ayush Dixit
> Sent: Tuesday, April 4, 2017 3:40 PM
> To: [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5796945&i=0>
> Cc: '[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5796945&i=1>' <[hidden
> email] <http:///user/SendEmail.jtp?type=node&node=5796945&i=2>>
> Subject: RE: Performance issue in Camel SFTP component
>
>
> Thanks Claus , I've tried stepwise=false in camel producer . No such
> luck
>
> Also, I have tried download=false and useList=false. Still the
> performance is same. I am referring the below link for camel
> documentation
>
> http://camel.apache.org/ftp2.html
>
> Is there is anything else you think I can try?
>
> Thanks
> Ayush
>
>
> -----Original Message-----
> From: Claus Ibsen [mailto:[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5796945&i=3>]
> Sent: Tuesday, April 4, 2017 1:35 PM
> To: [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5796945&i=4>
> Cc: [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5796945&i=5>
> Subject: Re: Performance issue in Camel SFTP component
>
> Please spend more time to read the documentation and you can find
> options to tweak the options to use or not use FTP list etc, and there
> is also a stepwise option
>
> On Tue, Apr 4, 2017 at 9:16 AM, Ayush Dixit <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5796945&i=6>>
> wrote:
>
> >
> >
> > Hi ,
> >
> >
> >
> >
> >
> > We have implemented a camel route where we are having camel sftp
> > producer to transfer a files to remote SFTP location but on
> > performance testing on client environment and on our local
> > environment we have observed degradation in the time for
> > transferring files to
> remote SFTP location.
> > Please find the detailed analysis below. The we tried the various
> > test in our local environment. In each test we put around 22 files
> > on camel file consumer and each file took below time to write the
> > file. PFB details
> >
> >
> >
> >
> >
> > ============================================================
> > ============================
> >
> > When target directory having 20,000 files. Camel sftp producer took
> > around
> > 1 minute 43 second to a transfer file
> >
> > ============================================================
> > ============================
> >
> >
> >
> > DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.FileConsumer> About to process file:
> > GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses
> > /1 /ST30PERFORPMAJAR17020726.txt] using exchange:
> > Exchange[ST30PERFORPMAJAR17020726.txt]
> >
> > INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRes
> > po
> > nseFtpRouteBuilder> Picked IOP response file :
> > ST30PERFORPMAJAR17020726.txt from IOP FTP directory
> >
> > INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRes
> > po
> > nseFtpRouteBuilder> Sending IOP response file :
> > ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory
> >
> > DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://
> 10.100.150.
> > 190/icoms/1?download=false&maxMessagesPerPoll=10&
> > password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[
> > ST30PERFORPMAJAR17020726.txt]
> >
> >
> >
> >
> >
> >
> >
> > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.GenericFileConverter> Read file
> > /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFO
> > RP
> > MAJAR17020726.txt
> > (no charset)
> >
> > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.remote.SftpOperations> About to
> > store
> > file: QST30PERFORPMAJAR17020726.txt using stream:
> > java.io.BufferedInputStream@54a89ff5
> >
> > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.remote.SftpOperations> Took 0.658
> > seconds
> > (658 millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP
> > client
> > returned: true
> >
> >
> >
> > DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.remote.RemoteFileProducer> Wrote
> > [icoms/1/QST30PERFORPMAJAR17020726.txt]
> > to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&;
> > maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmause
> > r]
> > ]
> >
> > DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.remote.SftpOperations> Renaming file:
> > icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/
> > ST30PERFORPMAJAR17020726.txt
> >
> > INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRes
> > po
> > nseFtpRouteBuilder> IOP response file : ST30PERFORPMAJAR17020726.txt
> > successfully sent to ICOMS FTP directory
> >
> >
> >
> >
> >
> > ============================================================
> > ===========================
> >
> > When target directory having 40,000 files. Camel sftp producer took
> > around
> > 3 minute 17 second to transfer file
> >
> > ============================================================
> > ===========================
> >
> >
> >
> > DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.FileConsumer> About to process file:
> > GenericFile[/data/users/slvm02/smp53/domains/v
> >
> >
> > mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt]
> > using exchange: Exchange[SX30RPMAJAR1702483756.txt]
> >
> > INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6
> > -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRes
> > po
> > nseFtpRouteBuilder>
> > Picked IOP response file :
> >
> > SX30RPMAJAR1702483756.txt from IOP
> > FTP directory
> >
> > INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6
> > -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRes
> > po
> > nseFtpRouteBuilder>
> > Sending IOP response file
> >
> > : SX30RPMAJAR1702483756.txt to ICOMS
> > FTP directory
> >
> > DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6
> > -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://
> 10.100.150.
> > 190/icoms/1?download=false&maxMessagesPer
> >
> > Poll=10&password=xxxxxx&
> > tempPrefix=Q&username=sigmauser] Exchange[SX30RPMAJAR1702483756.txt]
> >
> >
> >
> >
> >
> > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
> > -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.GenericFileConverter> Read file
> > /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJ
> > AR
> > 1702483756.txt
> > (no charset)
> >
> > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
> > -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.remote.SftpOperations> About to
> > store
> > file: QSX30RPMAJAR1702483756.txt using stream:
> > java.io.BufferedInputStream@
> > 7954cfa8
> >
> > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
> > -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.remote.SftpOperations> Took 0.659
> > seconds
> > (659 millis) to store file: QSX30RPMAJAR1702483756.txt and FTP
> > client
> > returned: true
> >
> > DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6
> > -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.remote.RemoteFileProducer> Wrote
> > [icoms/1/QSX30RPMAJAR1702483756.txt]
> > to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&;
> > maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmause
> > r]
> > ]
> >
> > DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6
> > -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > org.apache.camel.component.file.remote.SftpOperations> Renaming file:
> > icoms/1/QSX30RPMAJAR1702483756.txt to:
> > icoms/1/SX30RPMAJAR1702483756.txt
> >
> > INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6
> > -
> > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRes
> > po
> > nseFtpRouteBuilder> IOP response file : SX30RPMAJAR1702483756.txt
> > successfully sent to ICOMS FTP directory
> >
> >
> >
> >
> >
> >
> >
> > ============================================================
> > ========================================
> >
> > Similarly when we achieved the files from target directory. It took
> > around
> > 6 sec.It seems like there is a performance issue with camel sftp
> component.
> > Does it list the files in target directory which is taking time.
> >
> > ============================================================
> > =======================================
> >
> >
> >
> > PFB the producer route which we set up
> >
> > sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&;
> > password=xxxxxx&tempPrefix=Q&username=sigmauser
> >
> >
> >
> >
> >
> > Please let us know if you need anything else.
> >
> >
> >
> >
> >
> > *Ayush Dixit*
> > *Software Engineer*
> > [hidden email]
> > <http:///user/SendEmail.jtp?type=node&node=5796945&i=7>
> >
> > O: +91.20.25661992 x 321 <+91%2020%202566%201992>
> <+91%2020%202566%201992> | M:
> > +91-814-982-6260 <+91%2081498%2026260> <+91%2081498%2026260> | Skype:
> ayushdixit421
> >
> > [image: cid:image001.jpg@01D16809.E12C0D50]
> > <http://www.sigma-systems.com/>
> >
> >
> >
> >
> >
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
>
> ------------------------------
> If you reply to this email, your message will be added to the
> discussion
> below:
> http://camel.465427.n5.nabble.com/Performance-issue-in-
> Camel-SFTP-component-tp5796893p5796945.html
> To start a new topic under Camel - Users, email
> ml-node+s465427n465428...@n5.nabble.com
> To unsubscribe from Camel - Users, click here
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsu
> bscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aUBnbWFpb
> C5jb218NDY1NDI4fDE1MzI5MTE2NTY=>
> .
> NAML
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macr
> o_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namesp
> aces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.vi
> ew.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%
> 3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%2
> 1nabble%3Aemail.naml>
>
--
View this message in context:
http://camel.465427.n5.nabble.com/Performance-issue-in-Camel-SFTP-component-tp5796893p5796947.html
Sent from the Camel - Users mailing list archive at Nabble.com.
