Well, have you transferred the same amount of files with some other FTP client just to compare how much "slower" the ftp component is? 40,000 files in 3 minutes doesn't sounds ok.
On Wed, Apr 5, 2017 at 7:53 AM, Ayush Dixit [via Camel] < ml-node+s465427n5796945...@n5.nabble.com> wrote: > Hi, > > Can someone help me regarding the performance issue I am facing in camel > SFTP component ? I have gone through the documentation but did not find > anything that can improve performance. > > Thanks > Ayush > > -----Original Message----- > From: Ayush Dixit > Sent: Tuesday, April 4, 2017 3:40 PM > To: [hidden email] <http:///user/SendEmail.jtp?type=node&node=5796945&i=0> > Cc: '[hidden email] > <http:///user/SendEmail.jtp?type=node&node=5796945&i=1>' <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=5796945&i=2>> > Subject: RE: Performance issue in Camel SFTP component > > > Thanks Claus , I've tried stepwise=false in camel producer . No such > luck > > Also, I have tried download=false and useList=false. Still the performance > is same. I am referring the below link for camel documentation > > http://camel.apache.org/ftp2.html > > Is there is anything else you think I can try? > > Thanks > Ayush > > > -----Original Message----- > From: Claus Ibsen [mailto:[hidden email] > <http:///user/SendEmail.jtp?type=node&node=5796945&i=3>] > Sent: Tuesday, April 4, 2017 1:35 PM > To: [hidden email] <http:///user/SendEmail.jtp?type=node&node=5796945&i=4> > Cc: [hidden email] <http:///user/SendEmail.jtp?type=node&node=5796945&i=5> > Subject: Re: Performance issue in Camel SFTP component > > Please spend more time to read the documentation and you can find options > to tweak the options to use or not use FTP list etc, and there is also a > stepwise option > > On Tue, Apr 4, 2017 at 9:16 AM, Ayush Dixit <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=5796945&i=6>> > wrote: > > > > > > > Hi , > > > > > > > > > > > > We have implemented a camel route where we are having camel sftp > > producer to transfer a files to remote SFTP location but on > > performance testing on client environment and on our local environment > > we have observed degradation in the time for transferring files to > remote SFTP location. > > Please find the detailed analysis below. The we tried the various test > > in our local environment. In each test we put around 22 files on camel > > file consumer and each file took below time to write the file. PFB > > details > > > > > > > > > > > > ============================================================ > > ============================ > > > > When target directory having 20,000 files. Camel sftp producer took > > around > > 1 minute 43 second to a transfer file > > > > ============================================================ > > ============================ > > > > > > > > DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.FileConsumer> About to process file: > > GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1 > > /ST30PERFORPMAJAR17020726.txt] using exchange: > > Exchange[ST30PERFORPMAJAR17020726.txt] > > > > INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > > nseFtpRouteBuilder> Picked IOP response file : > > ST30PERFORPMAJAR17020726.txt from IOP FTP directory > > > > INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > > nseFtpRouteBuilder> Sending IOP response file : > > ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory > > > > DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp:// > 10.100.150. > > 190/icoms/1?download=false&maxMessagesPerPoll=10& > > password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[ > > ST30PERFORPMAJAR17020726.txt] > > > > > > > > > > > > > > > > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.GenericFileConverter> Read file > > /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORP > > MAJAR17020726.txt > > (no charset) > > > > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.remote.SftpOperations> About to store > > file: QST30PERFORPMAJAR17020726.txt using stream: > > java.io.BufferedInputStream@54a89ff5 > > > > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.remote.SftpOperations> Took 0.658 > > seconds > > (658 millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP > > client > > returned: true > > > > > > > > DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.remote.RemoteFileProducer> Wrote > > [icoms/1/QST30PERFORPMAJAR17020726.txt] > > to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&; > > maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser] > > ] > > > > DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.remote.SftpOperations> Renaming file: > > icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/ > > ST30PERFORPMAJAR17020726.txt > > > > INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > > nseFtpRouteBuilder> IOP response file : ST30PERFORPMAJAR17020726.txt > > successfully sent to ICOMS FTP directory > > > > > > > > > > > > ============================================================ > > =========================== > > > > When target directory having 40,000 files. Camel sftp producer took > > around > > 3 minute 17 second to transfer file > > > > ============================================================ > > =========================== > > > > > > > > DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.FileConsumer> About to process file: > > GenericFile[/data/users/slvm02/smp53/domains/v > > > > > > mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt] > > using exchange: Exchange[SX30RPMAJAR1702483756.txt] > > > > INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > > nseFtpRouteBuilder> > > Picked IOP response file : > > > > SX30RPMAJAR1702483756.txt from IOP FTP > > directory > > > > INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > > nseFtpRouteBuilder> > > Sending IOP response file > > > > : SX30RPMAJAR1702483756.txt to ICOMS > > FTP directory > > > > DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp:// > 10.100.150. > > 190/icoms/1?download=false&maxMessagesPer > > > > Poll=10&password=xxxxxx& > > tempPrefix=Q&username=sigmauser] Exchange[SX30RPMAJAR1702483756.txt] > > > > > > > > > > > > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.GenericFileConverter> Read file > > /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJAR > > 1702483756.txt > > (no charset) > > > > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.remote.SftpOperations> About to store > > file: QSX30RPMAJAR1702483756.txt using stream: > > java.io.BufferedInputStream@ > > 7954cfa8 > > > > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.remote.SftpOperations> Took 0.659 > > seconds > > (659 millis) to store file: QSX30RPMAJAR1702483756.txt and FTP client > > returned: true > > > > DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.remote.RemoteFileProducer> Wrote > > [icoms/1/QSX30RPMAJAR1702483756.txt] > > to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&; > > maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser] > > ] > > > > DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > org.apache.camel.component.file.remote.SftpOperations> Renaming file: > > icoms/1/QSX30RPMAJAR1702483756.txt to: > > icoms/1/SX30RPMAJAR1702483756.txt > > > > INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - > > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > > nseFtpRouteBuilder> IOP response file : SX30RPMAJAR1702483756.txt > > successfully sent to ICOMS FTP directory > > > > > > > > > > > > > > > > ============================================================ > > ======================================== > > > > Similarly when we achieved the files from target directory. It took > > around > > 6 sec.It seems like there is a performance issue with camel sftp > component. > > Does it list the files in target directory which is taking time. > > > > ============================================================ > > ======================================= > > > > > > > > PFB the producer route which we set up > > > > sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&; > > password=xxxxxx&tempPrefix=Q&username=sigmauser > > > > > > > > > > > > Please let us know if you need anything else. > > > > > > > > > > > > *Ayush Dixit* > > *Software Engineer* > > [hidden email] <http:///user/SendEmail.jtp?type=node&node=5796945&i=7> > > > > O: +91.20.25661992 x 321 <+91%2020%202566%201992> > <+91%2020%202566%201992> | M: > > +91-814-982-6260 <+91%2081498%2026260> <+91%2081498%2026260> | Skype: > ayushdixit421 > > > > [image: cid:image001.jpg@01D16809.E12C0D50] > > <http://www.sigma-systems.com/> > > > > > > > > > > > > > > -- > Claus Ibsen > ----------------- > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://camel.465427.n5.nabble.com/Performance-issue-in- > Camel-SFTP-component-tp5796893p5796945.html > To start a new topic under Camel - Users, email > ml-node+s465427n465428...@n5.nabble.com > To unsubscribe from Camel - Users, click here > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aUBnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=> > . > NAML > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://camel.465427.n5.nabble.com/Performance-issue-in-Camel-SFTP-component-tp5796893p5796947.html Sent from the Camel - Users mailing list archive at Nabble.com.
