Hi, Can someone help me regarding the performance issue I am facing in camel SFTP component ? I have gone through the documentation but did not find anything that can improve performance.
Thanks Ayush -----Original Message----- From: Ayush Dixit Sent: Tuesday, April 4, 2017 3:40 PM To: users@camel.apache.org Cc: 'claus.ib...@gmail.com' <claus.ib...@gmail.com> Subject: RE: Performance issue in Camel SFTP component Thanks Claus , I've tried stepwise=false in camel producer . No such luck Also, I have tried download=false and useList=false. Still the performance is same. I am referring the below link for camel documentation http://camel.apache.org/ftp2.html Is there is anything else you think I can try? Thanks Ayush -----Original Message----- From: Claus Ibsen [mailto:claus.ib...@gmail.com] Sent: Tuesday, April 4, 2017 1:35 PM To: users@camel.apache.org Cc: users-subscr...@camel.apache.org Subject: Re: Performance issue in Camel SFTP component Please spend more time to read the documentation and you can find options to tweak the options to use or not use FTP list etc, and there is also a stepwise option On Tue, Apr 4, 2017 at 9:16 AM, Ayush Dixit <ayush.di...@sigma-systems.com> wrote: > > > Hi , > > > > > > We have implemented a camel route where we are having camel sftp > producer to transfer a files to remote SFTP location but on > performance testing on client environment and on our local environment > we have observed degradation in the time for transferring files to remote > SFTP location. > Please find the detailed analysis below. The we tried the various test > in our local environment. In each test we put around 22 files on camel > file consumer and each file took below time to write the file. PFB > details > > > > > > ============================================================ > ============================ > > When target directory having 20,000 files. Camel sftp producer took > around > 1 minute 43 second to a transfer file > > ============================================================ > ============================ > > > > DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.FileConsumer> About to process file: > GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1 > /ST30PERFORPMAJAR17020726.txt] using exchange: > Exchange[ST30PERFORPMAJAR17020726.txt] > > INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > nseFtpRouteBuilder> Picked IOP response file : > ST30PERFORPMAJAR17020726.txt from IOP FTP directory > > INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > nseFtpRouteBuilder> Sending IOP response file : > ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory > > DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150. > 190/icoms/1?download=false&maxMessagesPerPoll=10& > password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[ > ST30PERFORPMAJAR17020726.txt] > > > > > > > > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.GenericFileConverter> Read file > /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORP > MAJAR17020726.txt > (no charset) > > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> About to store > file: QST30PERFORPMAJAR17020726.txt using stream: > java.io.BufferedInputStream@54a89ff5 > > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> Took 0.658 > seconds > (658 millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP > client > returned: true > > > > DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.RemoteFileProducer> Wrote > [icoms/1/QST30PERFORPMAJAR17020726.txt] > to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&; > maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser] > ] > > DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> Renaming file: > icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/ > ST30PERFORPMAJAR17020726.txt > > INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > nseFtpRouteBuilder> IOP response file : ST30PERFORPMAJAR17020726.txt > successfully sent to ICOMS FTP directory > > > > > > ============================================================ > =========================== > > When target directory having 40,000 files. Camel sftp producer took > around > 3 minute 17 second to transfer file > > ============================================================ > =========================== > > > > DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.FileConsumer> About to process file: > GenericFile[/data/users/slvm02/smp53/domains/v > > > mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt] > using exchange: Exchange[SX30RPMAJAR1702483756.txt] > > INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > nseFtpRouteBuilder> > Picked IOP response file : > > SX30RPMAJAR1702483756.txt from IOP FTP > directory > > INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > nseFtpRouteBuilder> > Sending IOP response file > > : SX30RPMAJAR1702483756.txt to ICOMS > FTP directory > > DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150. > 190/icoms/1?download=false&maxMessagesPer > > Poll=10&password=xxxxxx& > tempPrefix=Q&username=sigmauser] Exchange[SX30RPMAJAR1702483756.txt] > > > > > > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.GenericFileConverter> Read file > /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJAR > 1702483756.txt > (no charset) > > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> About to store > file: QSX30RPMAJAR1702483756.txt using stream: > java.io.BufferedInputStream@ > 7954cfa8 > > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> Took 0.659 > seconds > (659 millis) to store file: QSX30RPMAJAR1702483756.txt and FTP client > returned: true > > DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.RemoteFileProducer> Wrote > [icoms/1/QSX30RPMAJAR1702483756.txt] > to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&; > maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser] > ] > > DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> Renaming file: > icoms/1/QSX30RPMAJAR1702483756.txt to: > icoms/1/SX30RPMAJAR1702483756.txt > > INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo > nseFtpRouteBuilder> IOP response file : SX30RPMAJAR1702483756.txt > successfully sent to ICOMS FTP directory > > > > > > > > ============================================================ > ======================================== > > Similarly when we achieved the files from target directory. It took > around > 6 sec.It seems like there is a performance issue with camel sftp component. > Does it list the files in target directory which is taking time. > > ============================================================ > ======================================= > > > > PFB the producer route which we set up > > sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&; > password=xxxxxx&tempPrefix=Q&username=sigmauser > > > > > > Please let us know if you need anything else. > > > > > > *Ayush Dixit* > *Software Engineer* > ayush.di...@sigma-systems.com > > O: +91.20.25661992 x 321 <+91%2020%202566%201992> | M: > +91-814-982-6260 <+91%2081498%2026260> | Skype: ayushdixit421 > > [image: cid:image001.jpg@01D16809.E12C0D50] > <http://www.sigma-systems.com/> > > > > > -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
