Please spend more time to read the documentation and you can find options to tweak the options to use or not use FTP list etc, and there is also a stepwise option
On Tue, Apr 4, 2017 at 9:16 AM, Ayush Dixit <ayush.di...@sigma-systems.com> wrote: > > > Hi , > > > > > > We have implemented a camel route where we are having camel sftp producer > to transfer a files to remote SFTP location but on performance testing on > client environment and on our local environment we have observed > degradation in the time for transferring files to remote SFTP location. > Please find the detailed analysis below. The we tried the various test in > our local environment. In each test we put around 22 files on camel file > consumer and each file took below time to write the file. PFB details > > > > > > ============================================================ > ============================ > > When target directory having 20,000 files. Camel sftp producer took around > 1 minute 43 second to a transfer file > > ============================================================ > ============================ > > > > DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.FileConsumer> About to process file: > GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt] > using exchange: Exchange[ST30PERFORPMAJAR17020726.txt] > > INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> > Picked IOP response file : ST30PERFORPMAJAR17020726.txt from IOP FTP > directory > > INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> > Sending IOP response file : ST30PERFORPMAJAR17020726.txt to ICOMS FTP > directory > > DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150. > 190/icoms/1?download=false&maxMessagesPerPoll=10& > password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[ > ST30PERFORPMAJAR17020726.txt] > > > > > > > > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.GenericFileConverter> Read file > /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt > (no charset) > > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> About to store > file: QST30PERFORPMAJAR17020726.txt using stream: > java.io.BufferedInputStream@54a89ff5 > > DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> Took 0.658 seconds > (658 millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP client > returned: true > > > > DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.RemoteFileProducer> Wrote > [icoms/1/QST30PERFORPMAJAR17020726.txt] > to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&; > maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]] > > DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> Renaming file: > icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/ > ST30PERFORPMAJAR17020726.txt > > INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> > IOP response file : ST30PERFORPMAJAR17020726.txt successfully sent to ICOMS > FTP directory > > > > > > ============================================================ > =========================== > > When target directory having 40,000 files. Camel sftp producer took around > 3 minute 17 second to transfer file > > ============================================================ > =========================== > > > > DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.FileConsumer> About to process file: > GenericFile[/data/users/slvm02/smp53/domains/v > > > mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt] > using exchange: Exchange[SX30RPMAJAR1702483756.txt] > > INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> > Picked IOP response file : > > SX30RPMAJAR1702483756.txt from IOP FTP > directory > > INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> > Sending IOP response file > > : SX30RPMAJAR1702483756.txt to ICOMS FTP > directory > > DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150. > 190/icoms/1?download=false&maxMessagesPer > > Poll=10&password=xxxxxx& > tempPrefix=Q&username=sigmauser] Exchange[SX30RPMAJAR1702483756.txt] > > > > > > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.GenericFileConverter> Read file > /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt > (no charset) > > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> About to store > file: QSX30RPMAJAR1702483756.txt using stream: java.io.BufferedInputStream@ > 7954cfa8 > > DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> Took 0.659 seconds > (659 millis) to store file: QSX30RPMAJAR1702483756.txt and FTP client > returned: true > > DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.RemoteFileProducer> Wrote > [icoms/1/QSX30RPMAJAR1702483756.txt] > to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&; > maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]] > > DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > org.apache.camel.component.file.remote.SftpOperations> Renaming file: > icoms/1/QSX30RPMAJAR1702483756.txt to: icoms/1/SX30RPMAJAR1702483756.txt > > INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - > file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) > com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> > IOP response file : SX30RPMAJAR1702483756.txt successfully sent to ICOMS > FTP directory > > > > > > > > ============================================================ > ======================================== > > Similarly when we achieved the files from target directory. It took around > 6 sec.It seems like there is a performance issue with camel sftp component. > Does it list the files in target directory which is taking time. > > ============================================================ > ======================================= > > > > PFB the producer route which we set up > > sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&; > password=xxxxxx&tempPrefix=Q&username=sigmauser > > > > > > Please let us know if you need anything else. > > > > > > *Ayush Dixit* > *Software Engineer* > ayush.di...@sigma-systems.com > > O: +91.20.25661992 x 321 <+91%2020%202566%201992> | M: +91-814-982-6260 > <+91%2081498%2026260> | Skype: ayushdixit421 > > [image: cid:image001.jpg@01D16809.E12C0D50] > <http://www.sigma-systems.com/> > > > > > -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
