Try with maxMessagesPerPoll to set an upper limit. On Tue, Apr 4, 2017 at 12:10 PM, Ayush Dixit <ayush.di...@sigma-systems.com> wrote: > > Thanks Claus , I've tried stepwise=false in camel producer . No such luck > > Also, I have tried download=false and useList=false. Still the performance is > same. I am referring the below link for camel documentation > > http://camel.apache.org/ftp2.html > > Is there is anything else you think I can try? > > Thanks > Ayush > > > -----Original Message----- > From: Claus Ibsen [mailto:claus.ib...@gmail.com] > Sent: Tuesday, April 4, 2017 1:35 PM > To: users@camel.apache.org > Cc: users-subscr...@camel.apache.org > Subject: Re: Performance issue in Camel SFTP component > > Please spend more time to read the documentation and you can find options to > tweak the options to use or not use FTP list etc, and there is also a > stepwise option > > On Tue, Apr 4, 2017 at 9:16 AM, Ayush Dixit <ayush.di...@sigma-systems.com> > wrote: > >> >> >> Hi , >> >> >> >> >> >> We have implemented a camel route where we are having camel sftp >> producer to transfer a files to remote SFTP location but on >> performance testing on client environment and on our local environment >> we have observed degradation in the time for transferring files to remote >> SFTP location. >> Please find the detailed analysis below. The we tried the various test >> in our local environment. In each test we put around 22 files on camel >> file consumer and each file took below time to write the file. PFB >> details >> >> >> >> >> >> ============================================================ >> ============================ >> >> When target directory having 20,000 files. Camel sftp producer took >> around >> 1 minute 43 second to a transfer file >> >> ============================================================ >> ============================ >> >> >> >> DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.FileConsumer> About to process file: >> GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1 >> /ST30PERFORPMAJAR17020726.txt] using exchange: >> Exchange[ST30PERFORPMAJAR17020726.txt] >> >> INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo >> nseFtpRouteBuilder> Picked IOP response file : >> ST30PERFORPMAJAR17020726.txt from IOP FTP directory >> >> INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo >> nseFtpRouteBuilder> Sending IOP response file : >> ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory >> >> DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150. >> 190/icoms/1?download=false&maxMessagesPerPoll=10& >> password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[ >> ST30PERFORPMAJAR17020726.txt] >> >> >> >> >> >> >> >> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.GenericFileConverter> Read file >> /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORP >> MAJAR17020726.txt >> (no charset) >> >> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.remote.SftpOperations> About to store >> file: QST30PERFORPMAJAR17020726.txt using stream: >> java.io.BufferedInputStream@54a89ff5 >> >> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.remote.SftpOperations> Took 0.658 >> seconds >> (658 millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP >> client >> returned: true >> >> >> >> DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.remote.RemoteFileProducer> Wrote >> [icoms/1/QST30PERFORPMAJAR17020726.txt] >> to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&; >> maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser] >> ] >> >> DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.remote.SftpOperations> Renaming file: >> icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/ >> ST30PERFORPMAJAR17020726.txt >> >> INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo >> nseFtpRouteBuilder> IOP response file : ST30PERFORPMAJAR17020726.txt >> successfully sent to ICOMS FTP directory >> >> >> >> >> >> ============================================================ >> =========================== >> >> When target directory having 40,000 files. Camel sftp producer took >> around >> 3 minute 17 second to transfer file >> >> ============================================================ >> =========================== >> >> >> >> DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.FileConsumer> About to process file: >> GenericFile[/data/users/slvm02/smp53/domains/v >> >> >> mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt] >> using exchange: Exchange[SX30RPMAJAR1702483756.txt] >> >> INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo >> nseFtpRouteBuilder> >> Picked IOP response file : >> >> SX30RPMAJAR1702483756.txt from IOP FTP >> directory >> >> INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo >> nseFtpRouteBuilder> >> Sending IOP response file >> >> : SX30RPMAJAR1702483756.txt to ICOMS >> FTP directory >> >> DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150. >> 190/icoms/1?download=false&maxMessagesPer >> >> Poll=10&password=xxxxxx& >> tempPrefix=Q&username=sigmauser] Exchange[SX30RPMAJAR1702483756.txt] >> >> >> >> >> >> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.GenericFileConverter> Read file >> /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJAR >> 1702483756.txt >> (no charset) >> >> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.remote.SftpOperations> About to store >> file: QSX30RPMAJAR1702483756.txt using stream: >> java.io.BufferedInputStream@ >> 7954cfa8 >> >> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.remote.SftpOperations> Took 0.659 >> seconds >> (659 millis) to store file: QSX30RPMAJAR1702483756.txt and FTP client >> returned: true >> >> DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.remote.RemoteFileProducer> Wrote >> [icoms/1/QSX30RPMAJAR1702483756.txt] >> to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&; >> maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser] >> ] >> >> DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> org.apache.camel.component.file.remote.SftpOperations> Renaming file: >> icoms/1/QSX30RPMAJAR1702483756.txt to: >> icoms/1/SX30RPMAJAR1702483756.txt >> >> INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPRespo >> nseFtpRouteBuilder> IOP response file : SX30RPMAJAR1702483756.txt >> successfully sent to ICOMS FTP directory >> >> >> >> >> >> >> >> ============================================================ >> ======================================== >> >> Similarly when we achieved the files from target directory. It took >> around >> 6 sec.It seems like there is a performance issue with camel sftp component. >> Does it list the files in target directory which is taking time. >> >> ============================================================ >> ======================================= >> >> >> >> PFB the producer route which we set up >> >> sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&; >> password=xxxxxx&tempPrefix=Q&username=sigmauser >> >> >> >> >> >> Please let us know if you need anything else. >> >> >> >> >> >> *Ayush Dixit* >> *Software Engineer* >> ayush.di...@sigma-systems.com >> >> O: +91.20.25661992 x 321 <+91%2020%202566%201992> | M: >> +91-814-982-6260 <+91%2081498%2026260> | Skype: ayushdixit421 >> >> [image: cid:image001.jpg@01D16809.E12C0D50] >> <http://www.sigma-systems.com/> >> >> >> >> >> > > > > -- > Claus Ibsen > ----------------- > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2
-- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
