Hi Ayush,
As souciance suggested, i guess you should use another FTP client to
transfer files and better measure camel FTP component against and get some
benchmark results to compare in a better way.
If the file transfer is what we are talking, there may be lots of reason
why you have low performance. Network connectivity, firewalls, proxies, big
files & blocking IO etc..
Even if there may be a problem regarding camel, having what you provided is
not sufficient to diagnose and resolve the problem.
Rough to say that, you seem you have to deep dive into details to resolve
the problem.
Regards
On Mon, Apr 10, 2017 at 8:12 AM, Ayush Dixit <ayush.di...@sigma-systems.com>
wrote:
> Hi
>
> Any luck in solving the problem?
>
> Regards
> Ayush
>
> -----Original Message-----
> From: Ayush Dixit
> Sent: Thursday, April 6, 2017 11:01 AM
> To: 'users@camel.apache.org' <users@camel.apache.org>
> Subject: RE: Performance issue in Camel SFTP component
>
> Should I raise a JIRA for that?
>
> -----Original Message-----
> From: Ayush Dixit
> Sent: Wednesday, April 5, 2017 1:22 PM
> To: users@camel.apache.org
> Subject: RE: Performance issue in Camel SFTP component
>
> Hi Claus,
>
> Already tried that , See below endpoint configuration . No luck :(
>
> to("sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&;
> password=xxxxxx&tempPrefix=Q&username=sigmauser&stepwise=false")
>
> Thanks
> Ayush
>
> -----Original Message-----
> From: Claus Ibsen [mailto:claus.ib...@gmail.com]
> Sent: Wednesday, April 5, 2017 1:08 PM
> To: users@camel.apache.org
> Subject: Re: Performance issue in Camel SFTP component
>
> Try with maxMessagesPerPoll to set an upper limit.
>
> On Tue, Apr 4, 2017 at 12:10 PM, Ayush Dixit <
> ayush.di...@sigma-systems.com> wrote:
> >
> > Thanks Claus , I've tried stepwise=false in camel producer . No such
> > luck
> >
> > Also, I have tried download=false and useList=false. Still the
> > performance is same. I am referring the below link for camel
> > documentation
> >
> > http://camel.apache.org/ftp2.html
> >
> > Is there is anything else you think I can try?
> >
> > Thanks
> > Ayush
> >
> >
> > -----Original Message-----
> > From: Claus Ibsen [mailto:claus.ib...@gmail.com]
> > Sent: Tuesday, April 4, 2017 1:35 PM
> > To: users@camel.apache.org
> > Cc: users-subscr...@camel.apache.org
> > Subject: Re: Performance issue in Camel SFTP component
> >
> > Please spend more time to read the documentation and you can find
> > options to tweak the options to use or not use FTP list etc, and there
> > is also a stepwise option
> >
> > On Tue, Apr 4, 2017 at 9:16 AM, Ayush Dixit
> > <ayush.di...@sigma-systems.com>
> > wrote:
> >
> >>
> >>
> >> Hi ,
> >>
> >>
> >>
> >>
> >>
> >> We have implemented a camel route where we are having camel sftp
> >> producer to transfer a files to remote SFTP location but on
> >> performance testing on client environment and on our local
> >> environment we have observed degradation in the time for transferring
> files to remote SFTP location.
> >> Please find the detailed analysis below. The we tried the various
> >> test in our local environment. In each test we put around 22 files on
> >> camel file consumer and each file took below time to write the file.
> >> PFB details
> >>
> >>
> >>
> >>
> >>
> >> ============================================================
> >> ============================
> >>
> >> When target directory having 20,000 files. Camel sftp producer took
> >> around
> >> 1 minute 43 second to a transfer file
> >>
> >> ============================================================
> >> ============================
> >>
> >>
> >>
> >> DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.FileConsumer> About to process file:
> >> GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/
> >> 1 /ST30PERFORPMAJAR17020726.txt] using exchange:
> >> Exchange[ST30PERFORPMAJAR17020726.txt]
> >>
> >> INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
> >> o
> >> nseFtpRouteBuilder> Picked IOP response file :
> >> ST30PERFORPMAJAR17020726.txt from IOP FTP directory
> >>
> >> INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
> >> o
> >> nseFtpRouteBuilder> Sending IOP response file :
> >> ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory
> >>
> >> DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://
> 10.100.150.
> >> 190/icoms/1?download=false&maxMessagesPerPoll=10&
> >> password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[
> >> ST30PERFORPMAJAR17020726.txt]
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.GenericFileConverter> Read file
> >> /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFOR
> >> P
> >> MAJAR17020726.txt
> >> (no charset)
> >>
> >> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.remote.SftpOperations> About to store
> >> file: QST30PERFORPMAJAR17020726.txt using stream:
> >> java.io.BufferedInputStream@54a89ff5
> >>
> >> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.remote.SftpOperations> Took 0.658
> >> seconds
> >> (658 millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP
> >> client
> >> returned: true
> >>
> >>
> >>
> >> DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.remote.RemoteFileProducer> Wrote
> >> [icoms/1/QST30PERFORPMAJAR17020726.txt]
> >> to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&;
> >> maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser
> >> ]
> >> ]
> >>
> >> DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.remote.SftpOperations> Renaming file:
> >> icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/
> >> ST30PERFORPMAJAR17020726.txt
> >>
> >> INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
> >> o
> >> nseFtpRouteBuilder> IOP response file : ST30PERFORPMAJAR17020726.txt
> >> successfully sent to ICOMS FTP directory
> >>
> >>
> >>
> >>
> >>
> >> ============================================================
> >> ===========================
> >>
> >> When target directory having 40,000 files. Camel sftp producer took
> >> around
> >> 3 minute 17 second to transfer file
> >>
> >> ============================================================
> >> ===========================
> >>
> >>
> >>
> >> DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.FileConsumer> About to process file:
> >> GenericFile[/data/users/slvm02/smp53/domains/v
> >>
> >>
> >> mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt]
> >> using exchange: Exchange[SX30RPMAJAR1702483756.txt]
> >>
> >> INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
> >> o
> >> nseFtpRouteBuilder>
> >> Picked IOP response file :
> >>
> >> SX30RPMAJAR1702483756.txt from IOP
> >> FTP directory
> >>
> >> INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
> >> o
> >> nseFtpRouteBuilder>
> >> Sending IOP response file
> >>
> >> : SX30RPMAJAR1702483756.txt to ICOMS
> >> FTP directory
> >>
> >> DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6
> >> -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://
> 10.100.150.
> >> 190/icoms/1?download=false&maxMessagesPer
> >>
> >> Poll=10&password=xxxxxx&
> >> tempPrefix=Q&username=sigmauser] Exchange[SX30RPMAJAR1702483756.txt]
> >>
> >>
> >>
> >>
> >>
> >> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
> >> -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.GenericFileConverter> Read file
> >> /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJA
> >> R
> >> 1702483756.txt
> >> (no charset)
> >>
> >> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
> >> -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.remote.SftpOperations> About to store
> >> file: QSX30RPMAJAR1702483756.txt using stream:
> >> java.io.BufferedInputStream@
> >> 7954cfa8
> >>
> >> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
> >> -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.remote.SftpOperations> Took 0.659
> >> seconds
> >> (659 millis) to store file: QSX30RPMAJAR1702483756.txt and FTP client
> >> returned: true
> >>
> >> DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6
> >> -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.remote.RemoteFileProducer> Wrote
> >> [icoms/1/QSX30RPMAJAR1702483756.txt]
> >> to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&;
> >> maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser
> >> ]
> >> ]
> >>
> >> DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6
> >> -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> org.apache.camel.component.file.remote.SftpOperations> Renaming file:
> >> icoms/1/QSX30RPMAJAR1702483756.txt to:
> >> icoms/1/SX30RPMAJAR1702483756.txt
> >>
> >> INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 -
> >> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
> >> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
> >> o
> >> nseFtpRouteBuilder> IOP response file : SX30RPMAJAR1702483756.txt
> >> successfully sent to ICOMS FTP directory
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> ============================================================
> >> ========================================
> >>
> >> Similarly when we achieved the files from target directory. It took
> >> around
> >> 6 sec.It seems like there is a performance issue with camel sftp
> component.
> >> Does it list the files in target directory which is taking time.
> >>
> >> ============================================================
> >> =======================================
> >>
> >>
> >>
> >> PFB the producer route which we set up
> >>
> >> sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&;
> >> password=xxxxxx&tempPrefix=Q&username=sigmauser
> >>
> >>
> >>
> >>
> >>
> >> Please let us know if you need anything else.
> >>
> >>
> >>
> >>
> >>
> >> *Ayush Dixit*
> >> *Software Engineer*
> >> ayush.di...@sigma-systems.com
> >>
> >> O: +91.20.25661992 x 321 <+91%2020%202566%201992> | M:
> >> +91-814-982-6260 <+91%2081498%2026260> | Skype: ayushdixit421
> >>
> >> [image: cid:image001.jpg@01D16809.E12C0D50]
> >> <http://www.sigma-systems.com/>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > http://davsclaus.com @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
