Ayush Dixit created CAMEL-11104:
-----------------------------------
Summary: Performance issue in camel sftp
Key: CAMEL-11104
URL: https://issues.apache.org/jira/browse/CAMEL-11104
Project: Camel
Issue Type: Bug
Components: camel-sftp
Affects Versions: 2.15.2
Environment: Performance Enviroment
Reporter: Ayush Dixit
We have implemented a camel route where we are having camel sftp producer to
transfer a files to remote SFTP location but on performance testing on client
environment and on our local environment we have observed degradation in the
time for transferring files to remote SFTP location. Please find the detailed
analysis below. The we tried the various test in our local environment. In each
test we put around 22 files on camel file consumer and each file took below
time to write the file. PFB details
========================================================================================
When target directory having 20,000 files. Camel sftp producer took around 1
minute 43 second to a transfer file
========================================================================================
DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.FileConsumer> About to process file:
GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt]
using exchange: Exchange[ST30PERFORPMAJAR17020726.txt]
INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Picked IOP response file : ST30PERFORPMAJAR17020726.txt from IOP FTP directory
INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Sending IOP response file : ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory
DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.processor.SendProcessor> >>>>
Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]
Exchange[ST30PERFORPMAJAR17020726.txt]
DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.GenericFileConverter> Read file
/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt
(no charset)
DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> About to store file:
QST30PERFORPMAJAR17020726.txt using stream: java.io.BufferedInputStream@54a89ff5
DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Took 0.658 seconds (658
millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP client returned:
true
DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.RemoteFileProducer> Wrote
[icoms/1/QST30PERFORPMAJAR17020726.txt] to
[Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Renaming file:
icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/ST30PERFORPMAJAR17020726.txt
INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
IOP response file : ST30PERFORPMAJAR17020726.txt successfully sent to ICOMS
FTP directory
=======================================================================================
When target directory having 40,000 files. Camel sftp producer took around 3
minute 17 second to transfer file
=======================================================================================
DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.FileConsumer> About to process file:
GenericFile[/data/users/slvm02/smp53/domains/v
mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt] using exchange:
Exchange[SX30RPMAJAR1702483756.txt]
INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Picked IOP response file :
SX30RPMAJAR1702483756.txt from IOP FTP directory
INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Sending IOP response file
: SX30RPMAJAR1702483756.txt to ICOMS FTP
directory
DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.processor.SendProcessor> >>>>
Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPer
Poll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]
Exchange[SX30RPMAJAR1702483756.txt]
DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.GenericFileConverter> Read file
/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt
(no charset)
DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> About to store file:
QSX30RPMAJAR1702483756.txt using stream: java.io.BufferedInputStream@7954cfa8
DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Took 0.659 seconds (659
millis) to store file: QSX30RPMAJAR1702483756.txt and FTP client returned: true
DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.RemoteFileProducer> Wrote
[icoms/1/QSX30RPMAJAR1702483756.txt] to
[Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Renaming file:
icoms/1/QSX30RPMAJAR1702483756.txt to: icoms/1/SX30RPMAJAR1702483756.txt
INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
IOP response file : SX30RPMAJAR1702483756.txt successfully sent to ICOMS FTP
directory
====================================================================================================
Similarly when we achieved the files from target directory. It took around 6
sec.It seems like there is a performance issue with camel sftp component. Does
it list the files in target directory which is taking time.
===================================================================================================
PFB the producer route which we set up
sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
