Ayush Dixit created CAMEL-11102:
-----------------------------------
Summary: Camel SFTP is taking time to transfer file on sftp
server.When target target directory contains lot of files
Key: CAMEL-11102
URL: https://issues.apache.org/jira/browse/CAMEL-11102
Project: Camel
Issue Type: Bug
Components: camel-sftp
Affects Versions: 2.15.2
Environment: ==========================================================
When target directory having 20,000 files. Camel sftp producer took around 1
minute 43 second to transfer file
==========================================================
DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.FileConsumer> About to process file:
GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt]
using exchange: Exchange[ST30PERFORPMAJAR17020726.txt]
INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Picked IOP response file : ST30PERFORPMAJAR17020726.txt from IOP FTP directory
INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Sending IOP response file : ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory
DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.processor.SendProcessor> >>>>
Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]
Exchange[ST30PERFORPMAJAR17020726.txt]
DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.GenericFileConverter> Read file
/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt
(no charset)
DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> About to store file:
QST30PERFORPMAJAR17020726.txt using stream: java.io.BufferedInputStream@54a89ff5
DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Took 0.658 seconds (658
millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP client returned:
true
DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.RemoteFileProducer> Wrote
[icoms/1/QST30PERFORPMAJAR17020726.txt] to
[Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Renaming file:
icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/ST30PERFORPMAJAR17020726.txt
INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
IOP response file : ST30PERFORPMAJAR17020726.txt successfully sent to ICOMS
FTP directory
==========================================================When target directory
having 40,000 files. Camel sftp producer took around 3 minute 17 second to
transfer file
==================================== ====================
DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.FileConsumer> About to process file:
GenericFile[/data/users/slvm02/smp53/domains/v
mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt] using
exchange: Exchange[SX30RPMAJAR1702483756.txt]
INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Picked IOP response file :
SX30RPMAJAR1702483756.txt from IOP FTP directory
INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Sending IOP response file
: SX30RPMAJAR1702483756.txt to ICOMS FTP directory
DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.processor.SendProcessor> >>>>
Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPer
Poll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]
Exchange[SX30RPMAJAR1702483756.txt]
DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.GenericFileConverter> Read file
/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt
(no charset)
DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> About to store file:
QSX30RPMAJAR1702483756.txt using stream: java.io.BufferedInputStream@7954cfa8
DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Took 0.659 seconds (659
millis) to store file: QSX30RPMAJAR1702483756.txt and FTP client returned: true
DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.RemoteFileProducer> Wrote
[icoms/1/QSX30RPMAJAR1702483756.txt] to
[Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Renaming file:
icoms/1/QSX30RPMAJAR1702483756.txt to: icoms/1/SX30RPMAJAR1702483756.txt
INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 -
file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
IOP response file : SX30RPMAJAR1702483756.txt successfully sent to ICOMS FTP
directory
==========================================================
Similarly when we achieved the files from target directory. It took around 6
sec.It seems like there is a performance issue with camel sftp component. Does
it list the files in target directory which is taking time.
==========================================================
PFB the producer route which we set up
sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser
Reporter: Ayush Dixit
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
