Re: Why doesn't SFTP "move" option work??
I took the FromFtpMoveFileTest and modified it for SFTP - that works, so "move" works for SFTP. My problem is that because the current release requires hard-coding the URI for (s)ftp(s) consumer - I implemented a custom processor that instantiates the SFTP component, SFTP endpoint and SFTP consumer, all programatically. Obviously, I'm missing something. Unfortunately, it's pretty much a show-stopper for us to not be able to SFTP *from* (consume) dynamically (runtime-calculated) URIs. On Wed, Apr 17, 2013 at 4:54 PM, Chris Wolf wrote: > I notice there's a unit test, FromFtpMoveFileTest, for plain FTP, but > not for SFTP - should I file a bug, or am I doing something > wrong? > > On Wed, Apr 17, 2013 at 11:39 AM, Chris Wolf wrote: >> Trying to use Sftp component. I am using localWorkDirectory because >> I don't want the whole files loaded into memory. >> Seems like the files *are* being downloaded but they are NOT being >> moved, on the remote side to the directory indicated by the "move" >> option. >> >> >> Here, on a poll, it gets a file, while downloading to >> "localWorkDirectory", the file's name is appended with ".inprogress", >> then it's >> renamed to the base file name without ".inprogress" - so far, so good, >> except on the server-side, it's not moved to the "move" directory. >> >> [#4 - sftp://localhost/download] SftpOperations DEBUG Retrieve file to >> local work file result: true >> [#4 - sftp://localhost/download] SftpOperations TRACE Renaming local >> in progress file from: >> \tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv.inprogress to: >> \tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv >> [#4 - sftp://localhost/download] SftpConsumer TRACE Retrieved file: >> download/ADPT5_MKT_20120319_VOIR_DAILY.csv from: >> Endpoint[sftp://localhost/download] >> >> ...then, at poll n+1, I see: >> >> [#4 - sftp://localhost/download] SftpConsumer TRACE >> SftpFile[fileName=ADPT5_MKT_20120319_VOIR_DAILY.csv, >> longName=-rwxrwxrwx 1 adpt5adpt5 298313 Apr 17 09:42 >> ADPT5_MKT_20120319_VOIR_DAILY.csv, dir=false] >> [#4 - sftp://localhost/download] SftpConsumer TRACE Skipping as file >> is already in progress: ADPT5_MKT_20120319_VOIR_DAILY.csv >> >> I guess that's the idempotent mechanism, but that file should have >> been moved out of that directory. >> >> >> SftpConfiguration: >> >> chmod = null, >> ciphers = null, >> compression = 0, >> connectTimeout = 1, >> directory = download, >> host = localhost, >> knownHostsFile = null, >> password = adpt5, >> port = 22, >> privateKeyFile = null, >> privateKeyFilePassphrase = null, >> protocol = sftp, >> separator = UNIX, >> serverAliveCountMax = 1, >> serverAliveInterval = 0, >> siteCommand = null, >> soTimeout = 0, >> strictHostKeyChecking = no, >> timeout = 3, >> username = adpt5, >> binary = true, >> passiveMode = false, >> stepwise = true, >> throwExceptionOnConnectFailed = false >> >> [1) thread #0 - timer://kickoff] INFO SftpEndpoint: >> antFilter = null, >> bufferSize = 131072, >> camelContext = CamelContext(camel-1), >> charset = null, >> component = org.apache.camel.component.file.remote.SftpComponent@1e34f445, >> configuration = download, >> consumerProperties = null, >> doneFileName = null, >> endpointConfiguration = >> org.apache.camel.impl.MappedEndpointConfiguration@8292f75a, >> endpointKey = sftp://localhost/download, >> endpointUri = sftp://localhost/download, >> exchangePattern = InOnly, >> exchanges = [Exchange[ADPT5_MKT_20090211_MICF.csv], >> Exchange[ADPT5_MKT_20090211_MINOCF.csv], >> Exchange[ADPT5_MKT_20090305_VOE.csv], >> Exchange[ADPT5_MKT_20090305_VOIR.csv], >> Exchange[ADPT5_MKT_20120319_DAILY.csv], >> Exchange[ADPT5_MKT_20120319_FX.csv], >> Exchange[ADPT5_MKT_20120319_MTX_DAILY.csv], >> Exchange[ADPT5_MKT_20120319_VOIR_DAILY.csv], >> Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-D.csv], >> Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-M.csv], >> Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-Y.csv]], >> exclude = null, >> exclusiveReadLockStrategy = null, >> fileExist = Override, >> fileName = null, >> fileSeparator = /, >> filter = ms.algo.adapt.camel.cpmd.CPMDFileFilter@54aa2db, >> genericFileProcessStrategy = >> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530, >> id = endpoint13, >> idempotentRepository = null, >> include = null, >> inProgressRepository = >> org.apache.camel.processor.idempotent.MemoryIdempotentRepository@563b100c, >> localWorkDirectory = /tmp/local/ftpwork, >> maxDepth = 2147483647, >> maximumReconnectAttempts = 3, >> maxMessagesPerPoll = 0, >> minDepth = 0, >> move = ${file:parent}/../archive/${file:onlyname},, >> moveExisting = null, >> moveFailed = null, >> preMove = null, >> processStrategy = >> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530, >> readLock = none, >> readLockCheckInterval = 5000, >> readLockMinLength = 1, >> readLockTimeout = 2, >> reconnectDelay = 1000, >> scheme = sftp, >> so
Re: Why doesn't SFTP "move" option work??
I notice there's a unit test, FromFtpMoveFileTest, for plain FTP, but not for SFTP - should I file a bug, or am I doing something wrong? On Wed, Apr 17, 2013 at 11:39 AM, Chris Wolf wrote: > Trying to use Sftp component. I am using localWorkDirectory because > I don't want the whole files loaded into memory. > Seems like the files *are* being downloaded but they are NOT being > moved, on the remote side to the directory indicated by the "move" > option. > > > Here, on a poll, it gets a file, while downloading to > "localWorkDirectory", the file's name is appended with ".inprogress", > then it's > renamed to the base file name without ".inprogress" - so far, so good, > except on the server-side, it's not moved to the "move" directory. > > [#4 - sftp://localhost/download] SftpOperations DEBUG Retrieve file to > local work file result: true > [#4 - sftp://localhost/download] SftpOperations TRACE Renaming local > in progress file from: > \tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv.inprogress to: > \tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv > [#4 - sftp://localhost/download] SftpConsumer TRACE Retrieved file: > download/ADPT5_MKT_20120319_VOIR_DAILY.csv from: > Endpoint[sftp://localhost/download] > > ...then, at poll n+1, I see: > > [#4 - sftp://localhost/download] SftpConsumer TRACE > SftpFile[fileName=ADPT5_MKT_20120319_VOIR_DAILY.csv, > longName=-rwxrwxrwx 1 adpt5adpt5 298313 Apr 17 09:42 > ADPT5_MKT_20120319_VOIR_DAILY.csv, dir=false] > [#4 - sftp://localhost/download] SftpConsumer TRACE Skipping as file > is already in progress: ADPT5_MKT_20120319_VOIR_DAILY.csv > > I guess that's the idempotent mechanism, but that file should have > been moved out of that directory. > > > SftpConfiguration: > > chmod = null, > ciphers = null, > compression = 0, > connectTimeout = 1, > directory = download, > host = localhost, > knownHostsFile = null, > password = adpt5, > port = 22, > privateKeyFile = null, > privateKeyFilePassphrase = null, > protocol = sftp, > separator = UNIX, > serverAliveCountMax = 1, > serverAliveInterval = 0, > siteCommand = null, > soTimeout = 0, > strictHostKeyChecking = no, > timeout = 3, > username = adpt5, > binary = true, > passiveMode = false, > stepwise = true, > throwExceptionOnConnectFailed = false > > [1) thread #0 - timer://kickoff] INFO SftpEndpoint: > antFilter = null, > bufferSize = 131072, > camelContext = CamelContext(camel-1), > charset = null, > component = org.apache.camel.component.file.remote.SftpComponent@1e34f445, > configuration = download, > consumerProperties = null, > doneFileName = null, > endpointConfiguration = > org.apache.camel.impl.MappedEndpointConfiguration@8292f75a, > endpointKey = sftp://localhost/download, > endpointUri = sftp://localhost/download, > exchangePattern = InOnly, > exchanges = [Exchange[ADPT5_MKT_20090211_MICF.csv], > Exchange[ADPT5_MKT_20090211_MINOCF.csv], > Exchange[ADPT5_MKT_20090305_VOE.csv], > Exchange[ADPT5_MKT_20090305_VOIR.csv], > Exchange[ADPT5_MKT_20120319_DAILY.csv], > Exchange[ADPT5_MKT_20120319_FX.csv], > Exchange[ADPT5_MKT_20120319_MTX_DAILY.csv], > Exchange[ADPT5_MKT_20120319_VOIR_DAILY.csv], > Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-D.csv], > Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-M.csv], > Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-Y.csv]], > exclude = null, > exclusiveReadLockStrategy = null, > fileExist = Override, > fileName = null, > fileSeparator = /, > filter = ms.algo.adapt.camel.cpmd.CPMDFileFilter@54aa2db, > genericFileProcessStrategy = > org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530, > id = endpoint13, > idempotentRepository = null, > include = null, > inProgressRepository = > org.apache.camel.processor.idempotent.MemoryIdempotentRepository@563b100c, > localWorkDirectory = /tmp/local/ftpwork, > maxDepth = 2147483647, > maximumReconnectAttempts = 3, > maxMessagesPerPoll = 0, > minDepth = 0, > move = ${file:parent}/../archive/${file:onlyname},, > moveExisting = null, > moveFailed = null, > preMove = null, > processStrategy = > org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530, > readLock = none, > readLockCheckInterval = 5000, > readLockMinLength = 1, > readLockTimeout = 2, > reconnectDelay = 1000, > scheme = sftp, > sortBy = null, > sorter = null, > status = Started, > tempFileName = null, > tempPrefix = null, > version = 2.10.4, > allowNullBody = false, > autoCreate = true, > delete = false, > directoryMustExist = false, > disconnect = false, > eagerDeleteTargetFile = true, > eagerMaxMessagesPerPoll = true, > fastExistsCheck = false, > flatten = false, > keepLastModified = false, > lenientProperties = false, > noop = false, > recursive = false, > runAllowed = true, > singleton = true, > started = true, > starting = false, > startingDirectoryMustExist = false, > stopped = false, > stopping = false, > suspended = false, > suspending = false, > synchronous = false
Why doesn't SFTP "move" option work??
Trying to use Sftp component. I am using localWorkDirectory because I don't want the whole files loaded into memory. Seems like the files *are* being downloaded but they are NOT being moved, on the remote side to the directory indicated by the "move" option. Here, on a poll, it gets a file, while downloading to "localWorkDirectory", the file's name is appended with ".inprogress", then it's renamed to the base file name without ".inprogress" - so far, so good, except on the server-side, it's not moved to the "move" directory. [#4 - sftp://localhost/download] SftpOperations DEBUG Retrieve file to local work file result: true [#4 - sftp://localhost/download] SftpOperations TRACE Renaming local in progress file from: \tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv.inprogress to: \tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv [#4 - sftp://localhost/download] SftpConsumer TRACE Retrieved file: download/ADPT5_MKT_20120319_VOIR_DAILY.csv from: Endpoint[sftp://localhost/download] ...then, at poll n+1, I see: [#4 - sftp://localhost/download] SftpConsumer TRACE SftpFile[fileName=ADPT5_MKT_20120319_VOIR_DAILY.csv, longName=-rwxrwxrwx 1 adpt5adpt5 298313 Apr 17 09:42 ADPT5_MKT_20120319_VOIR_DAILY.csv, dir=false] [#4 - sftp://localhost/download] SftpConsumer TRACE Skipping as file is already in progress: ADPT5_MKT_20120319_VOIR_DAILY.csv I guess that's the idempotent mechanism, but that file should have been moved out of that directory. SftpConfiguration: chmod = null, ciphers = null, compression = 0, connectTimeout = 1, directory = download, host = localhost, knownHostsFile = null, password = adpt5, port = 22, privateKeyFile = null, privateKeyFilePassphrase = null, protocol = sftp, separator = UNIX, serverAliveCountMax = 1, serverAliveInterval = 0, siteCommand = null, soTimeout = 0, strictHostKeyChecking = no, timeout = 3, username = adpt5, binary = true, passiveMode = false, stepwise = true, throwExceptionOnConnectFailed = false [1) thread #0 - timer://kickoff] INFO SftpEndpoint: antFilter = null, bufferSize = 131072, camelContext = CamelContext(camel-1), charset = null, component = org.apache.camel.component.file.remote.SftpComponent@1e34f445, configuration = download, consumerProperties = null, doneFileName = null, endpointConfiguration = org.apache.camel.impl.MappedEndpointConfiguration@8292f75a, endpointKey = sftp://localhost/download, endpointUri = sftp://localhost/download, exchangePattern = InOnly, exchanges = [Exchange[ADPT5_MKT_20090211_MICF.csv], Exchange[ADPT5_MKT_20090211_MINOCF.csv], Exchange[ADPT5_MKT_20090305_VOE.csv], Exchange[ADPT5_MKT_20090305_VOIR.csv], Exchange[ADPT5_MKT_20120319_DAILY.csv], Exchange[ADPT5_MKT_20120319_FX.csv], Exchange[ADPT5_MKT_20120319_MTX_DAILY.csv], Exchange[ADPT5_MKT_20120319_VOIR_DAILY.csv], Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-D.csv], Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-M.csv], Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-Y.csv]], exclude = null, exclusiveReadLockStrategy = null, fileExist = Override, fileName = null, fileSeparator = /, filter = ms.algo.adapt.camel.cpmd.CPMDFileFilter@54aa2db, genericFileProcessStrategy = org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530, id = endpoint13, idempotentRepository = null, include = null, inProgressRepository = org.apache.camel.processor.idempotent.MemoryIdempotentRepository@563b100c, localWorkDirectory = /tmp/local/ftpwork, maxDepth = 2147483647, maximumReconnectAttempts = 3, maxMessagesPerPoll = 0, minDepth = 0, move = ${file:parent}/../archive/${file:onlyname},, moveExisting = null, moveFailed = null, preMove = null, processStrategy = org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530, readLock = none, readLockCheckInterval = 5000, readLockMinLength = 1, readLockTimeout = 2, reconnectDelay = 1000, scheme = sftp, sortBy = null, sorter = null, status = Started, tempFileName = null, tempPrefix = null, version = 2.10.4, allowNullBody = false, autoCreate = true, delete = false, directoryMustExist = false, disconnect = false, eagerDeleteTargetFile = true, eagerMaxMessagesPerPoll = true, fastExistsCheck = false, flatten = false, keepLastModified = false, lenientProperties = false, noop = false, recursive = false, runAllowed = true, singleton = true, started = true, starting = false, startingDirectoryMustExist = false, stopped = false, stopping = false, suspended = false, suspending = false, synchronous = false