[jira] [Commented] (CAMEL-10782) SFTP: cannot get files from users home with readlock changed
[ https://issues.apache.org/jira/browse/CAMEL-10782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871638#comment-15871638 ] Stefan Roos commented on CAMEL-10782: - Actually camel is not able to retrieve the files ever from the root of the users home directory, when readLock=changed. This is because parent dir is resolved to "/" in SftpChangedExclusiveReadLockStrategy: line 81: files = operations.listFiles(file.getParent()); File absolute path is "test.xml" because leading slash is stripped away from the path, so in getParent the parent is resolved to "//", which will the be normalized in the methods last line to "/". So getParent method returns server root as an argument to listFiles method. I don't know what would be the best way to fix this, but I will continue to investigate it if I have the time... > SFTP: cannot get files from users home with readlock changed > > > Key: CAMEL-10782 > URL: https://issues.apache.org/jira/browse/CAMEL-10782 > Project: Camel > Issue Type: Bug > Components: camel-ftp >Affects Versions: 2.16.4, 2.18.2 > Environment: servicemix 7.0.0, Linux SFTP server >Reporter: Stefan Roos >Priority: Minor > Attachments: camel_2.18.2_sftp_problem_with_homedir > > > Camel cannot fetch files from users home directory when using > readlock=changed. > Route from URI is defined like this: > uri="sftp://testi@server:22?password=passwordinclude=PSOP_123-.*\.(xml|pdf)delete=truereadLock=changedreadLockTimeout=36readLockCheckInterval=18readLockMinAge=10smaxMessagesPerPoll=100"/> > And from TRACE logs I can see stuff like this: > 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Using full directory listing to update file information for > RemoteFile[PSOP_123-sdfsd.xml]. Consider enabling fastExistsCheck option. > 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 | SftpOperations > | 225 - org.apache.camel.camel-ftp - 2.16.4 | listFiles(/) > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | List files PSOP_123-sdfsd.xml found 23 files > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Previous last modified: 0, new last modified: 0 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Previous length: 0, new length: 0 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | New older than threshold: 1486112375150 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Exclusive read lock not granted. Sleeping for 2 millis. > User home directory contains only 1 file, which does match to include > parameter. It does seem, that camel-ftp is trying to check file readlock from > the server root. > When placing the files to a subdirectory in users home, this does not happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CAMEL-10782) SFTP: cannot get files from users home with readlock changed
[ https://issues.apache.org/jira/browse/CAMEL-10782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Roos updated CAMEL-10782: Affects Version/s: 2.18.2 > SFTP: cannot get files from users home with readlock changed > > > Key: CAMEL-10782 > URL: https://issues.apache.org/jira/browse/CAMEL-10782 > Project: Camel > Issue Type: Bug > Components: camel-ftp >Affects Versions: 2.16.4, 2.18.2 > Environment: servicemix 7.0.0, Linux SFTP server >Reporter: Stefan Roos >Priority: Minor > Attachments: camel_2.18.2_sftp_problem_with_homedir > > > Camel cannot fetch files from users home directory when using > readlock=changed. > Route from URI is defined like this: > uri="sftp://testi@server:22?password=passwordinclude=PSOP_123-.*\.(xml|pdf)delete=truereadLock=changedreadLockTimeout=36readLockCheckInterval=18readLockMinAge=10smaxMessagesPerPoll=100"/> > And from TRACE logs I can see stuff like this: > 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Using full directory listing to update file information for > RemoteFile[PSOP_123-sdfsd.xml]. Consider enabling fastExistsCheck option. > 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 | SftpOperations > | 225 - org.apache.camel.camel-ftp - 2.16.4 | listFiles(/) > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | List files PSOP_123-sdfsd.xml found 23 files > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Previous last modified: 0, new last modified: 0 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Previous length: 0, new length: 0 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | New older than threshold: 1486112375150 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Exclusive read lock not granted. Sleeping for 2 millis. > User home directory contains only 1 file, which does match to include > parameter. It does seem, that camel-ftp is trying to check file readlock from > the server root. > When placing the files to a subdirectory in users home, this does not happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CAMEL-10782) SFTP: cannot get files from users home with readlock changed
[ https://issues.apache.org/jira/browse/CAMEL-10782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Roos updated CAMEL-10782: Attachment: camel_2.18.2_sftp_problem_with_homedir Added full trace log from camel 2.18.2 session with similar incident. Notice how list file count changes after camel goes to sleeping (from 10 files to 23). > SFTP: cannot get files from users home with readlock changed > > > Key: CAMEL-10782 > URL: https://issues.apache.org/jira/browse/CAMEL-10782 > Project: Camel > Issue Type: Bug > Components: camel-ftp >Affects Versions: 2.16.4 > Environment: servicemix 7.0.0, Linux SFTP server >Reporter: Stefan Roos >Priority: Minor > Attachments: camel_2.18.2_sftp_problem_with_homedir > > > Camel cannot fetch files from users home directory when using > readlock=changed. > Route from URI is defined like this: > uri="sftp://testi@server:22?password=passwordinclude=PSOP_123-.*\.(xml|pdf)delete=truereadLock=changedreadLockTimeout=36readLockCheckInterval=18readLockMinAge=10smaxMessagesPerPoll=100"/> > And from TRACE logs I can see stuff like this: > 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Using full directory listing to update file information for > RemoteFile[PSOP_123-sdfsd.xml]. Consider enabling fastExistsCheck option. > 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 | SftpOperations > | 225 - org.apache.camel.camel-ftp - 2.16.4 | listFiles(/) > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | List files PSOP_123-sdfsd.xml found 23 files > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Previous last modified: 0, new last modified: 0 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Previous length: 0, new length: 0 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | New older than threshold: 1486112375150 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Exclusive read lock not granted. Sleeping for 2 millis. > User home directory contains only 1 file, which does match to include > parameter. It does seem, that camel-ftp is trying to check file readlock from > the server root. > When placing the files to a subdirectory in users home, this does not happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CAMEL-10782) SFTP: cannot get files from users home with readlock changed
[
https://issues.apache.org/jira/browse/CAMEL-10782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855457#comment-15855457
]
Stefan Roos commented on CAMEL-10782:
-
Yes its the same thing with Camel 2.18.2.
I created and started following route:
from("sftp://testi@infra:22/;
+"?password=passu"
+"=.*xml"
+"=true"
+"=changed"
+"=3"
+"=1"
+"=5s"
+"=100"
).routeId("SFTP-TEST-ROUTE")
.log("got file ${in.header.camelFileName}")
.to("file:sftp-files");
And then created a file with "cat > test.xml", now camel firts sees it as
empty, so it will sleep.
Then when trying to check it again:
[ main] DefaultCamelContextINFO Apache
Camel 2.18.2 (CamelContext: camel-1) started in 1.159 seconds
[ad #0 - sftp://testi@infra:22/] angedExclusiveReadLockStrategy WARN Cannot
acquire read lock within 3 millis. Will skip the file: RemoteFile[test.xml]
[ad #0 - sftp://testi@infra:22/] angedExclusiveReadLockStrategy WARN Cannot
acquire read lock within 3 millis. Will skip the file: RemoteFile[test.xml]
And after modifying the route by changing the directory from "/" to "/./":
from("sftp://testi@infra:22/./;
And starting it:
[ main] DefaultCamelContextINFO Apache
Camel 2.18.2 (CamelContext: camel-1) started in 1.060 seconds
[ #0 - sftp://testi@infra:22/./] SFTP-TEST-ROUTEINFO got file
test.xml
> SFTP: cannot get files from users home with readlock changed
>
>
> Key: CAMEL-10782
> URL: https://issues.apache.org/jira/browse/CAMEL-10782
> Project: Camel
> Issue Type: Bug
> Components: camel-ftp
>Affects Versions: 2.16.4
> Environment: servicemix 7.0.0, Linux SFTP server
>Reporter: Stefan Roos
>Priority: Minor
>
> Camel cannot fetch files from users home directory when using
> readlock=changed.
> Route from URI is defined like this:
> uri="sftp://testi@server:22?password=passwordinclude=PSOP_123-.*\.(xml|pdf)delete=truereadLock=changedreadLockTimeout=36readLockCheckInterval=18readLockMinAge=10smaxMessagesPerPoll=100"/>
> And from TRACE logs I can see stuff like this:
> 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 |
> ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4
> | Using full directory listing to update file information for
> RemoteFile[PSOP_123-sdfsd.xml]. Consider enabling fastExistsCheck option.
> 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 | SftpOperations
> | 225 - org.apache.camel.camel-ftp - 2.16.4 | listFiles(/)
> 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 |
> ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4
> | List files PSOP_123-sdfsd.xml found 23 files
> 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 |
> ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4
> | Previous last modified: 0, new last modified: 0
> 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 |
> ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4
> | Previous length: 0, new length: 0
> 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 |
> ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4
> | New older than threshold: 1486112375150
> 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 |
> ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4
> | Exclusive read lock not granted. Sleeping for 2 millis.
> User home directory contains only 1 file, which does match to include
> parameter. It does seem, that camel-ftp is trying to check file readlock from
> the server root.
> When placing the files to a subdirectory in users home, this does not happen.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CAMEL-10782) SFTP: cannot get files from users home with readlock changed
[ https://issues.apache.org/jira/browse/CAMEL-10782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851411#comment-15851411 ] Stefan Roos commented on CAMEL-10782: - What was apparrently happening was that, when file was created it was zero in length. So camel-ftp went sleeping for short time, because it could not get readlock (minsize 1). After that it tried to reCheck the file, but for some reason from the server root, / , and not from users home, /home/testi/. File count which was reported by camel-ftp, matches the count from that servers root (home dir count would be less than 10 files/dirs). I found a way around this by setting source directory to "./" like this: ftp://testi@infra:22/./?password=passu;. With that kind of from URI it does seem to work normally. > SFTP: cannot get files from users home with readlock changed > > > Key: CAMEL-10782 > URL: https://issues.apache.org/jira/browse/CAMEL-10782 > Project: Camel > Issue Type: Bug > Components: camel-ftp >Affects Versions: 2.16.4 > Environment: servicemix 7.0.0, Linux SFTP server >Reporter: Stefan Roos >Priority: Minor > > Camel cannot fetch files from users home directory when using > readlock=changed. > Route from URI is defined like this: > uri="sftp://testi@server:22?password=passwordinclude=PSOP_123-.*\.(xml|pdf)delete=truereadLock=changedreadLockTimeout=36readLockCheckInterval=18readLockMinAge=10smaxMessagesPerPoll=100"/> > And from TRACE logs I can see stuff like this: > 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Using full directory listing to update file information for > RemoteFile[PSOP_123-sdfsd.xml]. Consider enabling fastExistsCheck option. > 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 | SftpOperations > | 225 - org.apache.camel.camel-ftp - 2.16.4 | listFiles(/) > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | List files PSOP_123-sdfsd.xml found 23 files > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Previous last modified: 0, new last modified: 0 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Previous length: 0, new length: 0 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | New older than threshold: 1486112375150 > 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | > ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 > | Exclusive read lock not granted. Sleeping for 2 millis. > User home directory contains only 1 file, which does match to include > parameter. It does seem, that camel-ftp is trying to check file readlock from > the server root. > When placing the files to a subdirectory in users home, this does not happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CAMEL-10782) SFTP: cannot get files from users home with readlock changed
Stefan Roos created CAMEL-10782: --- Summary: SFTP: cannot get files from users home with readlock changed Key: CAMEL-10782 URL: https://issues.apache.org/jira/browse/CAMEL-10782 Project: Camel Issue Type: Bug Components: camel-ftp Affects Versions: 2.16.4 Environment: servicemix 7.0.0, Linux SFTP server Reporter: Stefan Roos Priority: Minor Camel cannot fetch files from users home directory when using readlock=changed. Route from URI is defined like this: ftp://testi@server:22?password=passwordinclude=PSOP_123-.*\.(xml|pdf)delete=truereadLock=changedreadLockTimeout=36readLockCheckInterval=18readLockMinAge=10smaxMessagesPerPoll=100"/> And from TRACE logs I can see stuff like this: 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 | ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 | Using full directory listing to update file information for RemoteFile[PSOP_123-sdfsd.xml]. Consider enabling fastExistsCheck option. 2017-02-03 10:59:45,146 | TRACE | //testi@infra:22 | SftpOperations | 225 - org.apache.camel.camel-ftp - 2.16.4 | listFiles(/) 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 | List files PSOP_123-sdfsd.xml found 23 files 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 | Previous last modified: 0, new last modified: 0 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 | Previous length: 0, new length: 0 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 | New older than threshold: 1486112375150 2017-02-03 10:59:45,150 | TRACE | //testi@infra:22 | ChangedExclusiveReadLockStrategy | 225 - org.apache.camel.camel-ftp - 2.16.4 | Exclusive read lock not granted. Sleeping for 2 millis. User home directory contains only 1 file, which does match to include parameter. It does seem, that camel-ftp is trying to check file readlock from the server root. When placing the files to a subdirectory in users home, this does not happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
