[lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
Hi,
I am trying to mirror files using lftp, I have tested a use case where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully at the
destination, now if I update a.txt append some content and save it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_ behind. And
if i run the mirror multiple times it always downloads the changed a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they are
updated at the source. Like if a.txt is changed at source then lftp mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be
saved with a new time stamp. And mirror will consider these file to decide
whether to download or not.
Thanks,
Akshay
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
Can you provide a debug log? Why lftp cannot replace the old file?
2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com:
Hi,
I am trying to mirror files using lftp, I have tested a use case where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully at
the destination, now if I update a.txt append some content and save it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_ behind. And
if i run the mirror multiple times it always downloads the changed a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they are
updated at the source. Like if a.txt is changed at source then lftp mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be
saved with a new time stamp. And mirror will consider these file to decide
whether to download or not.
Thanks,
Akshay
--
Alexander.
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
Here are the debug logs test.txt is the file that is changed:
lftp akshaygupta@192.168.128.124:~/Desktop/lftp mirror
--- TYPE A
--- 200 Type set to A.
--- PASV
--- 227 Entering Passive Mode (192,168,128,124,228,174)
Connecting data socket to (192.168.128.124) port 58542
Data connection established
--- LIST
--- 150 Opening ASCII mode data connection for '/bin/ls'.
Got EOF on data connection
Closing data socket
--- 226 Transfer complete.
--- TYPE I
--- 200 Type set to I.
--- MDTM .DS_Store
--- 213 20141027064918
--- MDTM search-2.html
--- 213 2014081726
--- MDTM search-3.html
--- 213 2014081744
--- MDTM search-4.html
--- 213 2014081745
--- MDTM search.html
--- 213 2014081722
--- MDTM settings.xml
--- 213 20140820095244
--- MDTM test.txt
--- 213 20141027075438
--- PASV
--- 227 Entering Passive Mode (192,168,128,124,228,176)
Connecting data socket to (192.168.128.124) port 58544
Data connection established
--- RETR test.txt
--- 150 Opening BINARY mode data connection for 'test.txt' (202 bytes).
Got EOF on data connection
Closing data socket
--- 226 Transfer complete.
Total: 1 directory, 7 files, 0 symlinks
Modified: 1 file, 0 symlinks
202 bytes transferred
To be removed: 0 directories, 1 file, 0 symlinks
Closing idle connection
--- QUIT
Timeout - reconnecting
--- 221-
--- Data traffic for this session was 172301 bytes in 7 files.
--- Total traffic for this session was 181339 bytes in 13 transfers.
--- 221 Thank you for using the FTP service on 192.168.128.124.
Closing control socket
~Akshay
On Mon, Oct 27, 2014 at 1:18 PM, Alexander Lukyanov lavv...@gmail.com
wrote:
Can you provide a debug log? Why lftp cannot replace the old file?
2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com:
Hi,
I am trying to mirror files using lftp, I have tested a use case where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully at
the destination, now if I update a.txt append some content and save it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_ behind. And
if i run the mirror multiple times it always downloads the changed a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they are
updated at the source. Like if a.txt is changed at source then lftp mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be
saved with a new time stamp. And mirror will consider these file to decide
whether to download or not.
Thanks,
Akshay
--
Alexander.
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
On Mon, Oct 27, 2014 at 01:30:42PM +0530, akshay gupta wrote:
--- RETR test.txt
--- 150 Opening BINARY mode data connection for 'test.txt' (202 bytes).
Got EOF on data connection
Closing data socket
--- 226 Transfer complete.
So the file cannot be renamed locally, right? Is there a permissions problem?
--
Alexander.
On Mon, Oct 27, 2014 at 1:18 PM, Alexander Lukyanov lavv...@gmail.com
wrote:
Can you provide a debug log? Why lftp cannot replace the old file?
2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com:
Hi,
I am trying to mirror files using lftp, I have tested a use case where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully at
the destination, now if I update a.txt append some content and save it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_ behind. And
if i run the mirror multiple times it always downloads the changed a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they are
updated at the source. Like if a.txt is changed at source then lftp mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be
saved with a new time stamp. And mirror will consider these file to decide
whether to download or not.
Thanks,
Akshay
--
Alexander.
___
lftp mailing list
l...@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
no, I am running as a root user. There is some issue with lftp, when file
with same name already exist its not able to rename the temp file to actual
file.
On Mon, Oct 27, 2014 at 1:44 PM, Alexander V. Lukyanov l...@netis.ru wrote:
On Mon, Oct 27, 2014 at 01:30:42PM +0530, akshay gupta wrote:
--- RETR test.txt
--- 150 Opening BINARY mode data connection for 'test.txt' (202 bytes).
Got EOF on data connection
Closing data socket
--- 226 Transfer complete.
So the file cannot be renamed locally, right? Is there a permissions
problem?
--
Alexander.
On Mon, Oct 27, 2014 at 1:18 PM, Alexander Lukyanov lavv...@gmail.com
wrote:
Can you provide a debug log? Why lftp cannot replace the old file?
2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com:
Hi,
I am trying to mirror files using lftp, I have tested a use case
where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully
at
the destination, now if I update a.txt append some content and save
it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_
behind. And
if i run the mirror multiple times it always downloads the changed
a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they
are
updated at the source. Like if a.txt is changed at source then lftp
mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it
will be
saved with a new time stamp. And mirror will consider these file to
decide
whether to download or not.
Thanks,
Akshay
--
Alexander.
___
lftp mailing list
l...@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
I have found the bug. As a workaround, use set xfer:clobber yes.
2014-10-27 13:06 GMT+03:00 akshay gupta akshaygupta...@gmail.com:
no, I am running as a root user. There is some issue with lftp, when file
with same name already exist its not able to rename the temp file to actual
file.
On Mon, Oct 27, 2014 at 1:44 PM, Alexander V. Lukyanov l...@netis.ru
wrote:
On Mon, Oct 27, 2014 at 01:30:42PM +0530, akshay gupta wrote:
--- RETR test.txt
--- 150 Opening BINARY mode data connection for 'test.txt' (202 bytes).
Got EOF on data connection
Closing data socket
--- 226 Transfer complete.
So the file cannot be renamed locally, right? Is there a permissions
problem?
--
Alexander.
On Mon, Oct 27, 2014 at 1:18 PM, Alexander Lukyanov lavv...@gmail.com
wrote:
Can you provide a debug log? Why lftp cannot replace the old file?
2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com:
Hi,
I am trying to mirror files using lftp, I have tested a use case
where a
file at the source machine gets modified but the file name remains
the
same. After mirroring, lftp is unable to replace the old file and
also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully
at
the destination, now if I update a.txt append some content and save
it with
same name. If I run lftp mirror again then its not able to replace
the
existing a.txt but it also leaves the temp file a.txt_transit_
behind. And
if i run the mirror multiple times it always downloads the changed
a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they
are
updated at the source. Like if a.txt is changed at source then lftp
mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it
will be
saved with a new time stamp. And mirror will consider these file to
decide
whether to download or not.
Thanks,
Akshay
--
Alexander.
___
lftp mailing list
l...@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp
--
Alexander.
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
now its producing 0 byte files and i am also getting input/output error here are the debug logs: lftp akshaygupta@192.168.128.124:~/Desktop/lftp mirror --ignore-time --- PASV --- 227 Entering Passive Mode (192,168,128,124,248,18) Connecting data socket to (192.168.128.124) port 63506 Data connection established --- LIST --- 150 Opening ASCII mode data connection for '/bin/ls'. Got EOF on data connection Closing data socket --- 226 Transfer complete. --- TYPE I --- 200 Type set to I. --- PASV --- 227 Entering Passive Mode (192,168,128,124,248,19) Connecting data socket to (192.168.128.124) port 63507 Data connection established --- RETR test.txt --- 150 Opening BINARY mode data connection for 'test.txt' (240 bytes). Got EOF on data connection Closing data socket --- 226 Transfer complete. copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,248,51) Connecting data socket to (192.168.128.124) port 63539 Data connection established --- RETR .DS_Store --- 150 Opening BINARY mode data connection for '.DS_Store' (6148 bytes). Got EOF on data connection Closing data socket --- 226 Transfer complete. copy: renaming `/mount/user/tom/.DS_Store_tmp_' to `.DS_Store' mirror: rename(/mount/user/tom/.DS_Store_tmp_, /mount/user/tom/.DS_Store): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,248,131) Connecting data socket to (192.168.128.124) port 63619 Data connection established --- RETR settings.xml --- 150 Opening BINARY mode data connection for 'settings.xml' (1737 bytes). --- 226 Transfer complete. Got EOF on data connection Closing data socket copy: renaming `/mount/user/tom/settings.xml_tmp_' to `settings.xml' mirror: rename(/mount/user/tom/settings.xml_tmp_, /mount/user/tom/settings.xml): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,248,178) Connecting data socket to (192.168.128.124) port 63666 Data connection established --- RETR search-2.html --- 150 Opening BINARY mode data connection for 'search-2.html' (42518 bytes). --- 226 Transfer complete. Got EOF on data connection Closing data socket copy: renaming `/mount/user/tom/search-2.html_tmp_' to `search-2.html' mirror: rename(/mount/user/tom/search-2.html_tmp_, /mount/user/tom/search-2.html): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,249,146) Connecting data socket to (192.168.128.124) port 63890 Data connection established --- RETR search-3.html --- 150 Opening BINARY mode data connection for 'search-3.html' (42569 bytes). --- 226 Transfer complete. Got EOF on data connection Closing data socket copy: renaming `/mount/user/tom/search-3.html_tmp_' to `search-3.html' mirror: rename(/mount/user/tom/search-3.html_tmp_, /mount/user/tom/search-3.html): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,250,160) Connecting data socket to (192.168.128.124) port 64160 Data connection established --- RETR search-4.html --- 150 Opening BINARY mode data connection for 'search-4.html' (42569 bytes). --- 226 Transfer complete. Got EOF on data connection Closing data socket copy: renaming `/mount/user/tom/search-4.html_tmp_' to `search-4.html' mirror: rename(/mount/user/tom/search-4.html_tmp_, /mount/user/tom/search-4.html): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,251,142) Connecting data socket to (192.168.128.124) port 64398 Data connection established --- RETR search.html --- 150 Opening BINARY mode data connection for 'search.html' (42519 bytes). --- 226 Transfer complete. Got EOF on data connection Closing data socket copy: renaming `/mount/user/tom/search.html_tmp_' to `search.html' mirror: rename(/mount/user/tom/search.html_tmp_, /mount/user/tom/search.html): Input/output error Total: 1 directory, 7 files, 0 symlinks Modified: 7 files, 0 symlinks 178300 bytes transferred in 47 seconds (3.7 KiB/s) To be removed: 0 directories, 7 files, 0 symlinks 7 errors detected On Mon, Oct 27, 2014 at 5:01 PM, Alexander Lukyanov lavv...@gmail.com wrote: This change should fix the problem. https://github.com/lavv17/lftp/commit/b1164d695393cf8ec8429bb2b5caa9f05cfc4d39 2014-10-27 14:20 GMT+03:00 Alexander Lukyanov lavv...@gmail.com: I have found the bug. As a workaround, use set xfer:clobber yes. 2014-10-27 13:06 GMT+03:00 akshay gupta akshaygupta...@gmail.com: no, I am running as a root user. There is some issue with lftp, when file with same name already exist its not able to rename the temp file to actual file. On Mon, Oct 27, 2014 at 1:44 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Mon, Oct 27, 2014 at 01:30:42PM +0530, akshay gupta wrote: --- RETR test.txt --- 150 Opening BINARY
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
On Mon, Oct 27, 2014 at 05:53:19PM +0530, akshay gupta wrote: now its producing 0 byte files and i am also getting input/output error here are the debug logs: copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt): Input/output error Which filesystem do you use? Maybe it is necessary to unlink target file before renaming, although rename should do it automatically. If newpath already exists it will be atomically replaced (subject to a few conditions; see ERRORS below), so that there is no point at which another process attempting to access newpath will find it missing. And man page for rename(2) does not list EIO error. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
I am using fuse mounted hadoop distributed file system(HDFS). Can you suggest what changes need to be done to make it work as this is a very critical requirement for my project. Will deleting file and then renaming temp file help? how to achieve that? On Mon, Oct 27, 2014 at 7:38 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Mon, Oct 27, 2014 at 05:53:19PM +0530, akshay gupta wrote: now its producing 0 byte files and i am also getting input/output error here are the debug logs: copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt): Input/output error Which filesystem do you use? Maybe it is necessary to unlink target file before renaming, although rename should do it automatically. If newpath already exists it will be atomically replaced (subject to a few conditions; see ERRORS below), so that there is no point at which another process attempting to access newpath will find it missing. And man page for rename(2) does not list EIO error. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
