[lftp-devel] Handling for small files
Hi, I am trying to transfer about 80GB of data using mirror option. File sizes are very small about 0.5 KB to 3 KB per file. Transfer is taking a lot of time which I am attributing to small file size. Is there any option in lftp which I can use to increase the transfer speed. May be some sort of compression option like rsync -av*z.* I know about the parallel file transfer option but I cant use it because I want to maintain the order of file transfer. Please help. Thanks, Akshay ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Handling for small files
I am using fish protocol. On Tue, Feb 24, 2015, 4:23 PM Alexander Lukyanov lavv...@gmail.com wrote: Which protocol do you use? вт, 24 февр. 2015, 13:48, akshay gupta akshaygupta...@gmail.com: Hi, I am trying to transfer about 80GB of data using mirror option. File sizes are very small about 0.5 KB to 3 KB per file. Transfer is taking a lot of time which I am attributing to small file size. Is there any option in lftp which I can use to increase the transfer speed. May be some sort of compression option like rsync -av*z.* I know about the parallel file transfer option but I cant use it because I want to maintain the order of file transfer. Please help. Thanks, Akshay ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
Its not working: I run the ./autogen.sh --without-gnutls --with-openssl command and it cannot seem to find Makefile.in in the configure step. Attached the complete logs On Thu, Dec 4, 2014 at 8:54 PM, Alexander Lukyanov lavv...@gmail.com wrote: autogen.sh script should generate everything. You'll have to get gnulib too. четверг, 4 декабря 2014 г. пользователь akshay gupta написал: I have cloned the lftp repo, can you please tell me the steps to create configure file and make script. I tried automake --force-missing --add-missing but its giving src/Makefile.am:45: NEED_TRIO does not appear in AM_CONDITIONAL src/Makefile.am: installing `build-aux/depcomp' Makefile.am:4: NEED_TRIO does not appear in AM_CONDITIONAL configure.ac:607: required file `lib/Makefile.in' not found ~ Akshay On Thu, Dec 4, 2014 at 4:56 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Wed, Dec 03, 2014 at 02:11:28PM +0400, Alexander Lukyanov wrote: It's almost done, needs to be tested and also some issues with sorting to be fixed. 2014-12-03 10:26 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Are these feature present in LFTP implementation timeline? By when will they be implemented? You can now test the feature (git ee6391e adds --scan-all-first option). https://github.com/lavv17/lftp/commit/ee6391e3213e12013894aa98508a1a151f542da0 -- Alexander. -- Alexander. [root@localhost lftp]# ./autogen.sh --without-gnutls --with-openssl processing . Creating ./aclocal.m4 ... Running gettextize... Ignore non-fatal messages. Not copying intl/ directory. Copying file po/Makefile.in.in Copying file po/boldquot.sed Copying file po/en@boldquot.header Copying file po/en@quot.header Copying file po/insert-header.sin Copying file po/Makevars.template Copying file po/quot.sed Copying file po/remove-potcdate.sin Copying file po/Rules-quot perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_CTYPE = UTF-8, LANG = en_US.UTF-8 are supported and installed on your system. perl: warning: Falling back to the standard locale (C). perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_CTYPE = UTF-8, LANG = en_US.UTF-8 are supported and installed on your system. perl: warning: Falling back to the standard locale (C). Copying file m4/gettext.m4 cp: cannot stat `/usr/share/aclocal/gettext.m4': No such file or directory Copying file m4/iconv.m4 cp: cannot stat `/usr/share/aclocal/iconv.m4': No such file or directory Copying file m4/lib-ld.m4 cp: cannot stat `/usr/share/aclocal/lib-ld.m4': No such file or directory Copying file m4/lib-link.m4 cp: cannot stat `/usr/share/aclocal/lib-link.m4': No such file or directory Copying file m4/lib-prefix.m4 cp: cannot stat `/usr/share/aclocal/lib-prefix.m4': No such file or directory Copying file m4/nls.m4 cp: cannot stat `/usr/share/aclocal/nls.m4': No such file or directory Copying file m4/po.m4 cp: cannot stat `/usr/share/aclocal/po.m4': No such file or directory Copying file m4/progtest.m4 cp: cannot stat `/usr/share/aclocal/progtest.m4': No such file or directory Please update po/Makevars so that it defines all the variables mentioned in po/Makevars.template. You can then remove po/Makevars.template. Please run 'aclocal -I m4' to regenerate the aclocal.m4 file. You need aclocal from GNU automake 1.9 (or newer) to do this. Then run 'autoconf' to regenerate the configure file. You might also want to copy the convenience header file gettext.h from the /usr/share/gettext directory into your package. It is a wrapper around libintl.h that implements the configure --disable-nls option. Press Return to acknowledge the previous three paragraphs. mv: cannot stat `configure.ac~': No such file or directory mv: cannot stat `m4/Makefile.am~': No such file or directory Making ./aclocal.m4 writable ... Running libtoolize... libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `build-aux'. libtoolize: copying file `build-aux/ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'. libtoolize: copying file `m4/libtool.m4' libtoolize: copying file `m4/ltoptions.m4' libtoolize: copying file `m4/ltsugar.m4' libtoolize: copying file `m4/ltversion.m4' libtoolize: copying file `m4/lt~obsolete.m4' mv: cannot stat `Makefile.am~': No such file or directory Module list with included dependencies (indented): absolute-header alloca alloca-opt argmatch arpa_inet btowc c-ctype c-strcase c-strcaseeq chown clock-time cloexec close closedir configmake crypto/md5 crypto/sha1 d-type dirent dirfd dosname dup2 environ errno error exitfail extensions extern-inline fcntl fcntl-h fd-hook filemode filename
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
Have done some initial high level testing. It seems to work. Will test it thoroughly and will get back to you in case I find any bug. On Fri, Dec 5, 2014 at 10:20 PM, Alexander Lukyanov lavv...@gmail.com wrote: I think you need to install gettext-devel. I have also made a snapshot in case you don't want to build from git: http://lftp.yar.ru/ftp/devel/lftp-4.6.0.20141205.tar.gz 2014-12-05 9:43 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Its not working: I run the ./autogen.sh --without-gnutls --with-openssl command and it cannot seem to find Makefile.in in the configure step. Attached the complete logs On Thu, Dec 4, 2014 at 8:54 PM, Alexander Lukyanov lavv...@gmail.com wrote: autogen.sh script should generate everything. You'll have to get gnulib too. четверг, 4 декабря 2014 г. пользователь akshay gupta написал: I have cloned the lftp repo, can you please tell me the steps to create configure file and make script. I tried automake --force-missing --add-missing but its giving src/Makefile.am:45: NEED_TRIO does not appear in AM_CONDITIONAL src/Makefile.am: installing `build-aux/depcomp' Makefile.am:4: NEED_TRIO does not appear in AM_CONDITIONAL configure.ac:607: required file `lib/Makefile.in' not found ~ Akshay On Thu, Dec 4, 2014 at 4:56 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Wed, Dec 03, 2014 at 02:11:28PM +0400, Alexander Lukyanov wrote: It's almost done, needs to be tested and also some issues with sorting to be fixed. 2014-12-03 10:26 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Are these feature present in LFTP implementation timeline? By when will they be implemented? You can now test the feature (git ee6391e adds --scan-all-first option). https://github.com/lavv17/lftp/commit/ee6391e3213e12013894aa98508a1a151f542da0 -- Alexander. -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Compressing data at destination machine
I have tried using it. It does not update the files appearing in the mount point if the files in zip changes. And it also does not retains modification time of the file at the mount location this could affect the mirror. Do you have any other similar opensource software in mind that could also support writes. Thanks, Akshay On Fri, Nov 28, 2014 at 1:06 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Fri, Nov 28, 2014 at 11:55:07AM +0530, akshay gupta wrote: I dont want lftp to extract data from archives, I want it to compress it into archive so that it occupies less space at the destination side. Is this feasible to do in lftp? Take a look at zipfs: https://github.com/hanwen/go-fuse Unfortunately it is read-only. So you can mount a zip file, run mirror over it with --script option, run the script with an empty target directory, then run zip to update the archive. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
Are these feature present in LFTP implementation timeline? By when will they be implemented? ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Compressing data at destination machine
I dont want lftp to extract data from archives, I want it to compress it into archive so that it occupies less space at the destination side. Is this feasible to do in lftp? On Wed, Nov 26, 2014 at 11:33 AM, Alexander V. Lukyanov l...@netis.ru wrote: On Tue, Nov 25, 2014 at 06:44:32PM +0530, akshay gupta wrote: I want to store the data that is being pulled by lftp mirror job in a compressed format. Is it possible that mirror function can check file download status from inside of that tar, zip etc. Is it possible to implement such a functionality in lftp? I don't think mirror will be able to extract compressed archives, but you can download the archive and extract it at the same time, like this: lftp cat file.tar.gz | tar xv -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
[lftp-devel] Compressing data at destination machine
Hi, I want to store the data that is being pulled by lftp mirror job in a compressed format. Is it possible that mirror function can check file download status from inside of that tar, zip etc. Is it possible to implement such a functionality in lftp? Thanks, Akshay ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
We are in desperate need of this feature as without this feature we can only ensure file order within a folder. There is a huge dependency on file order for some of our hadoop oozie jobs. By when can we expect this feature to come in lftp? On Wed, Nov 12, 2014 at 1:54 PM, Alexander Lukyanov lavv...@gmail.com wrote: I think it is possible and the amount of programming work is moderate. The recursive model of mirror will stay but the file transfers will be deferred and done on the top level when recursive directory traversal is finished. 2014-11-12 8:32 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi Alexander, Are there any plans for redesigning LFTP Mirror algorithm to build the whole recursive list first. How much change would that require if we go that path? Thanks, Akshay On Thu, Oct 30, 2014 at 4:44 PM, akshay gupta akshaygupta...@gmail.com wrote: lftp have options like --depth-first and --no-recursion I think it will make sense to have an option like --iterative for iterative file transfer. This option will fetch the full file list and then fetch them iteratively. On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta akshaygupta...@gmail.com wrote: I think the functionality of mirror:sort-by option implies that the order is for all the files irrespective of the directory structure. Or may be it should support both the options and it would be configuration defined which one to use. On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov lavv...@gmail.com wrote: Mirror in lftp works directory by directory. It does not build full list of files first, so it cannot sort the whole list. Maybe the mirror algorithm should be redesigned to build the whole recursive list first. What do you think? 2014-10-29 10:07 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using mirror:sort-by option set to 'date-desc'. My file structure is like this $ ls -l -R total 3191264 drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv ./1: total 0 drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 ./1/2: total 515512 -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv I expect the file synch order should be ~/tmp.txt ~/tmp1.txt ~/abc.csv ~/1/2/bc.csv ~/1/2/3 but the synch order is: ~/tmp.txt ~/tmp1.txt ~/1/2/bc.csv ~/1/2/3 ~/abc.csv I think its the folder modification time that is causing this issue. Lftp considers folder synch time to decide the order even though files contained in the folder are newer then the remaining files in the parent folder. File synch order should be in order of the modification time irrespective of the folder structure. Please help. Thanks, Akshay -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
lftp have options like --depth-first and --no-recursion I think it will make sense to have an option like --iterative for iterative file transfer. This option will fetch the full file list and then fetch them iteratively. On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta akshaygupta...@gmail.com wrote: I think the functionality of mirror:sort-by option implies that the order is for all the files irrespective of the directory structure. Or may be it should support both the options and it would be configuration defined which one to use. On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov lavv...@gmail.com wrote: Mirror in lftp works directory by directory. It does not build full list of files first, so it cannot sort the whole list. Maybe the mirror algorithm should be redesigned to build the whole recursive list first. What do you think? 2014-10-29 10:07 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using mirror:sort-by option set to 'date-desc'. My file structure is like this $ ls -l -R total 3191264 drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv ./1: total 0 drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 ./1/2: total 515512 -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv I expect the file synch order should be ~/tmp.txt ~/tmp1.txt ~/abc.csv ~/1/2/bc.csv ~/1/2/3 but the synch order is: ~/tmp.txt ~/tmp1.txt ~/1/2/bc.csv ~/1/2/3 ~/abc.csv I think its the folder modification time that is causing this issue. Lftp considers folder synch time to decide the order even though files contained in the folder are newer then the remaining files in the parent folder. File synch order should be in order of the modification time irrespective of the folder structure. Please help. Thanks, Akshay -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
[lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
Hi, I am trying to mirror files using mirror:sort-by option set to 'date-desc'. My file structure is like this $ ls -l -R total 3191264 drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv ./1: total 0 drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 ./1/2: total 515512 -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv I expect the file synch order should be ~/tmp.txt ~/tmp1.txt ~/abc.csv ~/1/2/bc.csv ~/1/2/3 but the synch order is: ~/tmp.txt ~/tmp1.txt ~/1/2/bc.csv ~/1/2/3 ~/abc.csv I think its the folder modification time that is causing this issue. Lftp considers folder synch time to decide the order even though files contained in the folder are newer then the remaining files in the parent folder. File synch order should be in order of the modification time irrespective of the folder structure. Please help. Thanks, Akshay ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
I think the functionality of mirror:sort-by option implies that the order is for all the files irrespective of the directory structure. Or may be it should support both the options and it would be configuration defined which one to use. On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov lavv...@gmail.com wrote: Mirror in lftp works directory by directory. It does not build full list of files first, so it cannot sort the whole list. Maybe the mirror algorithm should be redesigned to build the whole recursive list first. What do you think? 2014-10-29 10:07 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using mirror:sort-by option set to 'date-desc'. My file structure is like this $ ls -l -R total 3191264 drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv ./1: total 0 drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 ./1/2: total 515512 -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv I expect the file synch order should be ~/tmp.txt ~/tmp1.txt ~/abc.csv ~/1/2/bc.csv ~/1/2/3 but the synch order is: ~/tmp.txt ~/tmp1.txt ~/1/2/bc.csv ~/1/2/3 ~/abc.csv I think its the folder modification time that is causing this issue. Lftp considers folder synch time to decide the order even though files contained in the folder are newer then the remaining files in the parent folder. File synch order should be in order of the modification time irrespective of the folder structure. Please help. Thanks, Akshay -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
will giving --delete-first option in lftp mirror help in my case? I think its not working in case of use-temp-file option. As per the man page its suppose to delete the old version of the file before it start downloading the new one. Its not doing anything. Please help. On Mon, Oct 27, 2014 at 11:50 PM, akshay gupta akshaygupta...@gmail.com wrote: I am using fuse mounted hadoop distributed file system(HDFS). Can you suggest what changes need to be done to make it work as this is a very critical requirement for my project. Will deleting file and then renaming temp file help? how to achieve that? On Mon, Oct 27, 2014 at 7:38 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Mon, Oct 27, 2014 at 05:53:19PM +0530, akshay gupta wrote: now its producing 0 byte files and i am also getting input/output error here are the debug logs: copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt): Input/output error Which filesystem do you use? Maybe it is necessary to unlink target file before renaming, although rename should do it automatically. If newpath already exists it will be atomically replaced (subject to a few conditions; see ERRORS below), so that there is no point at which another process attempting to access newpath will find it missing. And man page for rename(2) does not list EIO error. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
Thanks for the patch Alexander, files seems to be getting replaced now. Ill do some more testing to make sure every thing works perfectly. Thanks again, Akshay On Tue, Oct 28, 2014 at 3:21 PM, Alexander Lukyanov lavv...@gmail.com wrote: BTW, it looks like mirror should remove the old file before transferring the new one, please run mirror -vvv to check that. You can also run strace to check system call sequence. 2014-10-28 11:18 GMT+03:00 Alexander Lukyanov lavv...@gmail.com: It looks like a bug in the FS (fuse and/or hdfs) which also hits rsync: https://groups.google.com/a/cloudera.org/forum/#!msg/cdh-user/R1GR2Asj_lc/f_uuIjG5R1UJ Please try this workaround. 2014-10-28 10:29 GMT+03:00 akshay gupta akshaygupta...@gmail.com: will giving --delete-first option in lftp mirror help in my case? I think its not working in case of use-temp-file option. As per the man page its suppose to delete the old version of the file before it start downloading the new one. Its not doing anything. Please help. On Mon, Oct 27, 2014 at 11:50 PM, akshay gupta akshaygupta...@gmail.com wrote: I am using fuse mounted hadoop distributed file system(HDFS). Can you suggest what changes need to be done to make it work as this is a very critical requirement for my project. Will deleting file and then renaming temp file help? how to achieve that? On Mon, Oct 27, 2014 at 7:38 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Mon, Oct 27, 2014 at 05:53:19PM +0530, akshay gupta wrote: now its producing 0 byte files and i am also getting input/output error here are the debug logs: copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt): Input/output error Which filesystem do you use? Maybe it is necessary to unlink target file before renaming, although rename should do it automatically. If newpath already exists it will be atomically replaced (subject to a few conditions; see ERRORS below), so that there is no point at which another process attempting to access newpath will find it missing. And man page for rename(2) does not list EIO error. -- Alexander. -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
[lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
Hi,
I am trying to mirror files using lftp, I have tested a use case where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully at the
destination, now if I update a.txt append some content and save it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_ behind. And
if i run the mirror multiple times it always downloads the changed a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they are
updated at the source. Like if a.txt is changed at source then lftp mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be
saved with a new time stamp. And mirror will consider these file to decide
whether to download or not.
Thanks,
Akshay
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
Here are the debug logs test.txt is the file that is changed:
lftp akshaygupta@192.168.128.124:~/Desktop/lftp mirror
--- TYPE A
--- 200 Type set to A.
--- PASV
--- 227 Entering Passive Mode (192,168,128,124,228,174)
Connecting data socket to (192.168.128.124) port 58542
Data connection established
--- LIST
--- 150 Opening ASCII mode data connection for '/bin/ls'.
Got EOF on data connection
Closing data socket
--- 226 Transfer complete.
--- TYPE I
--- 200 Type set to I.
--- MDTM .DS_Store
--- 213 20141027064918
--- MDTM search-2.html
--- 213 2014081726
--- MDTM search-3.html
--- 213 2014081744
--- MDTM search-4.html
--- 213 2014081745
--- MDTM search.html
--- 213 2014081722
--- MDTM settings.xml
--- 213 20140820095244
--- MDTM test.txt
--- 213 20141027075438
--- PASV
--- 227 Entering Passive Mode (192,168,128,124,228,176)
Connecting data socket to (192.168.128.124) port 58544
Data connection established
--- RETR test.txt
--- 150 Opening BINARY mode data connection for 'test.txt' (202 bytes).
Got EOF on data connection
Closing data socket
--- 226 Transfer complete.
Total: 1 directory, 7 files, 0 symlinks
Modified: 1 file, 0 symlinks
202 bytes transferred
To be removed: 0 directories, 1 file, 0 symlinks
Closing idle connection
--- QUIT
Timeout - reconnecting
--- 221-
--- Data traffic for this session was 172301 bytes in 7 files.
--- Total traffic for this session was 181339 bytes in 13 transfers.
--- 221 Thank you for using the FTP service on 192.168.128.124.
Closing control socket
~Akshay
On Mon, Oct 27, 2014 at 1:18 PM, Alexander Lukyanov lavv...@gmail.com
wrote:
Can you provide a debug log? Why lftp cannot replace the old file?
2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com:
Hi,
I am trying to mirror files using lftp, I have tested a use case where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully at
the destination, now if I update a.txt append some content and save it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_ behind. And
if i run the mirror multiple times it always downloads the changed a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they are
updated at the source. Like if a.txt is changed at source then lftp mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be
saved with a new time stamp. And mirror will consider these file to decide
whether to download or not.
Thanks,
Akshay
--
Alexander.
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
no, I am running as a root user. There is some issue with lftp, when file
with same name already exist its not able to rename the temp file to actual
file.
On Mon, Oct 27, 2014 at 1:44 PM, Alexander V. Lukyanov l...@netis.ru wrote:
On Mon, Oct 27, 2014 at 01:30:42PM +0530, akshay gupta wrote:
--- RETR test.txt
--- 150 Opening BINARY mode data connection for 'test.txt' (202 bytes).
Got EOF on data connection
Closing data socket
--- 226 Transfer complete.
So the file cannot be renamed locally, right? Is there a permissions
problem?
--
Alexander.
On Mon, Oct 27, 2014 at 1:18 PM, Alexander Lukyanov lavv...@gmail.com
wrote:
Can you provide a debug log? Why lftp cannot replace the old file?
2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com:
Hi,
I am trying to mirror files using lftp, I have tested a use case
where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully
at
the destination, now if I update a.txt append some content and save
it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_
behind. And
if i run the mirror multiple times it always downloads the changed
a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they
are
updated at the source. Like if a.txt is changed at source then lftp
mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it
will be
saved with a new time stamp. And mirror will consider these file to
decide
whether to download or not.
Thanks,
Akshay
--
Alexander.
___
lftp mailing list
l...@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
now its producing 0 byte files and i am also getting input/output error here are the debug logs: lftp akshaygupta@192.168.128.124:~/Desktop/lftp mirror --ignore-time --- PASV --- 227 Entering Passive Mode (192,168,128,124,248,18) Connecting data socket to (192.168.128.124) port 63506 Data connection established --- LIST --- 150 Opening ASCII mode data connection for '/bin/ls'. Got EOF on data connection Closing data socket --- 226 Transfer complete. --- TYPE I --- 200 Type set to I. --- PASV --- 227 Entering Passive Mode (192,168,128,124,248,19) Connecting data socket to (192.168.128.124) port 63507 Data connection established --- RETR test.txt --- 150 Opening BINARY mode data connection for 'test.txt' (240 bytes). Got EOF on data connection Closing data socket --- 226 Transfer complete. copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,248,51) Connecting data socket to (192.168.128.124) port 63539 Data connection established --- RETR .DS_Store --- 150 Opening BINARY mode data connection for '.DS_Store' (6148 bytes). Got EOF on data connection Closing data socket --- 226 Transfer complete. copy: renaming `/mount/user/tom/.DS_Store_tmp_' to `.DS_Store' mirror: rename(/mount/user/tom/.DS_Store_tmp_, /mount/user/tom/.DS_Store): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,248,131) Connecting data socket to (192.168.128.124) port 63619 Data connection established --- RETR settings.xml --- 150 Opening BINARY mode data connection for 'settings.xml' (1737 bytes). --- 226 Transfer complete. Got EOF on data connection Closing data socket copy: renaming `/mount/user/tom/settings.xml_tmp_' to `settings.xml' mirror: rename(/mount/user/tom/settings.xml_tmp_, /mount/user/tom/settings.xml): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,248,178) Connecting data socket to (192.168.128.124) port 63666 Data connection established --- RETR search-2.html --- 150 Opening BINARY mode data connection for 'search-2.html' (42518 bytes). --- 226 Transfer complete. Got EOF on data connection Closing data socket copy: renaming `/mount/user/tom/search-2.html_tmp_' to `search-2.html' mirror: rename(/mount/user/tom/search-2.html_tmp_, /mount/user/tom/search-2.html): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,249,146) Connecting data socket to (192.168.128.124) port 63890 Data connection established --- RETR search-3.html --- 150 Opening BINARY mode data connection for 'search-3.html' (42569 bytes). --- 226 Transfer complete. Got EOF on data connection Closing data socket copy: renaming `/mount/user/tom/search-3.html_tmp_' to `search-3.html' mirror: rename(/mount/user/tom/search-3.html_tmp_, /mount/user/tom/search-3.html): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,250,160) Connecting data socket to (192.168.128.124) port 64160 Data connection established --- RETR search-4.html --- 150 Opening BINARY mode data connection for 'search-4.html' (42569 bytes). --- 226 Transfer complete. Got EOF on data connection Closing data socket copy: renaming `/mount/user/tom/search-4.html_tmp_' to `search-4.html' mirror: rename(/mount/user/tom/search-4.html_tmp_, /mount/user/tom/search-4.html): Input/output error --- PASV --- 227 Entering Passive Mode (192,168,128,124,251,142) Connecting data socket to (192.168.128.124) port 64398 Data connection established --- RETR search.html --- 150 Opening BINARY mode data connection for 'search.html' (42519 bytes). --- 226 Transfer complete. Got EOF on data connection Closing data socket copy: renaming `/mount/user/tom/search.html_tmp_' to `search.html' mirror: rename(/mount/user/tom/search.html_tmp_, /mount/user/tom/search.html): Input/output error Total: 1 directory, 7 files, 0 symlinks Modified: 7 files, 0 symlinks 178300 bytes transferred in 47 seconds (3.7 KiB/s) To be removed: 0 directories, 7 files, 0 symlinks 7 errors detected On Mon, Oct 27, 2014 at 5:01 PM, Alexander Lukyanov lavv...@gmail.com wrote: This change should fix the problem. https://github.com/lavv17/lftp/commit/b1164d695393cf8ec8429bb2b5caa9f05cfc4d39 2014-10-27 14:20 GMT+03:00 Alexander Lukyanov lavv...@gmail.com: I have found the bug. As a workaround, use set xfer:clobber yes. 2014-10-27 13:06 GMT+03:00 akshay gupta akshaygupta...@gmail.com: no, I am running as a root user. There is some issue with lftp, when file with same name already exist its not able to rename the temp file to actual file. On Mon, Oct 27, 2014 at 1:44 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Mon, Oct 27, 2014 at 01:30:42PM +0530, akshay gupta wrote: --- RETR test.txt --- 150 Opening BINARY
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
I am using fuse mounted hadoop distributed file system(HDFS). Can you suggest what changes need to be done to make it work as this is a very critical requirement for my project. Will deleting file and then renaming temp file help? how to achieve that? On Mon, Oct 27, 2014 at 7:38 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Mon, Oct 27, 2014 at 05:53:19PM +0530, akshay gupta wrote: now its producing 0 byte files and i am also getting input/output error here are the debug logs: copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt): Input/output error Which filesystem do you use? Maybe it is necessary to unlink target file before renaming, although rename should do it automatically. If newpath already exists it will be atomically replaced (subject to a few conditions; see ERRORS below), so that there is no point at which another process attempting to access newpath will find it missing. And man page for rename(2) does not list EIO error. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
[lftp-devel] providing key file for ssh in fish protocol
Hi All, I want provide a key file for authentication in fish protocol. Fish internally uses ssh for connection and ssh have a support for providing key files for password less authentication, how can I provide key file using lftp for FISH protocol. Thanks, Akshay ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Status for incomplete files
Thats just great, cheers!! On Fri, Sep 19, 2014 at 5:11 PM, Alexander Lukyanov lavv...@gmail.com wrote: It will be released in one or two weeks. 2014-09-19 15:38 GMT+04:00 akshay gupta akshaygupta...@gmail.com: Thanks a lot Alexander, when will these features come out in release version? I want to use these features in production, would be best if I could use a stable version. On Fri, Sep 19, 2014 at 4:58 PM, Alexander Lukyanov lavv...@gmail.com wrote: Please try http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20140919.tar.gz, it introduces another setting xfer:temp-file-name. You can thus change the temporary file name; an asterisk is replaced by the original file name. So you would do set xfer:temp-file-name _tmp_* to achieve wanted behaviour. 2014-09-19 11:11 GMT+04:00 akshay gupta akshaygupta...@gmail.com: Hi Alexander, Thanks for the reply! I tried using this and it seems to works prefixing .in. to files getting transferred. What is the expected release date for this development version? Does it also have an additional setting to use custom flags for temporary files. Like if i want _tmp_ prefixed to temporary files. ~ Akshay On Fri, Sep 19, 2014 at 12:09 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Fri, Sep 19, 2014 at 01:36:18AM +0530, akshay gupta wrote: I am looking for a way to know the file is still getting transferred and is incomplete. Some file downloaders append some token such as _tmp or something else to the file name until its getting transferred when the file transfer completes its renamed to its original name. I know about pget option which creates a *lftp-pget-status* file but I prefer more direct option. Please try development snapshot http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20140918.tar.gz with xfer:use-temp-file set to yes. -- Alexander. ___ lftp mailing list l...@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
[lftp-devel] Ways to know number of new files after mirror
Hi All, I want to know how many new files are added after mirror command has completed. I am going to run the lftp command through java code using runtime exec() call. Does lftp command provide some special exit status code when mirror succeeds but no new files are added. if not can you suggest some way of knowing when mirror command produces no new files. Thanks, Akshay ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] possible bug in lftp mirror using xfer:use-temp-file option
Its causing unpredictable behaviour in case of fuse mounted file system. Please look into it if it can be resolved. Thanks for the quick response. ~Akshay On Tue, Sep 30, 2014 at 9:15 PM, Alexander Lukyanov lavv...@gmail.com wrote: In unix it is usually allowed to rename open files, so lftp does that. I'll see if it is possible to reverse the order. 2014-09-30 12:18 GMT+04:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files on Fuse mounted hdfs file system ( https://wiki.apache.org/hadoop/MountableHDFS). Lftp is creating some 0 byte files. Observations: 1. Files of size larger then 64k are getting transferred properly. Smaller files have 0 byte file size. 2. Debug logs shows that lftp downloaded data for the files correctly. It also renamed them. Additional tests on fuse mounted hdfs. - I created a file with a temp file name, written the data to it, then renamed the file and finally closed the output stream. In this case it creates a 0 byte file. - I created a file with a temp file name, written the data to it, closed the file and then renamed the file. In this case it creates the file correctly. Can someone please look into this issue, it might be possible that stream is closed after the file is renamed. Thanks, Akshay -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] possible bug in lftp mirror using xfer:use-temp-file option
This option (xfer:use-temp-file) is available in development snapshot http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20140919.tar.gz. On Tue, Sep 30, 2014 at 10:35 PM, akshay gupta akshaygupta...@gmail.com wrote: Its causing unpredictable behaviour in case of fuse mounted file system. Please look into it if it can be resolved. Thanks for the quick response. ~Akshay On Tue, Sep 30, 2014 at 9:15 PM, Alexander Lukyanov lavv...@gmail.com wrote: In unix it is usually allowed to rename open files, so lftp does that. I'll see if it is possible to reverse the order. 2014-09-30 12:18 GMT+04:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files on Fuse mounted hdfs file system ( https://wiki.apache.org/hadoop/MountableHDFS). Lftp is creating some 0 byte files. Observations: 1. Files of size larger then 64k are getting transferred properly. Smaller files have 0 byte file size. 2. Debug logs shows that lftp downloaded data for the files correctly. It also renamed them. Additional tests on fuse mounted hdfs. - I created a file with a temp file name, written the data to it, then renamed the file and finally closed the output stream. In this case it creates a 0 byte file. - I created a file with a temp file name, written the data to it, closed the file and then renamed the file. In this case it creates the file correctly. Can someone please look into this issue, it might be possible that stream is closed after the file is renamed. Thanks, Akshay -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] possible bug in lftp mirror using xfer:use-temp-file option
I am using this feature to know whether a file is completely transferred or lftp is still downloading this file. On Wed, Oct 1, 2014 at 12:46 PM, akshay gupta akshaygupta...@gmail.com wrote: This option (xfer:use-temp-file) is available in development snapshot http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20140919.tar.gz. On Tue, Sep 30, 2014 at 10:35 PM, akshay gupta akshaygupta...@gmail.com wrote: Its causing unpredictable behaviour in case of fuse mounted file system. Please look into it if it can be resolved. Thanks for the quick response. ~Akshay On Tue, Sep 30, 2014 at 9:15 PM, Alexander Lukyanov lavv...@gmail.com wrote: In unix it is usually allowed to rename open files, so lftp does that. I'll see if it is possible to reverse the order. 2014-09-30 12:18 GMT+04:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files on Fuse mounted hdfs file system ( https://wiki.apache.org/hadoop/MountableHDFS). Lftp is creating some 0 byte files. Observations: 1. Files of size larger then 64k are getting transferred properly. Smaller files have 0 byte file size. 2. Debug logs shows that lftp downloaded data for the files correctly. It also renamed them. Additional tests on fuse mounted hdfs. - I created a file with a temp file name, written the data to it, then renamed the file and finally closed the output stream. In this case it creates a 0 byte file. - I created a file with a temp file name, written the data to it, closed the file and then renamed the file. In this case it creates the file correctly. Can someone please look into this issue, it might be possible that stream is closed after the file is renamed. Thanks, Akshay -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
