Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
Its not working: I run the ./autogen.sh --without-gnutls --with-openssl command and it cannot seem to find Makefile.in in the configure step. Attached the complete logs On Thu, Dec 4, 2014 at 8:54 PM, Alexander Lukyanov lavv...@gmail.com wrote: autogen.sh script should generate everything. You'll have to get gnulib too. четверг, 4 декабря 2014 г. пользователь akshay gupta написал: I have cloned the lftp repo, can you please tell me the steps to create configure file and make script. I tried automake --force-missing --add-missing but its giving src/Makefile.am:45: NEED_TRIO does not appear in AM_CONDITIONAL src/Makefile.am: installing `build-aux/depcomp' Makefile.am:4: NEED_TRIO does not appear in AM_CONDITIONAL configure.ac:607: required file `lib/Makefile.in' not found ~ Akshay On Thu, Dec 4, 2014 at 4:56 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Wed, Dec 03, 2014 at 02:11:28PM +0400, Alexander Lukyanov wrote: It's almost done, needs to be tested and also some issues with sorting to be fixed. 2014-12-03 10:26 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Are these feature present in LFTP implementation timeline? By when will they be implemented? You can now test the feature (git ee6391e adds --scan-all-first option). https://github.com/lavv17/lftp/commit/ee6391e3213e12013894aa98508a1a151f542da0 -- Alexander. -- Alexander. [root@localhost lftp]# ./autogen.sh --without-gnutls --with-openssl processing . Creating ./aclocal.m4 ... Running gettextize... Ignore non-fatal messages. Not copying intl/ directory. Copying file po/Makefile.in.in Copying file po/boldquot.sed Copying file po/en@boldquot.header Copying file po/en@quot.header Copying file po/insert-header.sin Copying file po/Makevars.template Copying file po/quot.sed Copying file po/remove-potcdate.sin Copying file po/Rules-quot perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_CTYPE = UTF-8, LANG = en_US.UTF-8 are supported and installed on your system. perl: warning: Falling back to the standard locale (C). perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_CTYPE = UTF-8, LANG = en_US.UTF-8 are supported and installed on your system. perl: warning: Falling back to the standard locale (C). Copying file m4/gettext.m4 cp: cannot stat `/usr/share/aclocal/gettext.m4': No such file or directory Copying file m4/iconv.m4 cp: cannot stat `/usr/share/aclocal/iconv.m4': No such file or directory Copying file m4/lib-ld.m4 cp: cannot stat `/usr/share/aclocal/lib-ld.m4': No such file or directory Copying file m4/lib-link.m4 cp: cannot stat `/usr/share/aclocal/lib-link.m4': No such file or directory Copying file m4/lib-prefix.m4 cp: cannot stat `/usr/share/aclocal/lib-prefix.m4': No such file or directory Copying file m4/nls.m4 cp: cannot stat `/usr/share/aclocal/nls.m4': No such file or directory Copying file m4/po.m4 cp: cannot stat `/usr/share/aclocal/po.m4': No such file or directory Copying file m4/progtest.m4 cp: cannot stat `/usr/share/aclocal/progtest.m4': No such file or directory Please update po/Makevars so that it defines all the variables mentioned in po/Makevars.template. You can then remove po/Makevars.template. Please run 'aclocal -I m4' to regenerate the aclocal.m4 file. You need aclocal from GNU automake 1.9 (or newer) to do this. Then run 'autoconf' to regenerate the configure file. You might also want to copy the convenience header file gettext.h from the /usr/share/gettext directory into your package. It is a wrapper around libintl.h that implements the configure --disable-nls option. Press Return to acknowledge the previous three paragraphs. mv: cannot stat `configure.ac~': No such file or directory mv: cannot stat `m4/Makefile.am~': No such file or directory Making ./aclocal.m4 writable ... Running libtoolize... libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `build-aux'. libtoolize: copying file `build-aux/ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'. libtoolize: copying file `m4/libtool.m4' libtoolize: copying file `m4/ltoptions.m4' libtoolize: copying file `m4/ltsugar.m4' libtoolize: copying file `m4/ltversion.m4' libtoolize: copying file `m4/lt~obsolete.m4' mv: cannot stat `Makefile.am~': No such file or directory Module list with included dependencies (indented): absolute-header alloca alloca-opt argmatch arpa_inet btowc c-ctype c-strcase c-strcaseeq chown clock-time cloexec close closedir configmake crypto/md5 crypto/sha1 d-type dirent dirfd dosname dup2 environ errno error exitfail extensions extern-inline fcntl fcntl-h fd-hook filemode filename
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
rpmbuild -tb lftp-VERSION.tar.gz should do the trick. 10:19, пн, 08.12.2014, akshay gupta akshaygupta...@gmail.com: Hi Alexander, Can you please also tell the steps needed to build an rpm for lftp. ~ Akshay On Sat, Dec 6, 2014 at 12:42 AM, akshay gupta akshaygupta...@gmail.com wrote: Have done some initial high level testing. It seems to work. Will test it thoroughly and will get back to you in case I find any bug. On Fri, Dec 5, 2014 at 10:20 PM, Alexander Lukyanov lavv...@gmail.com wrote: I think you need to install gettext-devel. I have also made a snapshot in case you don't want to build from git: http://lftp.yar.ru/ftp/devel/lftp-4.6.0.20141205.tar.gz 2014-12-05 9:43 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Its not working: I run the ./autogen.sh --without-gnutls --with-openssl command and it cannot seem to find Makefile.in in the configure step. Attached the complete logs On Thu, Dec 4, 2014 at 8:54 PM, Alexander Lukyanov lavv...@gmail.com wrote: autogen.sh script should generate everything. You'll have to get gnulib too. четверг, 4 декабря 2014 г. пользователь akshay gupta написал: I have cloned the lftp repo, can you please tell me the steps to create configure file and make script. I tried automake --force-missing --add-missing but its giving src/Makefile.am:45: NEED_TRIO does not appear in AM_CONDITIONAL src/Makefile.am: installing `build-aux/depcomp' Makefile.am:4: NEED_TRIO does not appear in AM_CONDITIONAL configure.ac:607: required file `lib/Makefile.in' not found ~ Akshay On Thu, Dec 4, 2014 at 4:56 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Wed, Dec 03, 2014 at 02:11:28PM +0400, Alexander Lukyanov wrote: It's almost done, needs to be tested and also some issues with sorting to be fixed. 2014-12-03 10:26 GMT+03:00 akshay gupta akshaygupta...@gmail.com : Are these feature present in LFTP implementation timeline? By when will they be implemented? You can now test the feature (git ee6391e adds --scan-all-first option). https://github.com/lavv17/lftp/commit/ee6391e3213e12013894aa98508a1a151f542da0 -- Alexander. -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
I think you need to install gettext-devel. I have also made a snapshot in case you don't want to build from git: http://lftp.yar.ru/ftp/devel/lftp-4.6.0.20141205.tar.gz 2014-12-05 9:43 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Its not working: I run the ./autogen.sh --without-gnutls --with-openssl command and it cannot seem to find Makefile.in in the configure step. Attached the complete logs On Thu, Dec 4, 2014 at 8:54 PM, Alexander Lukyanov lavv...@gmail.com wrote: autogen.sh script should generate everything. You'll have to get gnulib too. четверг, 4 декабря 2014 г. пользователь akshay gupta написал: I have cloned the lftp repo, can you please tell me the steps to create configure file and make script. I tried automake --force-missing --add-missing but its giving src/Makefile.am:45: NEED_TRIO does not appear in AM_CONDITIONAL src/Makefile.am: installing `build-aux/depcomp' Makefile.am:4: NEED_TRIO does not appear in AM_CONDITIONAL configure.ac:607: required file `lib/Makefile.in' not found ~ Akshay On Thu, Dec 4, 2014 at 4:56 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Wed, Dec 03, 2014 at 02:11:28PM +0400, Alexander Lukyanov wrote: It's almost done, needs to be tested and also some issues with sorting to be fixed. 2014-12-03 10:26 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Are these feature present in LFTP implementation timeline? By when will they be implemented? You can now test the feature (git ee6391e adds --scan-all-first option). https://github.com/lavv17/lftp/commit/ee6391e3213e12013894aa98508a1a151f542da0 -- Alexander. -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
Have done some initial high level testing. It seems to work. Will test it thoroughly and will get back to you in case I find any bug. On Fri, Dec 5, 2014 at 10:20 PM, Alexander Lukyanov lavv...@gmail.com wrote: I think you need to install gettext-devel. I have also made a snapshot in case you don't want to build from git: http://lftp.yar.ru/ftp/devel/lftp-4.6.0.20141205.tar.gz 2014-12-05 9:43 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Its not working: I run the ./autogen.sh --without-gnutls --with-openssl command and it cannot seem to find Makefile.in in the configure step. Attached the complete logs On Thu, Dec 4, 2014 at 8:54 PM, Alexander Lukyanov lavv...@gmail.com wrote: autogen.sh script should generate everything. You'll have to get gnulib too. четверг, 4 декабря 2014 г. пользователь akshay gupta написал: I have cloned the lftp repo, can you please tell me the steps to create configure file and make script. I tried automake --force-missing --add-missing but its giving src/Makefile.am:45: NEED_TRIO does not appear in AM_CONDITIONAL src/Makefile.am: installing `build-aux/depcomp' Makefile.am:4: NEED_TRIO does not appear in AM_CONDITIONAL configure.ac:607: required file `lib/Makefile.in' not found ~ Akshay On Thu, Dec 4, 2014 at 4:56 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Wed, Dec 03, 2014 at 02:11:28PM +0400, Alexander Lukyanov wrote: It's almost done, needs to be tested and also some issues with sorting to be fixed. 2014-12-03 10:26 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Are these feature present in LFTP implementation timeline? By when will they be implemented? You can now test the feature (git ee6391e adds --scan-all-first option). https://github.com/lavv17/lftp/commit/ee6391e3213e12013894aa98508a1a151f542da0 -- Alexander. -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
Are these feature present in LFTP implementation timeline? By when will they be implemented? ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
We are in desperate need of this feature as without this feature we can only ensure file order within a folder. There is a huge dependency on file order for some of our hadoop oozie jobs. By when can we expect this feature to come in lftp? On Wed, Nov 12, 2014 at 1:54 PM, Alexander Lukyanov lavv...@gmail.com wrote: I think it is possible and the amount of programming work is moderate. The recursive model of mirror will stay but the file transfers will be deferred and done on the top level when recursive directory traversal is finished. 2014-11-12 8:32 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi Alexander, Are there any plans for redesigning LFTP Mirror algorithm to build the whole recursive list first. How much change would that require if we go that path? Thanks, Akshay On Thu, Oct 30, 2014 at 4:44 PM, akshay gupta akshaygupta...@gmail.com wrote: lftp have options like --depth-first and --no-recursion I think it will make sense to have an option like --iterative for iterative file transfer. This option will fetch the full file list and then fetch them iteratively. On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta akshaygupta...@gmail.com wrote: I think the functionality of mirror:sort-by option implies that the order is for all the files irrespective of the directory structure. Or may be it should support both the options and it would be configuration defined which one to use. On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov lavv...@gmail.com wrote: Mirror in lftp works directory by directory. It does not build full list of files first, so it cannot sort the whole list. Maybe the mirror algorithm should be redesigned to build the whole recursive list first. What do you think? 2014-10-29 10:07 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using mirror:sort-by option set to 'date-desc'. My file structure is like this $ ls -l -R total 3191264 drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv ./1: total 0 drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 ./1/2: total 515512 -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv I expect the file synch order should be ~/tmp.txt ~/tmp1.txt ~/abc.csv ~/1/2/bc.csv ~/1/2/3 but the synch order is: ~/tmp.txt ~/tmp1.txt ~/1/2/bc.csv ~/1/2/3 ~/abc.csv I think its the folder modification time that is causing this issue. Lftp considers folder synch time to decide the order even though files contained in the folder are newer then the remaining files in the parent folder. File synch order should be in order of the modification time irrespective of the folder structure. Please help. Thanks, Akshay -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
I think it is possible and the amount of programming work is moderate. The recursive model of mirror will stay but the file transfers will be deferred and done on the top level when recursive directory traversal is finished. 2014-11-12 8:32 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi Alexander, Are there any plans for redesigning LFTP Mirror algorithm to build the whole recursive list first. How much change would that require if we go that path? Thanks, Akshay On Thu, Oct 30, 2014 at 4:44 PM, akshay gupta akshaygupta...@gmail.com wrote: lftp have options like --depth-first and --no-recursion I think it will make sense to have an option like --iterative for iterative file transfer. This option will fetch the full file list and then fetch them iteratively. On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta akshaygupta...@gmail.com wrote: I think the functionality of mirror:sort-by option implies that the order is for all the files irrespective of the directory structure. Or may be it should support both the options and it would be configuration defined which one to use. On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov lavv...@gmail.com wrote: Mirror in lftp works directory by directory. It does not build full list of files first, so it cannot sort the whole list. Maybe the mirror algorithm should be redesigned to build the whole recursive list first. What do you think? 2014-10-29 10:07 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using mirror:sort-by option set to 'date-desc'. My file structure is like this $ ls -l -R total 3191264 drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv ./1: total 0 drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 ./1/2: total 515512 -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv I expect the file synch order should be ~/tmp.txt ~/tmp1.txt ~/abc.csv ~/1/2/bc.csv ~/1/2/3 but the synch order is: ~/tmp.txt ~/tmp1.txt ~/1/2/bc.csv ~/1/2/3 ~/abc.csv I think its the folder modification time that is causing this issue. Lftp considers folder synch time to decide the order even though files contained in the folder are newer then the remaining files in the parent folder. File synch order should be in order of the modification time irrespective of the folder structure. Please help. Thanks, Akshay -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
lftp have options like --depth-first and --no-recursion I think it will make sense to have an option like --iterative for iterative file transfer. This option will fetch the full file list and then fetch them iteratively. On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta akshaygupta...@gmail.com wrote: I think the functionality of mirror:sort-by option implies that the order is for all the files irrespective of the directory structure. Or may be it should support both the options and it would be configuration defined which one to use. On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov lavv...@gmail.com wrote: Mirror in lftp works directory by directory. It does not build full list of files first, so it cannot sort the whole list. Maybe the mirror algorithm should be redesigned to build the whole recursive list first. What do you think? 2014-10-29 10:07 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using mirror:sort-by option set to 'date-desc'. My file structure is like this $ ls -l -R total 3191264 drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv ./1: total 0 drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 ./1/2: total 515512 -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv I expect the file synch order should be ~/tmp.txt ~/tmp1.txt ~/abc.csv ~/1/2/bc.csv ~/1/2/3 but the synch order is: ~/tmp.txt ~/tmp1.txt ~/1/2/bc.csv ~/1/2/3 ~/abc.csv I think its the folder modification time that is causing this issue. Lftp considers folder synch time to decide the order even though files contained in the folder are newer then the remaining files in the parent folder. File synch order should be in order of the modification time irrespective of the folder structure. Please help. Thanks, Akshay -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
Mirror in lftp works directory by directory. It does not build full list of files first, so it cannot sort the whole list. Maybe the mirror algorithm should be redesigned to build the whole recursive list first. What do you think? 2014-10-29 10:07 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using mirror:sort-by option set to 'date-desc'. My file structure is like this $ ls -l -R total 3191264 drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv ./1: total 0 drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 ./1/2: total 515512 -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv I expect the file synch order should be ~/tmp.txt ~/tmp1.txt ~/abc.csv ~/1/2/bc.csv ~/1/2/3 but the synch order is: ~/tmp.txt ~/tmp1.txt ~/1/2/bc.csv ~/1/2/3 ~/abc.csv I think its the folder modification time that is causing this issue. Lftp considers folder synch time to decide the order even though files contained in the folder are newer then the remaining files in the parent folder. File synch order should be in order of the modification time irrespective of the folder structure. Please help. Thanks, Akshay -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
I think the functionality of mirror:sort-by option implies that the order is for all the files irrespective of the directory structure. Or may be it should support both the options and it would be configuration defined which one to use. On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov lavv...@gmail.com wrote: Mirror in lftp works directory by directory. It does not build full list of files first, so it cannot sort the whole list. Maybe the mirror algorithm should be redesigned to build the whole recursive list first. What do you think? 2014-10-29 10:07 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using mirror:sort-by option set to 'date-desc'. My file structure is like this $ ls -l -R total 3191264 drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv ./1: total 0 drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 ./1/2: total 515512 -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv I expect the file synch order should be ~/tmp.txt ~/tmp1.txt ~/abc.csv ~/1/2/bc.csv ~/1/2/3 but the synch order is: ~/tmp.txt ~/tmp1.txt ~/1/2/bc.csv ~/1/2/3 ~/abc.csv I think its the folder modification time that is causing this issue. Lftp considers folder synch time to decide the order even though files contained in the folder are newer then the remaining files in the parent folder. File synch order should be in order of the modification time irrespective of the folder structure. Please help. Thanks, Akshay -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
