We are in desperate need of this feature as without this feature we can only ensure file order within a folder. There is a huge dependency on file order for some of our hadoop oozie jobs. By when can we expect this feature to come in lftp?
On Wed, Nov 12, 2014 at 1:54 PM, Alexander Lukyanov <lavv...@gmail.com> wrote: > I think it is possible and the amount of programming work is moderate. The > recursive model of mirror will stay but the file transfers will be deferred > and done on the top level when recursive directory traversal is finished. > > 2014-11-12 8:32 GMT+03:00 akshay gupta <akshaygupta...@gmail.com>: > >> Hi Alexander, >> >> Are there any plans for redesigning LFTP Mirror algorithm to build the >> whole recursive list first. How much change would that require if we go >> that path? >> >> Thanks, >> Akshay >> >> On Thu, Oct 30, 2014 at 4:44 PM, akshay gupta <akshaygupta...@gmail.com> >> wrote: >> >>> lftp have options like --depth-first and --no-recursion I think it will >>> make sense to have an option like " --iterative" for iterative file >>> transfer. This option will fetch the full file list and then fetch them >>> iteratively. >>> >>> On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta <akshaygupta...@gmail.com> >>> wrote: >>> >>>> I think the functionality of mirror:sort-by option implies that the >>>> order is for all the files irrespective of the directory structure. Or may >>>> be it should support both the options and it would be configuration defined >>>> which one to use. >>>> >>>> On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov <lavv...@gmail.com> >>>> wrote: >>>> >>>>> Mirror in lftp works directory by directory. It does not build full >>>>> list of files first, so it cannot sort the whole list. Maybe the mirror >>>>> algorithm should be redesigned to build the whole recursive list first. >>>>> What do you think? >>>>> >>>>> 2014-10-29 10:07 GMT+03:00 akshay gupta <akshaygupta...@gmail.com>: >>>>> >>>>>> Hi, >>>>>> >>>>>> I am trying to mirror files using mirror:sort-by option set to >>>>>> 'date-desc'. My file structure is like this >>>>>> >>>>>> $ ls -l -R >>>>>> total 3191264 >>>>>> drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 >>>>>> -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt >>>>>> -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt >>>>>> -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv >>>>>> >>>>>> ./1: >>>>>> total 0 >>>>>> drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 >>>>>> >>>>>> ./1/2: >>>>>> total 515512 >>>>>> -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 >>>>>> -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv >>>>>> >>>>>> >>>>>> I expect the file synch order should be >>>>>> >>>>>> ~/tmp.txt >>>>>> ~/tmp1.txt >>>>>> ~/abc.csv >>>>>> ~/1/2/bc.csv >>>>>> ~/1/2/3 >>>>>> >>>>>> >>>>>> but the synch order is: >>>>>> ~/tmp.txt >>>>>> ~/tmp1.txt >>>>>> ~/1/2/bc.csv >>>>>> ~/1/2/3 >>>>>> ~/abc.csv >>>>>> >>>>>> >>>>>> I think its the folder modification time that is causing this issue. >>>>>> Lftp considers folder synch time to decide the order even though files >>>>>> contained in the folder are newer then the remaining files in the parent >>>>>> folder. File synch order should be in order of the modification time >>>>>> irrespective of the folder structure. Please help. >>>>>> >>>>>> Thanks, >>>>>> Akshay >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Alexander. >>>>> >>>> >>>> >>> >> > > > -- > Alexander. >
_______________________________________________ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
