Re: [lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
The problem should be fixed by https://github.com/lavv17/lftp/commit/a2b4cdbf8ccdb98c8d5c79834ea6e29f83575da9 without unnecessary memory allocation. 2015-05-15 20:22 GMT+03:00 Alexander Lukyanov lavv...@gmail.com: Thanks for debugging! I'll fix it by using heap memory чт, 14 мая 2015, 14:59, Ankit Khurana ankit.khurana0...@gmail.com: Hi Found the issue, the lsToList is using string_alloca to allocate the ls result in stack memory which was limited. Tested by increasing the stack memory limit for a process. Fixing it by using heap memory using malloc instead of string_alloca. Thanks Ankit On Thu, May 14, 2015 at 2:48 PM, Ankit Khurana ankit.khurana0...@gmail.com wrote: Hi The segmention fault is coming only in case of mirror. ls -la is working when run on lftp using fish. lftp test@192.168.10.15:/data2/data/history/231 mirror data/ --- #CWD /data2/data/history/231/data cd /data2/data/history/231/data; echo '### 000' --- ### 000 --- #LIST ls -la ; echo '### 200' --- ### 200 Segmentation fault (core dumped) Thanks Ankit On Thu, May 14, 2015 at 11:57 AM, Ankit Khurana ankit.khurana0...@gmail.com wrote: Th last log printed is from Fish.CC line 410. The process is giving segmentation on every run. Thanks Ankit On Thu, May 14, 2015 at 10:16 AM, Ankit Khurana ankit.khurana0...@gmail.com wrote: Hi Alexander sending the core dump file for reference. Regards Ankit Khurana On Thu, May 14, 2015 at 9:32 AM, Ankit Khurana ankit.khurana0...@gmail.com wrote: Hi Alexander I am trying to generate the non stripped executable using the default make install packaged in lftp tar. But i am always getting the stripped executable. Thus not able to debug the core dump produced. Regards Ankit Khurana On Thu, May 14, 2015 at 9:05 AM, Alexander Lukyanov lavv...@gmail.com wrote: There should be no limitations. Please produce a stack trace and send it to me. ср, 13 мая 2015, 18:34, Ankit Khurana ankit.khurana0...@gmail.com: Hi, I am trying to mirror files using lftp, the mirror path contains around 1 million files. The problem i am facing is that when i start the mirror, it attempts to do `ls -la` on the mirror site and then the process dies with segmentation fault producing a core dump. Is there a limitation on the number of files to be present at the mirror site? Thanks Ankit -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
There should be no limitations. Please produce a stack trace and send it to me. ср, 13 мая 2015, 18:34, Ankit Khurana ankit.khurana0...@gmail.com: Hi, I am trying to mirror files using lftp, the mirror path contains around 1 million files. The problem i am facing is that when i start the mirror, it attempts to do `ls -la` on the mirror site and then the process dies with segmentation fault producing a core dump. Is there a limitation on the number of files to be present at the mirror site? Thanks Ankit ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Handling for small files
Which protocol do you use? вт, 24 февр. 2015, 13:48, akshay gupta akshaygupta...@gmail.com: Hi, I am trying to transfer about 80GB of data using mirror option. File sizes are very small about 0.5 KB to 3 KB per file. Transfer is taking a lot of time which I am attributing to small file size. Is there any option in lftp which I can use to increase the transfer speed. May be some sort of compression option like rsync -av*z.* I know about the parallel file transfer option but I cant use it because I want to maintain the order of file transfer. Please help. Thanks, Akshay ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
rpmbuild -tb lftp-VERSION.tar.gz should do the trick. 10:19, пн, 08.12.2014, akshay gupta akshaygupta...@gmail.com: Hi Alexander, Can you please also tell the steps needed to build an rpm for lftp. ~ Akshay On Sat, Dec 6, 2014 at 12:42 AM, akshay gupta akshaygupta...@gmail.com wrote: Have done some initial high level testing. It seems to work. Will test it thoroughly and will get back to you in case I find any bug. On Fri, Dec 5, 2014 at 10:20 PM, Alexander Lukyanov lavv...@gmail.com wrote: I think you need to install gettext-devel. I have also made a snapshot in case you don't want to build from git: http://lftp.yar.ru/ftp/devel/lftp-4.6.0.20141205.tar.gz 2014-12-05 9:43 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Its not working: I run the ./autogen.sh --without-gnutls --with-openssl command and it cannot seem to find Makefile.in in the configure step. Attached the complete logs On Thu, Dec 4, 2014 at 8:54 PM, Alexander Lukyanov lavv...@gmail.com wrote: autogen.sh script should generate everything. You'll have to get gnulib too. четверг, 4 декабря 2014 г. пользователь akshay gupta написал: I have cloned the lftp repo, can you please tell me the steps to create configure file and make script. I tried automake --force-missing --add-missing but its giving src/Makefile.am:45: NEED_TRIO does not appear in AM_CONDITIONAL src/Makefile.am: installing `build-aux/depcomp' Makefile.am:4: NEED_TRIO does not appear in AM_CONDITIONAL configure.ac:607: required file `lib/Makefile.in' not found ~ Akshay On Thu, Dec 4, 2014 at 4:56 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Wed, Dec 03, 2014 at 02:11:28PM +0400, Alexander Lukyanov wrote: It's almost done, needs to be tested and also some issues with sorting to be fixed. 2014-12-03 10:26 GMT+03:00 akshay gupta akshaygupta...@gmail.com : Are these feature present in LFTP implementation timeline? By when will they be implemented? You can now test the feature (git ee6391e adds --scan-all-first option). https://github.com/lavv17/lftp/commit/ee6391e3213e12013894aa98508a1a151f542da0 -- Alexander. -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
I think you need to install gettext-devel. I have also made a snapshot in case you don't want to build from git: http://lftp.yar.ru/ftp/devel/lftp-4.6.0.20141205.tar.gz 2014-12-05 9:43 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Its not working: I run the ./autogen.sh --without-gnutls --with-openssl command and it cannot seem to find Makefile.in in the configure step. Attached the complete logs On Thu, Dec 4, 2014 at 8:54 PM, Alexander Lukyanov lavv...@gmail.com wrote: autogen.sh script should generate everything. You'll have to get gnulib too. четверг, 4 декабря 2014 г. пользователь akshay gupta написал: I have cloned the lftp repo, can you please tell me the steps to create configure file and make script. I tried automake --force-missing --add-missing but its giving src/Makefile.am:45: NEED_TRIO does not appear in AM_CONDITIONAL src/Makefile.am: installing `build-aux/depcomp' Makefile.am:4: NEED_TRIO does not appear in AM_CONDITIONAL configure.ac:607: required file `lib/Makefile.in' not found ~ Akshay On Thu, Dec 4, 2014 at 4:56 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Wed, Dec 03, 2014 at 02:11:28PM +0400, Alexander Lukyanov wrote: It's almost done, needs to be tested and also some issues with sorting to be fixed. 2014-12-03 10:26 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Are these feature present in LFTP implementation timeline? By when will they be implemented? You can now test the feature (git ee6391e adds --scan-all-first option). https://github.com/lavv17/lftp/commit/ee6391e3213e12013894aa98508a1a151f542da0 -- Alexander. -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
I think it is possible and the amount of programming work is moderate. The recursive model of mirror will stay but the file transfers will be deferred and done on the top level when recursive directory traversal is finished. 2014-11-12 8:32 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi Alexander, Are there any plans for redesigning LFTP Mirror algorithm to build the whole recursive list first. How much change would that require if we go that path? Thanks, Akshay On Thu, Oct 30, 2014 at 4:44 PM, akshay gupta akshaygupta...@gmail.com wrote: lftp have options like --depth-first and --no-recursion I think it will make sense to have an option like --iterative for iterative file transfer. This option will fetch the full file list and then fetch them iteratively. On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta akshaygupta...@gmail.com wrote: I think the functionality of mirror:sort-by option implies that the order is for all the files irrespective of the directory structure. Or may be it should support both the options and it would be configuration defined which one to use. On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov lavv...@gmail.com wrote: Mirror in lftp works directory by directory. It does not build full list of files first, so it cannot sort the whole list. Maybe the mirror algorithm should be redesigned to build the whole recursive list first. What do you think? 2014-10-29 10:07 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using mirror:sort-by option set to 'date-desc'. My file structure is like this $ ls -l -R total 3191264 drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv ./1: total 0 drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 ./1/2: total 515512 -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv I expect the file synch order should be ~/tmp.txt ~/tmp1.txt ~/abc.csv ~/1/2/bc.csv ~/1/2/3 but the synch order is: ~/tmp.txt ~/tmp1.txt ~/1/2/bc.csv ~/1/2/3 ~/abc.csv I think its the folder modification time that is causing this issue. Lftp considers folder synch time to decide the order even though files contained in the folder are newer then the remaining files in the parent folder. File synch order should be in order of the modification time irrespective of the folder structure. Please help. Thanks, Akshay -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Possible Bug in lftp mirror using mirror:sort-by option
Mirror in lftp works directory by directory. It does not build full list of files first, so it cannot sort the whole list. Maybe the mirror algorithm should be redesigned to build the whole recursive list first. What do you think? 2014-10-29 10:07 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using mirror:sort-by option set to 'date-desc'. My file structure is like this $ ls -l -R total 3191264 drwxr-xr-x 3 vikasgoyal biadmin 102 Oct 22 12:35 1 -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 01:00 tmp.txt -rw-r--r-- 1 vikasgoyal biadmin 0 Aug 1 05:31 tmp1.txt -rw-r--r-- 1 vikasgoyal biadmin 263940738 Oct 28 13:51 abc.csv ./1: total 0 drwxr-xr-x 4 vikasgoyal biadmin 136 Oct 28 14:19 2 ./1/2: total 515512 -rw-r--r-- 1 vikasgoyal biadmin 0 Oct 28 14:19 3 -rw-r--r-- 1 vikasgoyal biadmin 263940732 Oct 28 14:03 bc.csv I expect the file synch order should be ~/tmp.txt ~/tmp1.txt ~/abc.csv ~/1/2/bc.csv ~/1/2/3 but the synch order is: ~/tmp.txt ~/tmp1.txt ~/1/2/bc.csv ~/1/2/3 ~/abc.csv I think its the folder modification time that is causing this issue. Lftp considers folder synch time to decide the order even though files contained in the folder are newer then the remaining files in the parent folder. File synch order should be in order of the modification time irrespective of the folder structure. Please help. Thanks, Akshay -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
It looks like a bug in the FS (fuse and/or hdfs) which also hits rsync: https://groups.google.com/a/cloudera.org/forum/#!msg/cdh-user/R1GR2Asj_lc/f_uuIjG5R1UJ Please try this workaround. 2014-10-28 10:29 GMT+03:00 akshay gupta akshaygupta...@gmail.com: will giving --delete-first option in lftp mirror help in my case? I think its not working in case of use-temp-file option. As per the man page its suppose to delete the old version of the file before it start downloading the new one. Its not doing anything. Please help. On Mon, Oct 27, 2014 at 11:50 PM, akshay gupta akshaygupta...@gmail.com wrote: I am using fuse mounted hadoop distributed file system(HDFS). Can you suggest what changes need to be done to make it work as this is a very critical requirement for my project. Will deleting file and then renaming temp file help? how to achieve that? On Mon, Oct 27, 2014 at 7:38 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Mon, Oct 27, 2014 at 05:53:19PM +0530, akshay gupta wrote: now its producing 0 byte files and i am also getting input/output error here are the debug logs: copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt): Input/output error Which filesystem do you use? Maybe it is necessary to unlink target file before renaming, although rename should do it automatically. If newpath already exists it will be atomically replaced (subject to a few conditions; see ERRORS below), so that there is no point at which another process attempting to access newpath will find it missing. And man page for rename(2) does not list EIO error. -- Alexander. -- Alexander. diff Description: Binary data ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
Can you provide a debug log? Why lftp cannot replace the old file? 2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using lftp, I have tested a use case where a file at the source machine gets modified but the file name remains the same. After mirroring, lftp is unable to replace the old file and also leaves a temporary file. For example. If source have file a.txt and its mirrored successfully at the destination, now if I update a.txt append some content and save it with same name. If I run lftp mirror again then its not able to replace the existing a.txt but it also leaves the temp file a.txt_transit_ behind. And if i run the mirror multiple times it always downloads the changed a.txt, i can see this from the lftp logs. Suggestion: Will it make sense to keep versions of the files as they are updated at the source. Like if a.txt is changed at source then lftp mirror will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be saved with a new time stamp. And mirror will consider these file to decide whether to download or not. Thanks, Akshay -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
I have found the bug. As a workaround, use set xfer:clobber yes. 2014-10-27 13:06 GMT+03:00 akshay gupta akshaygupta...@gmail.com: no, I am running as a root user. There is some issue with lftp, when file with same name already exist its not able to rename the temp file to actual file. On Mon, Oct 27, 2014 at 1:44 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Mon, Oct 27, 2014 at 01:30:42PM +0530, akshay gupta wrote: --- RETR test.txt --- 150 Opening BINARY mode data connection for 'test.txt' (202 bytes). Got EOF on data connection Closing data socket --- 226 Transfer complete. So the file cannot be renamed locally, right? Is there a permissions problem? -- Alexander. On Mon, Oct 27, 2014 at 1:18 PM, Alexander Lukyanov lavv...@gmail.com wrote: Can you provide a debug log? Why lftp cannot replace the old file? 2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files using lftp, I have tested a use case where a file at the source machine gets modified but the file name remains the same. After mirroring, lftp is unable to replace the old file and also leaves a temporary file. For example. If source have file a.txt and its mirrored successfully at the destination, now if I update a.txt append some content and save it with same name. If I run lftp mirror again then its not able to replace the existing a.txt but it also leaves the temp file a.txt_transit_ behind. And if i run the mirror multiple times it always downloads the changed a.txt, i can see this from the lftp logs. Suggestion: Will it make sense to keep versions of the files as they are updated at the source. Like if a.txt is changed at source then lftp mirror will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be saved with a new time stamp. And mirror will consider these file to decide whether to download or not. Thanks, Akshay -- Alexander. ___ lftp mailing list l...@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] possible bug in lftp mirror using xfer:use-temp-file option
In unix it is usually allowed to rename open files, so lftp does that. I'll see if it is possible to reverse the order. 2014-09-30 12:18 GMT+04:00 akshay gupta akshaygupta...@gmail.com: Hi, I am trying to mirror files on Fuse mounted hdfs file system ( https://wiki.apache.org/hadoop/MountableHDFS). Lftp is creating some 0 byte files. Observations: 1. Files of size larger then 64k are getting transferred properly. Smaller files have 0 byte file size. 2. Debug logs shows that lftp downloaded data for the files correctly. It also renamed them. Additional tests on fuse mounted hdfs. - I created a file with a temp file name, written the data to it, then renamed the file and finally closed the output stream. In this case it creates a 0 byte file. - I created a file with a temp file name, written the data to it, closed the file and then renamed the file. In this case it creates the file correctly. Can someone please look into this issue, it might be possible that stream is closed after the file is renamed. Thanks, Akshay -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Status for incomplete files
Please try http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20140919.tar.gz, it introduces another setting xfer:temp-file-name. You can thus change the temporary file name; an asterisk is replaced by the original file name. So you would do set xfer:temp-file-name _tmp_* to achieve wanted behaviour. 2014-09-19 11:11 GMT+04:00 akshay gupta akshaygupta...@gmail.com: Hi Alexander, Thanks for the reply! I tried using this and it seems to works prefixing .in. to files getting transferred. What is the expected release date for this development version? Does it also have an additional setting to use custom flags for temporary files. Like if i want _tmp_ prefixed to temporary files. ~ Akshay On Fri, Sep 19, 2014 at 12:09 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Fri, Sep 19, 2014 at 01:36:18AM +0530, akshay gupta wrote: I am looking for a way to know the file is still getting transferred and is incomplete. Some file downloaders append some token such as _tmp or something else to the file name until its getting transferred when the file transfer completes its renamed to its original name. I know about pget option which creates a *lftp-pget-status* file but I prefer more direct option. Please try development snapshot http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20140918.tar.gz with xfer:use-temp-file set to yes. -- Alexander. ___ lftp mailing list l...@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Status for incomplete files
It will be released in one or two weeks. 2014-09-19 15:38 GMT+04:00 akshay gupta akshaygupta...@gmail.com: Thanks a lot Alexander, when will these features come out in release version? I want to use these features in production, would be best if I could use a stable version. On Fri, Sep 19, 2014 at 4:58 PM, Alexander Lukyanov lavv...@gmail.com wrote: Please try http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20140919.tar.gz, it introduces another setting xfer:temp-file-name. You can thus change the temporary file name; an asterisk is replaced by the original file name. So you would do set xfer:temp-file-name _tmp_* to achieve wanted behaviour. 2014-09-19 11:11 GMT+04:00 akshay gupta akshaygupta...@gmail.com: Hi Alexander, Thanks for the reply! I tried using this and it seems to works prefixing .in. to files getting transferred. What is the expected release date for this development version? Does it also have an additional setting to use custom flags for temporary files. Like if i want _tmp_ prefixed to temporary files. ~ Akshay On Fri, Sep 19, 2014 at 12:09 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Fri, Sep 19, 2014 at 01:36:18AM +0530, akshay gupta wrote: I am looking for a way to know the file is still getting transferred and is incomplete. Some file downloaders append some token such as _tmp or something else to the file name until its getting transferred when the file transfer completes its renamed to its original name. I know about pget option which creates a *lftp-pget-status* file but I prefer more direct option. Please try development snapshot http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20140918.tar.gz with xfer:use-temp-file set to yes. -- Alexander. ___ lftp mailing list l...@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp -- Alexander. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] FW: Regarding LFTP checksum issue
Do you have checksum issues if you use plain get command? Does pget work as expected with other ftp servers? 2014-04-01 13:15 GMT+04:00 PRADHAN Priyabrata priyabrata.prad...@murex.com : Fyia Thanks Priya *From:* PRADHAN Priyabrata *Sent:* Tuesday, April 01, 2014 6:14 PM *To:* 'l...@yars.free.net' *Subject:* Regarding LFTP checksum issue Hi Alexander V. Lukyanov, Thanks for this lftp tool. We are testing this tool. It’s a good tool having good features. But checksum is not intact after ftp. Any work around? We are using lftp -u user,passwd -e 'pget -n 20 -c file' remote server Thanks Priya *** This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system. ___ lftp mailing list l...@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel