Re: [lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
The problem should be fixed by https://github.com/lavv17/lftp/commit/a2b4cdbf8ccdb98c8d5c79834ea6e29f83575da9 without unnecessary memory allocation. 2015-05-15 20:22 GMT+03:00 Alexander Lukyanov lavv...@gmail.com: Thanks for debugging! I'll fix it by using heap memory чт, 14 мая 2015, 14:59, Ankit Khurana ankit.khurana0...@gmail.com: Hi Found the issue, the lsToList is using string_alloca to allocate the ls result in stack memory which was limited. Tested by increasing the stack memory limit for a process. Fixing it by using heap memory using malloc instead of string_alloca. Thanks Ankit On Thu, May 14, 2015 at 2:48 PM, Ankit Khurana ankit.khurana0...@gmail.com wrote: Hi The segmention fault is coming only in case of mirror. ls -la is working when run on lftp using fish. lftp test@192.168.10.15:/data2/data/history/231 mirror data/ --- #CWD /data2/data/history/231/data cd /data2/data/history/231/data; echo '### 000' --- ### 000 --- #LIST ls -la ; echo '### 200' --- ### 200 Segmentation fault (core dumped) Thanks Ankit On Thu, May 14, 2015 at 11:57 AM, Ankit Khurana ankit.khurana0...@gmail.com wrote: Th last log printed is from Fish.CC line 410. The process is giving segmentation on every run. Thanks Ankit On Thu, May 14, 2015 at 10:16 AM, Ankit Khurana ankit.khurana0...@gmail.com wrote: Hi Alexander sending the core dump file for reference. Regards Ankit Khurana On Thu, May 14, 2015 at 9:32 AM, Ankit Khurana ankit.khurana0...@gmail.com wrote: Hi Alexander I am trying to generate the non stripped executable using the default make install packaged in lftp tar. But i am always getting the stripped executable. Thus not able to debug the core dump produced. Regards Ankit Khurana On Thu, May 14, 2015 at 9:05 AM, Alexander Lukyanov lavv...@gmail.com wrote: There should be no limitations. Please produce a stack trace and send it to me. ср, 13 мая 2015, 18:34, Ankit Khurana ankit.khurana0...@gmail.com: Hi, I am trying to mirror files using lftp, the mirror path contains around 1 million files. The problem i am facing is that when i start the mirror, it attempts to do `ls -la` on the mirror site and then the process dies with segmentation fault producing a core dump. Is there a limitation on the number of files to be present at the mirror site? Thanks Ankit -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
There should be no limitations. Please produce a stack trace and send it to me. ср, 13 мая 2015, 18:34, Ankit Khurana ankit.khurana0...@gmail.com: Hi, I am trying to mirror files using lftp, the mirror path contains around 1 million files. The problem i am facing is that when i start the mirror, it attempts to do `ls -la` on the mirror site and then the process dies with segmentation fault producing a core dump. Is there a limitation on the number of files to be present at the mirror site? Thanks Ankit ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
[lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
Hi,
I am trying to mirror files using lftp, I have tested a use case where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully at the
destination, now if I update a.txt append some content and save it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_ behind. And
if i run the mirror multiple times it always downloads the changed a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they are
updated at the source. Like if a.txt is changed at source then lftp mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be
saved with a new time stamp. And mirror will consider these file to decide
whether to download or not.
Thanks,
Akshay
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
Can you provide a debug log? Why lftp cannot replace the old file?
2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com:
Hi,
I am trying to mirror files using lftp, I have tested a use case where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully at
the destination, now if I update a.txt append some content and save it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_ behind. And
if i run the mirror multiple times it always downloads the changed a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they are
updated at the source. Like if a.txt is changed at source then lftp mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be
saved with a new time stamp. And mirror will consider these file to decide
whether to download or not.
Thanks,
Akshay
--
Alexander.
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Bug in lftp mirror when using xfer:use-temp-file option
Here are the debug logs test.txt is the file that is changed:
lftp akshaygupta@192.168.128.124:~/Desktop/lftp mirror
--- TYPE A
--- 200 Type set to A.
--- PASV
--- 227 Entering Passive Mode (192,168,128,124,228,174)
Connecting data socket to (192.168.128.124) port 58542
Data connection established
--- LIST
--- 150 Opening ASCII mode data connection for '/bin/ls'.
Got EOF on data connection
Closing data socket
--- 226 Transfer complete.
--- TYPE I
--- 200 Type set to I.
--- MDTM .DS_Store
--- 213 20141027064918
--- MDTM search-2.html
--- 213 2014081726
--- MDTM search-3.html
--- 213 2014081744
--- MDTM search-4.html
--- 213 2014081745
--- MDTM search.html
--- 213 2014081722
--- MDTM settings.xml
--- 213 20140820095244
--- MDTM test.txt
--- 213 20141027075438
--- PASV
--- 227 Entering Passive Mode (192,168,128,124,228,176)
Connecting data socket to (192.168.128.124) port 58544
Data connection established
--- RETR test.txt
--- 150 Opening BINARY mode data connection for 'test.txt' (202 bytes).
Got EOF on data connection
Closing data socket
--- 226 Transfer complete.
Total: 1 directory, 7 files, 0 symlinks
Modified: 1 file, 0 symlinks
202 bytes transferred
To be removed: 0 directories, 1 file, 0 symlinks
Closing idle connection
--- QUIT
Timeout - reconnecting
--- 221-
--- Data traffic for this session was 172301 bytes in 7 files.
--- Total traffic for this session was 181339 bytes in 13 transfers.
--- 221 Thank you for using the FTP service on 192.168.128.124.
Closing control socket
~Akshay
On Mon, Oct 27, 2014 at 1:18 PM, Alexander Lukyanov lavv...@gmail.com
wrote:
Can you provide a debug log? Why lftp cannot replace the old file?
2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com:
Hi,
I am trying to mirror files using lftp, I have tested a use case where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully at
the destination, now if I update a.txt append some content and save it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_ behind. And
if i run the mirror multiple times it always downloads the changed a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they are
updated at the source. Like if a.txt is changed at source then lftp mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be
saved with a new time stamp. And mirror will consider these file to decide
whether to download or not.
Thanks,
Akshay
--
Alexander.
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
