Re: lftp 4.1.0 patches in Owl
On Fri, Nov 26, 2010 at 02:46:17AM +0300, Solar Designer wrote: glob.c:23:1: warning: _GL_ARG_NONNULL redefined ./string.h:324:1: warning: this is the location of the previous definition Please send this one to gnulib maintainers. Also, JFYI, the -owl-warnings patch was by Vasiliy (CC'ed), not by me. I've corrected ChangeLog. -- Alexander.
Re: [PATCH] Add xfer:log-dir parameter to write transfer_log in arbitrary dir.
On Tue, Feb 01, 2011 at 10:42:22AM +0300, Timur Sufiev wrote: I think it would be convenient to specify arbitrary directory for storing transfer_log (it is useful when several processes using lftp I think it is better to specify a file name rather than a directory. Use xfer:log-file setting name. -- Alexander.
Re: [PATCH] some servers needs force SSL3.0 when lftp is built with gnutls
On Thu, Jul 01, 2010 at 05:17:30PM +0200, Jiri Skala wrote: Hi, some servers needs force SSL3.0 when lftp is built with gnutls. There is mandatory setting to enable this forcing: Patch applied with some changes. Thanks! -- Alexander.
Re: [PATCH] Add xfer:log-dir parameter to write transfer_log in arbitrary dir.
On Thu, Feb 10, 2011 at 02:07:35PM +0300, Timur Sufiev wrote: On Tue, Feb 01, 2011 at 10:42:22AM +0300, Timur Sufiev wrote: I think it would be convenient to specify arbitrary directory for storing transfer_log (it is useful when several processes using lftp I think it is better to specify a file name rather than a directory. Use xfer:log-file setting name. Here it is. Thanks! I have applied the patch with some modifications. -- Alexander.
Re: [BUG] server time zone ignored in some situations
On Sat, Mar 05, 2011 at 05:22:32PM -0800, Chris Moore wrote: This ends up using Ftp::ConvertFtpDate() in ftpclass.cc, which ignores the timezone, instead doing: return mktime_from_utc(tm); This is correct. MDTM command must return GMT time. From the draft-ietf-ftpext-mlst-16.txt: Time values are always represented in UTC (GMT), and in the Gregorian calendar regardless of what calendar may have been in use at the date and time indicated at the location of the server-PI. So the bug is on the server side. -- Alexander.
Re: buffer overflow with certain ssh servers
On Mon, Mar 14, 2011 at 11:08:49AM +, Sean Purdy wrote: On Mon, 14 Mar 2011, Alexander V. Lukyanov said: On Fri, Mar 11, 2011 at 04:09:17PM +, Sean Purdy wrote: I'm getting a buffer overflow when connecting to certain ssh servers with lftp. Please recompile lftp with debug (configure --with-debug) and send the backtrace. OK, tried that. Unfortunately the debug version worked fine! Please configure as usual, edit src/Makefile and remove -s option from LDFLAGS variable, then compile, run and send the backtrace. -- Alexander.
Re: buffer overflow with certain ssh servers
On Thu, Mar 17, 2011 at 01:37:53PM +, Sean Purdy wrote: On Thu, 17 Mar 2011, Alexander V. Lukyanov said: On Mon, Mar 14, 2011 at 11:08:49AM +, Sean Purdy wrote: On Mon, 14 Mar 2011, Alexander V. Lukyanov said: On Fri, Mar 11, 2011 at 04:09:17PM +, Sean Purdy wrote: I'm getting a buffer overflow when connecting to certain ssh servers with lftp. Please recompile lftp with debug (configure --with-debug) and send the backtrace. OK, tried that. Unfortunately the debug version worked fine! Please configure as usual, edit src/Makefile and remove -s option from LDFLAGS variable, then compile, run and send the backtrace. Sorry - LDFLAGS has nothing set in Makefile or src/Makefile for version 4.2.0: What file src/lftp says? If not stripped, run that version and produce a backtrace. -- Alexander.
Re: [PATCH] Reconfig() is not executed after set commands when using lftp -c or lftp -e
On Wed, Jun 01, 2011 at 06:50:53PM +0200, Fernando Gutierrez wrote:
**Problem:
Certain options like cmd:parallel won't work when used with lftp -c
because Reconfig is never called. This happens to all the options
This patch should fix the problem properly. ResMgr::Set calls
Reconfig methods of all ResClient instances automatically.
--
Alexander.
Index: CmdExec.h
===
RCS file: /home/lav/cvsroot/lftp/src/CmdExec.h,v
retrieving revision 1.74
diff -u -p -r1.74 CmdExec.h
--- CmdExec.h 29 Apr 2011 04:58:27 - 1.74
+++ CmdExec.h 8 Jun 2011 13:12:54 -
@@ -59,7 +59,7 @@ public:
extern CmdFeeder *lftp_feeder; // feeder to use after 'lftp' command
-class CmdExec : public SessionJob
+class CmdExec : public SessionJob, public ResClient
{
public:
// current command data
Re: [PATCH] Ideas about the exit code of lftp
On Wed, Jun 01, 2011 at 06:58:29PM +0200, Fernando Gutierrez wrote: When fixing another bug I found out the exit code lftp returned to shellscripts wasn't what I was expecting it to be. Patch applied. Thanks! -- Alexander.
Re: [lftp-devel] lftp find extension { -name ? -type ? -mtime ? like local GNU find }
On Mon, Jan 09, 2012 at 10:57:04AM -0800, Cheng Renquan wrote: I found current lftp find command only list dir/files recursively, it's not so useful, What I expect is find could support like local GNU find that support -name ? -type ? -mtime ? those options ? -name option can be replaced by find | grep combination. Other options are not implemented yet. I wonder if lftp current roadmap has a plan to support that, or I could contribute some time on that, may be able to borrow some code from GNU find, since both GPL licensed, may be not that hard; Patches are always welcome. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] mirror -n -R over webdav ignores ready-mirrored files (#19)
On Mon, Jun 18, 2012 at 11:47:10AM -0700, Christian Harms wrote:
If found the difference. The WebDAV server cant handle the
allprop-request-body. Try the PROPFIND without a request-body:
Please try this patch.
--
Alexander.
diff --git a/src/FileSet.cc b/src/FileSet.cc
index 3d42dde..8b932d0 100644
--- a/src/FileSet.cc
+++ b/src/FileSet.cc
@@ -935,7 +935,7 @@ void FileInfo::MakeLongName()
int w=20-strlen(usergroup);
if(w1)
w=1;
- char size_str[20];
+ char size_str[21];
if(definedSIZE)
snprintf(size_str,sizeof(size_str),%*lld,w,(long long)size);
else
diff --git a/src/Http.cc b/src/Http.cc
index a8c4d2d..447effe 100644
--- a/src/Http.cc
+++ b/src/Http.cc
@@ -471,6 +471,9 @@ void Http::SendRequest(const char *connection,const char *f)
propfind xmlns=\DAV:\
allprop/
/propfind\r\n;
+ if(!QueryBool(use-allprop,hostname))
+ allprop=;
+ int allprop_len=strlen(allprop);
xstring pfile;
if(proxy !https)
@@ -604,7 +607,8 @@ void Http::SendRequest(const char *connection,const char *f)
SendMethod(PROPFIND,efile);
Send(Depth: 0\r\n); // no directory listing required
Send(Content-Type: text/xml\r\n);
- Send(Content-Length: %d\r\n,(int)strlen(allprop));
+ if(allprop_len0)
+ Send(Content-Length: %d\r\n,allprop_len);
}
else
SendMethod(HEAD,efile);
@@ -627,7 +631,8 @@ void Http::SendRequest(const char *connection,const char *f)
SendMethod(PROPFIND,efile);
Send(Depth: 1\r\n); // directory listing required
Send(Content-Type: text/xml\r\n);
-Send(Content-Length: %d\r\n,(int)strlen(allprop));
+if(allprop_len0)
+ Send(Content-Length: %d\r\n,allprop_len);
pos=0;
}
break;
diff --git a/src/HttpDirXML.cc b/src/HttpDirXML.cc
index c20a27f..59bb9dd 100644
--- a/src/HttpDirXML.cc
+++ b/src/HttpDirXML.cc
@@ -33,11 +33,15 @@ struct xml_context
xarray_sxstring_c stack;
RefFileSet fs;
RefFileInfo fi;
- xstring_c base_dir;
+ xstring base_dir;
void push(const char *);
void pop();
- void set_base_dir(const char *d) { base_dir.set(d); }
+ void set_base_dir(const char *d) {
+ base_dir.set(d);
+ if(base_dir.length()1)
+base_dir.chomp('/');
+ }
const char *top(int i=0) { return stack.count()i ?
stack[stack.count()-i-1].get() : 0; }
};
@@ -62,6 +66,7 @@ static void start_handle(void *data, const char *el, const
char **attr)
else if(!strcmp(ctx-top(), DAV:collection))
{
ctx-fi-SetType(ctx-fi-DIRECTORY);
+ ctx-fi-SetMode(0755);
}
}
static void end_handle(void *data, const char *el)
@@ -95,7 +100,8 @@ static void chardata_handle(void *data, const char
*chardata, int len)
int s_len=strlen(s);
if(s_len0 s[s_len-1]=='/')
{
-s[s_len-1]=0;
+if(s_len1)
+ s[--s_len]=0;
ctx-fi-SetType(ctx-fi-DIRECTORY);
ctx-fi-SetMode(0755);
}
@@ -106,12 +112,7 @@ static void chardata_handle(void *data, const char
*chardata, int len)
}
if(s[0]=='/' s[1]=='~')
s++;
- const char *name;
- if(ctx-base_dir !strcmp(s,ctx-base_dir))
-name=.;
- else
-name=basename_ptr(s);
- ctx-fi-SetName(name);
+ ctx-fi-SetName(ctx-base_dir.eq(s) ? . : basename_ptr(s));
}
else if(!strcmp(tag,DAV:getcontentlength))
{
diff --git a/src/resource.cc b/src/resource.cc
index 959b8e7..06d1d9c 100644
--- a/src/resource.cc
+++ b/src/resource.cc
@@ -253,6 +253,7 @@ static ResType lftp_vars[] = {
{hftp:use-head,yes, ResMgr::BoolValidate,0},
{hftp:use-mkcol, no,ResMgr::BoolValidate,0},
{hftp:use-propfind,no,ResMgr::BoolValidate,0},
+ {hftp:use-allprop, no,ResMgr::BoolValidate,0},
{hftp:use-type,yes, ResMgr::BoolValidate,0},
{http:accept, */*, 0,0},
{http:accept-language, , 0,0},
@@ -263,6 +264,7 @@ static ResType lftp_vars[] = {
{http:proxy, , HttpProxyValidate,0},
{http:use-mkcol, yes, ResMgr::BoolValidate,0},
{http:use-propfind,no,ResMgr::BoolValidate,0},
+ {http:use-allprop, no,ResMgr::BoolValidate,0},
{http:user-agent, PACKAGE/VERSION,0,0},
{http:cookie, , 0,0},
{http:set-cookies, no,0,0},
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] lftp patch for dnssec checks
On Wed, Sep 05, 2012 at 11:42:14AM -0400, Robert Story wrote: - update README.dnssec - fix a typo in error message and add a warning for untrusted domains if dns:strict-dnssec is not set Patch applied. Thanks! -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
[lftp-devel] lftp-pre4.5.0-20131206
A new development snapshot is available from http://lftp.yar.ru/ftp/devel/lftp-pre4.5.0-20131206.tar.gz * optimized cpu usage. * fixed file info fetching from http server. Please test. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] lftp-pre4.5.0-20131206
BTW, I have just fixed a failed assert in heap code (removing the last heap item is a special case that was not handled properly). -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] lftp-pre4.5.0-20131206
On Fri, Dec 13, 2013 at 09:03:00AM -0500, Justin Piszcz wrote: This latest version (lftp-pre4.5.0.20131214)--is showing improved performance, including two sub-3 minute transfers for the dataset, also at the server-level, CPU utilization increased (pure-ftpd) from 35% to 45-47% (which isn't a problem but demonstrates lftp can pull the data faster now).. I'm glad to hear that. Please use strace to see if system call pattern is optimal. After (newest code base) Version: lftp-pre4.5.0.20131214 10.33user 165.36system 2:56.93elapsed 99%CPU (0avgtext+0avgdata 17.34user 165.72system 3:03.97elapsed 99%CPU (0avgtext+0avgdata 13.40user 159.24system 2:54.20elapsed 99%CPU (0avgtext+0avgdata Before: Version (lftp-pre4.5.0-20131206) and it was normal: 22.34user 164.00system 3:07.52elapsed 99%CPU 22.29user 165.31system 3:09.03elapsed 99%CPU I see that user time is decreased (probably because of buffer swapping optimization), but system time is still the same. Please try larger settings for xfer:buffer-size to see if they decrease system time. Use strace to ensure lftp really uses those buffer sizes (it does here). -- Alexander. -Original Message- From: Alexander V. Lukyanov [mailto:l...@netis.ru] Sent: Friday, December 13, 2013 8:39 AM To: Justin Piszcz Cc: lftp-devel@uniyar.ac.ru Subject: Re: lftp-pre4.5.0-20131206 On Thu, Dec 12, 2013 at 04:24:14AM -0500, Justin Piszcz wrote: lftp is still reading 65536 buffers, but not larger than that: Ok, I have made another snapshot: http://lftp.yar.ru/ftp/devel/lftp-pre4.5.0.20131214.tar.gz Please give it a try. You can test various values for xfer:buffer-size and see which works better. By default it is still 0x1. For me that value works best for transfers from localhost. For some obscure reason 0x2 bytes cannot be passed at once from localhost and are split into two packets of 131004 and 68 bytes (linux-3.11.4-201.fc19.x86_64). -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] lftp-pre4.5.0-20131206
On Fri, Dec 13, 2013 at 10:06:34AM -0500, Justin Piszcz wrote: 1. Yes, there are only 17 instances of EAGAIN for the first 10-20GB (test run), the strace log is now filled with read and write activity. That's great. 2. The default buffer size remains 65536 (as shown in strace) (with the default buffer size and with 0x2) Testing with a higher buffer size as noted in the previous e-mail (0x2) 0x2:: Unbelievable! With this buffer optimization I am seeing peaks of 970MiB/s++ on 3.12.x: `...-11e0-a3b1-806e6f6e6963.vhd' at 78537702112 (50%) 977.35M/s eta:81s [Receiv RESULTS:: 156505837056 bytes transferred in 159 seconds (939.49M/s) 5.86user 152.38system 2:38.88elapsed 99%CPU (0avgtext+0avgdata 3244maxresident)k Ok, I think linux kernel should be optimized now - with that relation between user and system time, if system calls are used efficiently enough. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Fix posix_fallocate m4 check for cross-compilation.
I have applied your patch. Thanks! -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
[lftp-devel] lftp snapshot 20140205 available
http://lftp.yar.ru/ftp/devel/lftp-4.4.99.20140205.tar.gz Please test. CPU usage comparision for 4929376768 bytes transferred via ftp over 100Mb/s ethernet: 4.4.15 4.4.99.20140205 Ratio -- -- - user1m0.437suser0m15.093s 0.25 sys 0m15.860s sys 0m6.734s0.42 -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
[lftp-devel] lftp snapshot 20140513 is available
http://lftp.yar.ru/ftp/devel/lftp-4.4.99.20140513.tar.gz Please test. Most notable changes: * show last disconnect cause for a few seconds in the session status. * improved torrent DHT search. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
[lftp-devel] lftp snapshot 20140522 is available
http://lftp.yar.ru/ftp/devel/lftp-4.4.99.20140522.tar.gz Please test. Most notable changes: * new setting xfer:max-log-size to limit transfer log size. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] possible bug in lftp mirror using xfer:use-temp-file option
On Tue, Sep 30, 2014 at 10:35:37PM +0530, akshay gupta wrote: Its causing unpredictable behaviour in case of fuse mounted file system. Please look into it if it can be resolved. Please test the new snapshot http://lftp.yar.ru/ftp/devel/lftp-4.5.5.20141003.tar.gz -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
On Mon, Oct 27, 2014 at 01:30:42PM +0530, akshay gupta wrote:
--- RETR test.txt
--- 150 Opening BINARY mode data connection for 'test.txt' (202 bytes).
Got EOF on data connection
Closing data socket
--- 226 Transfer complete.
So the file cannot be renamed locally, right? Is there a permissions problem?
--
Alexander.
On Mon, Oct 27, 2014 at 1:18 PM, Alexander Lukyanov lavv...@gmail.com
wrote:
Can you provide a debug log? Why lftp cannot replace the old file?
2014-10-27 10:44 GMT+03:00 akshay gupta akshaygupta...@gmail.com:
Hi,
I am trying to mirror files using lftp, I have tested a use case where a
file at the source machine gets modified but the file name remains the
same. After mirroring, lftp is unable to replace the old file and also
leaves a temporary file.
For example. If source have file a.txt and its mirrored successfully at
the destination, now if I update a.txt append some content and save it with
same name. If I run lftp mirror again then its not able to replace the
existing a.txt but it also leaves the temp file a.txt_transit_ behind. And
if i run the mirror multiple times it always downloads the changed a.txt, i
can see this from the lftp logs.
Suggestion: Will it make sense to keep versions of the files as they are
updated at the source. Like if a.txt is changed at source then lftp mirror
will save it as a.txt_{TIMESTAMP}_. Every time a file is updated it will be
saved with a new time stamp. And mirror will consider these file to decide
whether to download or not.
Thanks,
Akshay
--
Alexander.
___
lftp mailing list
l...@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Bug in lftp mirror when using xfer:use-temp-file option
On Mon, Oct 27, 2014 at 05:53:19PM +0530, akshay gupta wrote: now its producing 0 byte files and i am also getting input/output error here are the debug logs: copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt): Input/output error Which filesystem do you use? Maybe it is necessary to unlink target file before renaming, although rename should do it automatically. If newpath already exists it will be atomically replaced (subject to a few conditions; see ERRORS below), so that there is no point at which another process attempting to access newpath will find it missing. And man page for rename(2) does not list EIO error. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Compressing data at destination machine
On Tue, Nov 25, 2014 at 06:44:32PM +0530, akshay gupta wrote: I want to store the data that is being pulled by lftp mirror job in a compressed format. Is it possible that mirror function can check file download status from inside of that tar, zip etc. Is it possible to implement such a functionality in lftp? I don't think mirror will be able to extract compressed archives, but you can download the archive and extract it at the same time, like this: lftp cat file.tar.gz | tar xv -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Compressing data at destination machine
On Fri, Nov 28, 2014 at 11:55:07AM +0530, akshay gupta wrote: I dont want lftp to extract data from archives, I want it to compress it into archive so that it occupies less space at the destination side. Is this feasible to do in lftp? Take a look at zipfs: https://github.com/hanwen/go-fuse Unfortunately it is read-only. So you can mount a zip file, run mirror over it with --script option, run the script with an empty target directory, then run zip to update the archive. -- Alexander. ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] [lftp] Compressing data at destination machine
On Tue, Dec 02, 2014 at 09:53:08PM +0530, akshay gupta wrote: I have tried using it. It does not update the files appearing in the mount point if the files in zip changes. And it also does not retains modification time of the file at the mount location this could affect the mirror. You have to unmount it before changing the zip file. Do you have any other similar opensource software in mind that could also support writes. You can try the other way: ftpfs. -- Alexander. On Fri, Nov 28, 2014 at 1:06 PM, Alexander V. Lukyanov l...@netis.ru wrote: On Fri, Nov 28, 2014 at 11:55:07AM +0530, akshay gupta wrote: I dont want lftp to extract data from archives, I want it to compress it into archive so that it occupies less space at the destination side. Is this feasible to do in lftp? Take a look at zipfs: https://github.com/hanwen/go-fuse Unfortunately it is read-only. So you can mount a zip file, run mirror over it with --script option, run the script with an empty target directory, then run zip to update the archive. -- Alexander. ___ lftp mailing list l...@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
Re: [lftp-devel] Handling for small files
On Tue, Feb 24, 2015 at 11:07:23AM +, akshay gupta wrote: I am using fish protocol. If possible, try to use sftp. Fish is not quite the most efficient option. Even ftp is better, especially with pipelining (set ftp:sync-mode off). -- Alexander. On Tue, Feb 24, 2015, 4:23 PM Alexander Lukyanov lavv...@gmail.com wrote: Which protocol do you use? вт, 24 февр. 2015, 13:48, akshay gupta akshaygupta...@gmail.com: Hi, I am trying to transfer about 80GB of data using mirror option. File sizes are very small about 0.5 KB to 3 KB per file. Transfer is taking a lot of time which I am attributing to small file size. Is there any option in lftp which I can use to increase the transfer speed. May be some sort of compression option like rsync -av*z.* I know about the parallel file transfer option but I cant use it because I want to maintain the order of file transfer. Please help. Thanks, Akshay ___ lftp-devel mailing list lftp-devel@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel
