Re: [lftp-devel] lftp-pre4.5.0-20131206

2013-12-13 Thread Justin Piszcz 
Alexander,

This latest version (lftp-pre4.5.0.20131214)--is showing improved
performance, including two sub-3 minute transfers for the dataset, also at
the server-level, CPU utilization increased (pure-ftpd) from 35% to 45-47%
(which isn't a problem but demonstrates lftp can pull the data faster now)..

Re/TOP:
31511 user20   0 38356 2324 1796 S  [43.9% CPU]  0.0   1:09.17
pure-ftpd 

Results:

After (newest code base)
Version: lftp-pre4.5.0.20131214
156505837056 bytes transferred in 177 seconds (843.64M/s)

10.33user 165.36system 2:56.93elapsed 99%CPU (0avgtext+0avgdata
3068maxresident)k
0inputs+0outputs (2major+1225minor)pagefaults 0swaps

156505837056 bytes transferred in 184 seconds (811.40M/s)

17.34user 165.72system 3:03.97elapsed 99%CPU (0avgtext+0avgdata
3140maxresident)k

156505837056 bytes transferred in 174 seconds (856.90M/s)

13.40user 159.24system 2:54.20elapsed 99%CPU (0avgtext+0avgdata
3080maxresident)k
0inputs+0outputs (5major+1243minor)pagefaults 0swaps


Before:
Version (lftp-pre4.5.0-20131206) and it was normal:
156505837056 bytes transferred in 187 seconds (796.08M/s)

22.34user 164.00system 3:07.52elapsed 99%CPU
(0avgtext+0avgdata2704maxresident)k
0inputs+0outputs (6major+2406minor)pagefaults 0swaps
156505837056 bytes transferred in 189 seconds (789.70M/s)

22.29user 165.31system 3:09.03elapsed 99%CPU
(0avgtext+0avgdata2684maxresident)k
0inputs+0outputs (8major+7492minor)pagefaults 0swaps
156505837056 bytes transferred in 187 seconds (799.54M/s)

Justin.


 -Original Message-
 From: Alexander V. Lukyanov [mailto:l...@netis.ru]
 Sent: Friday, December 13, 2013 8:39 AM
 To: Justin Piszcz
 Cc: lftp-devel@uniyar.ac.ru
 Subject: Re: lftp-pre4.5.0-20131206
 
 On Thu, Dec 12, 2013 at 04:24:14AM -0500, Justin Piszcz wrote:
  lftp is still reading 65536 buffers, but not larger than that:
 
 Ok, I have made another snapshot:
   http://lftp.yar.ru/ftp/devel/lftp-pre4.5.0.20131214.tar.gz
 
 Please give it a try. You can test various values for xfer:buffer-size
 and see which works better. By default it is still 0x1. For me that
 value works best for transfers from localhost. For some obscure reason
 0x2 bytes cannot be passed at once from localhost and are split into
 two packets of 131004 and 68 bytes (linux-3.11.4-201.fc19.x86_64).
 
 --
Alexander.

___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel

Re: [lftp-devel] lftp-pre4.5.0-20131206

2013-12-13 Thread Alexander V. Lukyanov 
On Fri, Dec 13, 2013 at 09:03:00AM -0500, Justin Piszcz wrote:
 This latest version (lftp-pre4.5.0.20131214)--is showing improved
 performance, including two sub-3 minute transfers for the dataset, also at
 the server-level, CPU utilization increased (pure-ftpd) from 35% to 45-47%
 (which isn't a problem but demonstrates lftp can pull the data faster now)..

I'm glad to hear that. Please use strace to see if system call pattern
is optimal.

 After (newest code base)
 Version: lftp-pre4.5.0.20131214
 
 10.33user 165.36system 2:56.93elapsed 99%CPU (0avgtext+0avgdata
 17.34user 165.72system 3:03.97elapsed 99%CPU (0avgtext+0avgdata
 13.40user 159.24system 2:54.20elapsed 99%CPU (0avgtext+0avgdata
 
 Before:
 Version (lftp-pre4.5.0-20131206) and it was normal:

 22.34user 164.00system 3:07.52elapsed 99%CPU
 22.29user 165.31system 3:09.03elapsed 99%CPU

I see that user time is decreased (probably because of buffer swapping
optimization), but system time is still the same. Please try larger
settings for xfer:buffer-size to see if they decrease system time. Use
strace to ensure lftp really uses those buffer sizes (it does here).

-- 
   Alexander.


  -Original Message-
  From: Alexander V. Lukyanov [mailto:l...@netis.ru]
  Sent: Friday, December 13, 2013 8:39 AM
  To: Justin Piszcz
  Cc: lftp-devel@uniyar.ac.ru
  Subject: Re: lftp-pre4.5.0-20131206
  
  On Thu, Dec 12, 2013 at 04:24:14AM -0500, Justin Piszcz wrote:
   lftp is still reading 65536 buffers, but not larger than that:
  
  Ok, I have made another snapshot:
  http://lftp.yar.ru/ftp/devel/lftp-pre4.5.0.20131214.tar.gz
  
  Please give it a try. You can test various values for xfer:buffer-size
  and see which works better. By default it is still 0x1. For me that
  value works best for transfers from localhost. For some obscure reason
  0x2 bytes cannot be passed at once from localhost and are split into
  two packets of 131004 and 68 bytes (linux-3.11.4-201.fc19.x86_64).
  
  --
 Alexander.
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel

Re: [lftp-devel] lftp-pre4.5.0-20131206

2013-12-13 Thread Justin Piszcz 
Hi,

Alexander, this is great!.

Response:

1. Yes, there are only 17 instances of EAGAIN for the first 10-20GB (test
run), the strace log is now filled with read and write activity.
2. The default buffer size remains 65536 (as shown in strace) (with the
default buffer size and with 0x2)

Testing with a higher buffer size as noted in the previous e-mail (0x2)

0x2:: Unbelievable!  With this buffer optimization I am seeing peaks of
970MiB/s++ on 3.12.x:
`...-11e0-a3b1-806e6f6e6963.vhd' at 78537702112 (50%) 977.35M/s eta:81s
[Receiv

RESULTS::
156505837056 bytes transferred in 159 seconds (939.49M/s)

5.86user 152.38system 2:38.88elapsed 99%CPU (0avgtext+0avgdata
3244maxresident)k
0inputs+0outputs (2major+1288minor)pagefaults 0swaps
156505837056 bytes transferred in 173 seconds (862.45M/s)

8.60user 162.37system 2:53.09elapsed 98%CPU (0avgtext+0avgdata
3388maxresident)k
0inputs+0outputs (12major+1337minor)pagefaults 0swaps
156505837056 bytes transferred in 169 seconds (885.41M/s)

6.37user 159.36system 2:48.60elapsed 98%CPU (0avgtext+0avgdata
3364maxresident)k
0inputs+0outputs (12major+1333minor)pagefaults 0swaps

Testing an even higher buffer to see if there is benefit above 0x2 vs
0x4, it does not appear to be the case:

So with lftp-pre4.5.0.20131214 and a buffer xfer-size of 0x2 clearly
gives the best results here, I'll switch over my production systems to this
for now and let you know if I have any problems, so far everything is
looking great, especially the performance..!

(with 0x4:)
156505837056 bytes transferred in 185 seconds (807.17M/s)

3.78user 179.07system 3:04.94elapsed 98%CPU (0avgtext+0avgdata
3744maxresident)k
0inputs+0outputs (6major+1451minor)pagefaults 0swaps
156505837056 bytes transferred in 191 seconds (779.53M/s)

4.05user 185.18system 3:11.50elapsed 98%CPU (0avgtext+0avgdata
3904maxresident)k
0inputs+0outputs (12major+1499minor)pagefaults 0swaps
156505837056 bytes transferred in 197 seconds (757.18M/s)

4.22user 190.66system 3:17.14elapsed 98%CPU (0avgtext+0avgdata
4000maxresident)k
0inputs+0outputs (8major+1482minor)pagefaults 0swaps

Justin.

 -Original Message-
 From: Alexander V. Lukyanov [mailto:l...@netis.ru]
 Sent: Friday, December 13, 2013 9:37 AM
 To: Justin Piszcz
 Cc: lftp-devel@uniyar.ac.ru
 Subject: Re: lftp-pre4.5.0-20131206
 
 On Fri, Dec 13, 2013 at 09:03:00AM -0500, Justin Piszcz wrote:
  This latest version (lftp-pre4.5.0.20131214)--is showing improved
  performance, including two sub-3 minute transfers for the dataset, also
at
  the server-level, CPU utilization increased (pure-ftpd) from 35% to
45-47%
  (which isn't a problem but demonstrates lftp can pull the data faster
now)..
 
 I'm glad to hear that. Please use strace to see if system call pattern
 is optimal.
 
  After (newest code base)
  Version: lftp-pre4.5.0.20131214
 
  10.33user 165.36system 2:56.93elapsed 99%CPU (0avgtext+0avgdata
  17.34user 165.72system 3:03.97elapsed 99%CPU (0avgtext+0avgdata
  13.40user 159.24system 2:54.20elapsed 99%CPU (0avgtext+0avgdata
 
  Before:
  Version (lftp-pre4.5.0-20131206) and it was normal:
 
  22.34user 164.00system 3:07.52elapsed 99%CPU
  22.29user 165.31system 3:09.03elapsed 99%CPU
 
 I see that user time is decreased (probably because of buffer swapping
 optimization), but system time is still the same. Please try larger
 settings for xfer:buffer-size to see if they decrease system time. Use
 strace to ensure lftp really uses those buffer sizes (it does here).
 
 --
Alexander.
 
 
   -Original Message-
   From: Alexander V. Lukyanov [mailto:l...@netis.ru]
   Sent: Friday, December 13, 2013 8:39 AM
   To: Justin Piszcz
   Cc: lftp-devel@uniyar.ac.ru
   Subject: Re: lftp-pre4.5.0-20131206
  
   On Thu, Dec 12, 2013 at 04:24:14AM -0500, Justin Piszcz wrote:
lftp is still reading 65536 buffers, but not larger than that:
  
   Ok, I have made another snapshot:
 http://lftp.yar.ru/ftp/devel/lftp-pre4.5.0.20131214.tar.gz
  
   Please give it a try. You can test various values for xfer:buffer-size
   and see which works better. By default it is still 0x1. For me
that
   value works best for transfers from localhost. For some obscure reason
   0x2 bytes cannot be passed at once from localhost and are split
into
   two packets of 131004 and 68 bytes (linux-3.11.4-201.fc19.x86_64).
  
   --
  Alexander.

___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel

Re: [lftp-devel] lftp-pre4.5.0-20131206

2013-12-13 Thread Alexander V. Lukyanov 
On Fri, Dec 13, 2013 at 10:06:34AM -0500, Justin Piszcz wrote:
 1. Yes, there are only 17 instances of EAGAIN for the first 10-20GB (test
 run), the strace log is now filled with read and write activity.

That's great.

 2. The default buffer size remains 65536 (as shown in strace) (with the
 default buffer size and with 0x2)
 
 Testing with a higher buffer size as noted in the previous e-mail (0x2)
 
 0x2:: Unbelievable!  With this buffer optimization I am seeing peaks of
 970MiB/s++ on 3.12.x:
 `...-11e0-a3b1-806e6f6e6963.vhd' at 78537702112 (50%) 977.35M/s eta:81s
 [Receiv
 
 RESULTS::
 156505837056 bytes transferred in 159 seconds (939.49M/s)
 
 5.86user 152.38system 2:38.88elapsed 99%CPU (0avgtext+0avgdata 
 3244maxresident)k

Ok, I think linux kernel should be optimized now - with that relation between
user and system time, if system calls are used efficiently enough.

-- 
   Alexander.
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel

Re: [lftp-devel] lftp-pre4.5.0-20131206

2013-12-12 Thread Justin Piszcz 


 -Original Message-
 From: Alexander V. Lukyanov [mailto:l...@netis.ru]
 Sent: Thursday, December 12, 2013 2:47 AM
 To: Justin Piszcz
 Cc: lftp-devel@uniyar.ac.ru
 Subject: Re: lftp-pre4.5.0-20131206
 
 On Wed, Dec 11, 2013 at 06:37:16AM -0500, Justin Piszcz wrote:
  I am using Debian Testing, I re-compiled without OpenSSL to perform the
  testing, but a heads up OpenSSL support is broken (see the error below):
 
  == COMPARISON: lftp-pre4.5.0.20131211
 
  156505837056 bytes transferred in 194 seconds (770.50M/s)
 
  23.76user 169.16system 3:13.73elapsed 99%CPU (0avgtext+0avgdata
  2812maxresident)k
  0inputs+0outputs (2major+1201minor)pagefaults 0swaps
 
 I don't see much of improvement, probably something is wrong. Can you
 please run strace and see if lftp reads data in larger than 65536 buffers?

lftp is still reading 65536 buffers, but not larger than that:

read(4, ::GetBuffer(int)\0\0\0\0\0\0\0\0\0\0\0__cd..., 65536) = 65536
read(5, 0x1da1de0, 65536)   = -1 EAGAIN (Resource temporarily
unavailable)
write(6, ::GetBuffer(int)\0\0\0\0\0\0\0\0\0\0\0__cd..., 65536) = 65536
read(4, ArrayListclass UnBCL::Dictionar..., 65536) = 65536
read(5, 0x1da1de0, 65536)   = -1 EAGAIN (Resource temporarily
unavailable)
write(6, scrubbed..., 65536) = 65536
read(4, scrubbed..., 65536) = 65536
read(5, scrubbed, 65536)   = -1 EAGAIN (Resource temporarily
unavailable)

$ lftp --version
LFTP | Version pre4.5.0.20131211 | Copyright (c) 1996-2013 Alexander V.
Lukyanov

 
 
  In file included from ../lib/md5.h:66:0,
   from FileAccess.cc:1120:
  ../lib/gl_openssl.h: In function 'void* md5_finish_ctx(md5_ctx*,
void*)':
  ../lib/gl_openssl.h:94:44: error: invalid conversion from 'void*' to
  'unsigned char*' [-fpermissive]
   { OPENSSL_FN (_Final) (res, (_gl_CTX *) ctx); return res; }
  ^
 
 Probably it is something new in gnulib. I'll try to fix it.
Ok-- thanks.

 
 --
Alexander.

___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel

[lftp-devel] lftp-pre4.5.0-20131206

2013-12-06 Thread Alexander V. Lukyanov 
A new development snapshot is available from
http://lftp.yar.ru/ftp/devel/lftp-pre4.5.0-20131206.tar.gz

* optimized cpu usage.
* fixed file info fetching from http server.

Please test.

-- 
   Alexander.

___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel

Re: [lftp-devel] lftp-pre4.5.0-20131206

2013-12-06 Thread Justin Piszcz 
Hello Alexander,

$ /vapp/bin/lftp --version
LFTP | Version pre4.5.0-20131206 | Copyright (c) 1996-2013 Alexander V.
Lukyanov

Performed same test under same conditions using the same file, 3-times:
156505837056 bytes transferred in 205 seconds (726.62M/s)

156505837056 bytes transferred in 193 seconds (774.69M/s)

156505837056 bytes transferred in 195 seconds (765.09M/s)


Justin.

 -Original Message-
 From: Alexander V. Lukyanov [mailto:l...@netis.ru]
 Sent: Friday, December 06, 2013 7:19 AM
 To: lftp-devel@uniyar.ac.ru
 Cc: jpis...@lucidpixels.com
 Subject: lftp-pre4.5.0-20131206
 
 A new development snapshot is available from
   http://lftp.yar.ru/ftp/devel/lftp-pre4.5.0-20131206.tar.gz
 
 * optimized cpu usage.
 * fixed file info fetching from http server.
 
 Please test.
 
 --
Alexander.


___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel

Re: [lftp-devel] lftp-pre4.5.0-20131206

2013-12-06 Thread Lars Viklund 
On Fri, Dec 06, 2013 at 04:19:22PM +0400, Alexander V. Lukyanov wrote:
 A new development snapshot is available from
   http://lftp.yar.ru/ftp/devel/lftp-pre4.5.0-20131206.tar.gz
 
 * optimized cpu usage.
 * fixed file info fetching from http server.
 
 Please test.

I grabbed this version to see if it reintroduced a problem I had with
one of my servers (server does not support HEAD when redirected), but
ran into a more severe problem.

Whenever I've connected to a https server, lftp segfaults at shutdown,
built with: Readline 6.2, GnuTLS 2.12.23, zlib 1.2.8

It is not reproducible with 4.4.13 built with the same set of libraries.

Stack trace:
8
lftp :~ open https://www.acc.umu.se/
cd ok, cwd=/
lftp www.acc.umu.se:/ exit

Program received signal SIGSEGV, Segmentation fault.
0x0047f447 in Log::WillOutput (this=0x0, l=13) at log.cc:47
47 if(!enabled || llevel || output==-1)
(gdb) bt
#0  0x0047f447 in Log::WillOutput (this=0x0, l=13) at log.cc:47
#1  0x0047f7d9 in Log::Format (this=0x0, l=13, f=0x512054
GNUTLS: %s) at log.cc:108
#2  0x004e56f3 in lftp_ssl_gnutls_log_func (level=4,
msg=0x775fd0 REC[0xb73e10]: Epoch #1 freed\n)
at lftp_ssl.cc:222
#3  0x77923d7f in _gnutls_log () from
/usr/lib/x86_64-linux-gnu/libgnutls.so.26
#4  0x7793387e in ?? () from
/usr/lib/x86_64-linux-gnu/libgnutls.so.26
#5  0x7793b8e4 in gnutls_deinit () from
/usr/lib/x86_64-linux-gnu/libgnutls.so.26
#6  0x004e5ba5 in lftp_ssl_gnutls::~lftp_ssl_gnutls
(this=0x7a3760, __in_chrg=optimized out)
at lftp_ssl.cc:324
#7  0x004b6d23 in Reflftp_ssl_gnutls::~Ref (this=0x76dff0,
__in_chrg=optimized out) at Ref.h:34
#8  0x004bd2de in Http::Connection::~Connection (this=0x76dfd0,
__in_chrg=optimized out) at Http.cc:125
#9  0x004c7875 in RefHttp::Connection::operator=
(this=0x78ff08, p=0x0) at Ref.h:35
#10 0x004bd956 in Http::Disconnect (this=0x78fb20) at
Http.cc:219
#11 0x004bd71f in Http::~Http (this=0x78fb20,
__in_chrg=optimized out) at Http.cc:198
#12 0x004c6c63 in Https::~Https (this=0x78fb20,
__in_chrg=optimized out) at Http.cc:2365
#13 0x004c6c98 in Https::~Https (this=0x78fb20,
__in_chrg=optimized out) at Http.cc:2367
#14 0x0046316b in SMTask::CollectGarbage () at SMTask.cc:187
#15 0x0046350d in SMTask::Cleanup () at SMTask.cc:273
#16 0x0040836e in main (argc=1, argv=0x7fffe4f8) at
lftp.cc:597
8

-- 
Lars Viklund | z...@acc.umu.se
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel

Re: [lftp-devel] lftp-pre4.5.0-20131206

2013-12-06 Thread Alexander V. Lukyanov 
BTW, I have just fixed a failed assert in heap code (removing the last
heap item is a special case that was not handled properly).

-- 
   Alexander.
___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel

Re: [lftp-devel] lftp-pre4.5.0-20131206

2013-12-06 Thread Justin Piszcz 


 -Original Message-
 From: Alexander V. Lukyanov [mailto:l...@netis.ru]
 Sent: Friday, December 06, 2013 8:39 AM
 To: Justin Piszcz
 Cc: lftp-devel@uniyar.ac.ru
 Subject: Re: lftp-pre4.5.0-20131206
 
 On Fri, Dec 06, 2013 at 08:31:38AM -0500, Justin Piszcz wrote:
  Hello Alexander,
 
  $ /vapp/bin/lftp --version
  LFTP | Version pre4.5.0-20131206 | Copyright (c) 1996-2013 Alexander V.
  Lukyanov
 
  Performed same test under same conditions using the same file, 3-times:
  156505837056 bytes transferred in 205 seconds (726.62M/s)
 
  156505837056 bytes transferred in 193 seconds (774.69M/s)
 
  156505837056 bytes transferred in 195 seconds (765.09M/s)
 
 Please use time to measure cpu time, compare it with previous versions.
 Also strace should show that lftp uses larger read/write buffers.
 
 --
Alexander.

Hello,

A definite improvement, see the results below:

New code base (lftp-pre4.5.0-20131206):
156505837056 bytes transferred in 180 seconds (830.33M/s)

21.75user 157.61system 2:59.77elapsed 99%CPU (0avgtext+0avgdata
2688maxresident)k
0inputs+0outputs (1major+1578minor)pagefaults 0swaps
156505837056 bytes transferred in 184 seconds (813.29M/s)

22.03user 161.07system 3:03.54elapsed 99%CPU (0avgtext+0avgdata
2700maxresident)k
0inputs+0outputs (12major+1639minor)pagefaults 0swaps
156505837056 bytes transferred in 186 seconds (803.33M/s)

23.13user 162.26system 3:05.82elapsed 99%CPU (0avgtext+0avgdata
2692maxresident)k
0inputs+0outputs (18major+1241minor)pagefaults 0swaps

Previous release version (4.4.13):
156505837056 bytes transferred in 198 seconds (753.69M/s)

32.59user 165.19system 3:18.04elapsed 99%CPU (0avgtext+0avgdata
3148maxresident)k
0inputs+0outputs (0major+1267minor)pagefaults 0swaps
156505837056 bytes transferred in 198 seconds (753.35M/s)

33.16user 164.73system 3:18.16elapsed 99%CPU (0avgtext+0avgdata
3144maxresident)k
0inputs+0outputs (12major+1255minor)pagefaults 0swaps
156505837056 bytes transferred in 207 seconds (720.27M/s)

33.42user 173.45system 3:27.26elapsed 99%CPU (0avgtext+0avgdata
3076maxresident)k
0inputs+0outputs (12major+1238minor)pagefaults 0swaps

Justin.

___
lftp-devel mailing list
lftp-devel@uniyar.ac.ru
http://univ.uniyar.ac.ru/mailman/listinfo/lftp-devel