Re: wget default behavior [was Re: working on patch to limit to percent of bandwidth]

[Matthias Vill]

Tony Godshall wrote:
 If it was me, I'd have it default to backing off to 95% by default and
 have options for more aggressive behavior, like the multiple
 connections, etc.

I don't like a default back-off rule. I often encounter downloads with
often changing download speeds. The idea that the first few seconds I
only have a quite bad speed and could get much more out of it is just
not satisfying.

 I'm surprised multiple connections would buy you anything, though.  I
 guess I'll take a look through the archives and see what the argument
 is.  Does one tcp connection back off on a lost packet and the other
 one gets to keep going?  Hmmm.

I guess you get improvements if e.g. on your side you have more free
bandwidth than on the source-side. Having two connections than means,
that you get almost twice the download speed, because you have two
connections competing for free bandwidth and ideally every connection
made two the sever is equally fast.

So in cases, where you are the only one connecting, you probably win
nothing.

Greetings

Matthias

Re: wget default behavior [was Re: working on patch to limit to percent of bandwidth]

[Tony Godshall]

On 10/17/07, Matthias Vill [EMAIL PROTECTED] wrote:
 Tony Godshall wrote:
  If it was me, I'd have it default to backing off to 95% by default and
  have options for more aggressive behavior, like the multiple
  connections, etc.

 I don't like a default back-off rule. I often encounter downloads with
 often changing download speeds. The idea that the first few seconds I
 only have a quite bad speed and could get much more out of it is just
 not satisfying.

You might be surprised, but I totally agree with you.  A default
backoff rule would only make sense if the measuring was better.  E.g.
a periodic ramp-up/back-off behavior to achieve 95% of the maximum
measured rate.

  I'm surprised multiple connections would buy you anything, though.  I
  guess I'll take a look through the archives and see what the argument
  is.  Does one tcp connection back off on a lost packet and the other
  one gets to keep going?  Hmmm.

 I guess you get improvements if e.g. on your side you have more free
 bandwidth than on the source-side. Having two connections than means,
 that you get almost twice the download speed, because you have two
 connections competing for free bandwidth and ideally every connection
 made two the sever is equally fast.

 So in cases, where you are the only one connecting, you probably win
 nothing.

Ah, I get it.  People want to defeat sender rate-limiting or other QOS
controls.

The opposite of nice.  We could call it --mean-mode.  Or --meanness n,
where n=2 means I want to have two threads/connections, i.e. twice as
mean as the default.

No, well, actually, I guess there can be cases where bad upstream
configurations result in a situation where more connections don't
necessarily mean one is taking more than one's fair share of
bandwidth, but I bet this option will be result in more harm than
good.  Perhaps it should be one of those things that one can do
oneself if one must but is generally frowned upon (like making a
version of wget that ignores robots.txt).

TG

Re: wget default behavior [was Re: working on patch to limit to percent of bandwidth]

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

(Accidentally sent private reply).

Tony Godshall wrote:
 On 10/17/07, Matthias Vill [EMAIL PROTECTED] wrote:
 Tony Godshall wrote:
 If it was me, I'd have it default to backing off to 95% by default and
 have options for more aggressive behavior, like the multiple
 connections, etc.
 I don't like a default back-off rule. I often encounter downloads with
 often changing download speeds. The idea that the first few seconds I
 only have a quite bad speed and could get much more out of it is just
 not satisfying.

 You might be surprised, but I totally agree with you.  A default
 backoff rule would only make sense if the measuring was better.  E.g.
 a periodic ramp-up/back-off behavior to achieve 95% of the maximum
 measured rate.

 I'm surprised multiple connections would buy you anything, though.  I
 guess I'll take a look through the archives and see what the argument
 is.  Does one tcp connection back off on a lost packet and the other
 one gets to keep going?  Hmmm.
 I guess you get improvements if e.g. on your side you have more free
 bandwidth than on the source-side. Having two connections than means,
 that you get almost twice the download speed, because you have two
 connections competing for free bandwidth and ideally every connection
 made two the sever is equally fast.

 So in cases, where you are the only one connecting, you probably win
 nothing.

 Ah, I get it.  People want to defeat sender rate-limiting or other QOS
 controls.

 The opposite of nice.  We could call it --mean-mode.  Or --meanness n,
 where n=2 means I want to have two threads/connections, i.e. twice as
 mean as the default.

Oh, think you misunderstand. I have no intention of providing such a
thing. That's what download accelerators are for, and as much as some
people may want Wget to be one, I'm against it.

However, multiple simultaneous connections to _different_ hosts, could
be very beneficial, as latency for one server won't mean we sit around
waiting for it before downloading from others. And, up to two
connections to the same host will also be supported: but probably only
for separate downloads (that way, we can be sending requests on one
connection while we're downloading on another). The HTTP spec says that
clients _should_ have a maximum of two connections to any one host, so
we appear to be justified to do that. However, it will absolutely not be
done by default. Among other things, multiple connections will destroy
the way we currently do logging, which in and of itself is a good reason
not to do it, apart from niceness.

 No, well, actually, I guess there can be cases where bad upstream
 configurations result in a situation where more connections don't
 necessarily mean one is taking more than one's fair share of
 bandwidth, but I bet this option will be result in more harm than
 good.  Perhaps it should be one of those things that one can do
 oneself if one must but is generally frowned upon (like making a
 version of wget that ignores robots.txt).

You do know that Wget already can be configured to ignore robots.txt, right?

Yeah, I'm already cringing at the idea that people will alter the two
connections per host limit to higher values. Even if we limit it to
_one_ per host, though, as long as we're including support for multiple
connections of any sort, it'd be easy to modify Wget to allow them for
the same host.

And multiple connections to multiple hosts will be obviously beneficial,
to avoid bottlenecks and the like. Plus, the planned (plugged-in)
support for Metalink, using multiple connections to different hosts to
obtain the _same_ file, could be very nice for large and/or very popular
downloads.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHFmYO7M8hyUobTrERCEDOAJ0T/a70fraMdvQMGgIGPl2XXoprHACcDlI0
eqx0DfnsJ+NUAkzJhUMQS68=
=EGay
-END PGP SIGNATURE-

Re: wget default behavior [was Re: working on patch to limit to percent of bandwidth]

[Tony Godshall]

On 10/13/07, Micah Cowan [EMAIL PROTECTED] wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256


  On 10/13/07, Tony Godshall [EMAIL PROTECTED] wrote:
  OK, so let's go back to basics for a moment.
 
  wget's default behavior is to use all available bandwidth.
 
  Is this the right thing to do?
 
  Or is it better to back off a little after a bit?

 Heh. Well, some people are saying that Wget should support accelerated
 downloads; several connections to download a single resource, which can
 sometimes give a speed increase at the expense of nice-ness.

 So you could say we're at a happy medium between those options! :)

 Actually, Wget probably will get support for multiple simultaneous
 connections; but number of connections to one host will be limited to a
 max of two.

 It's impossible for Wget to know how much is appropriate to back off,
 and in most situations I can think of, backing off isn't appropriate.

 In general, though, I agree that Wget's policy should be nice by default.

If it was me, I'd have it default to backing off to 95% by default and
have options for more aggressive behavior, like the multiple
connections, etc.

I'm surprised multiple connections would buy you anything, though.  I
guess I'll take a look through the archives and see what the argument
is.  Does one tcp connection back off on a lost packet and the other
one gets to keep going?  Hmmm.

 Josh Williams wrote:
  That's one of the reasons I believe this
  should be a module instead, because it's more or less a hack to patch
  what the environment should be doing for wget, not vice versa.

 At this point, since it seems to have some demand, I'll probably put it
 in for 1.12.x; but I may very well move it to a module when we have
 support for that.

Thanks, yes that makes sense.

 Of course, Tony G indicated that he would prefer it to be
 conditionally-compiled, for concerns that the plugin architecture will
 add overhead to the wget binary. Wget is such a lightweight app, though,
 I'm not thinking that the plugin architecture is going to be very
 significant. It would be interesting to see if we can add support for
 some modules to be linked in directly, rather than dynamically; however,
 it'd still probably have to use the same mechanisms as the normal
 modules in order to work. Anyway, I'm sure we'll think about those
 things more when the time comes.

Makes sense.

 Or you could be proactive and start work on
 http://wget.addictivecode.org/FeatureSpecifications/Plugins
 (non-existent, but already linked to from FeatureSpecifications). :)

I'll look into that.

 On 10/14/07, Hrvoje Niksic [EMAIL PROTECTED] wrote:
  Tony Godshall [EMAIL PROTECTED] writes:
 
   OK, so let's go back to basics for a moment.
  
   wget's default behavior is to use all available bandwidth.
 
  And so is the default behavior of curl, Firefox, Opera, and so on.
  The expected behavior of a program that receives data over a TCP
  stream is to consume data as fast as it arrives.

What was your point exactly?  All the other kids do it?

Tony G

Re: wget default behavior [was Re: working on patch to limit to percent of bandwidth]

[Tony Godshall]

On 10/13/07, Josh Williams [EMAIL PROTECTED] wrote:
 On 10/13/07, Tony Godshall [EMAIL PROTECTED] wrote:
  Well, you may have such problems but you are very much reaching in
  thinking that my --linux-percent has anything to do with any failing
  in linux.
 
  It's about dealing with unfair upstream switches, which, I'm quite
  sure, were not running Linux.
 
  Let's not hijack this into a linux-bash.

 I really don't know what you were trying to say here...

You seemed to think --limit-percent was a solution for a misbehavior of linux.

My experience with linux networking is that it's very effective and
that upstream non-linux switches don't handle such an effective client
well.

When a linux box is my gateway/firewall I don't experience
single-client monopolization at all.

As to your linux issues, that's a topic that should probably discussed
in another forum, but I will say that I'm quite happy with the latest
Linux kernels- with the low-latency patch integrated and enabled my
desktop experience is quite snappy, even on this four-year-old 1.2GHz
laptop.  And stay away from the distro server kernels- they are
optimized for throughput at the cost of latency- they do their I/O in
bigger chunks.  And stay away from the RT kernels- they go too far in
giving I/O priority over everything else and end up churning on IRQs
unless they are very carefully tuned.

And no, I won't call the linux kernel GNU/Linux, if that was what you
were after.  The kernel is after all the one Linux thing in a
GNU/Linux system.

 .. I use GNU/Linux.

Anyone try Debian GNU/BSD yet?  Or Debian/Nexenta/GNU/Solaris?

-- 
Best Regards.
Please keep in touch.

wget default behavior [was Re: working on patch to limit to percent of bandwidth]

[Tony Godshall]

OK, so let's go back to basics for a moment.

wget's default behavior is to use all available bandwidth.

Is this the right thing to do?

Or is it better to back off a little after a bit?

Tony

Re: wget default behavior [was Re: working on patch to limit to percent of bandwidth]

[Josh Williams]

On 10/13/07, Tony Godshall [EMAIL PROTECTED] wrote:
 OK, so let's go back to basics for a moment.

 wget's default behavior is to use all available bandwidth.

 Is this the right thing to do?

 Or is it better to back off a little after a bit?

 Tony

IMO, this should be handled by the operating system, not the
individual applications. That's one of the reasons I believe this
should be a module instead, because it's more or less a hack to patch
what the environment should be doing for wget, not vice versa.

In my experience, GNU/Linux tends to consume all the resources
unbiasedly, seemingly on a first come first serve *until you're
done* basis. This should be brought to the attention of the LKML.

However, other operating systems do not seem to have this problem as
much. Even Windows networks seem to prioritise packets.

This is a problem I've been having major headaches with lately. It
would be nice if wget had a patch for this problem, but that would not
solve the problem of my web browser or sftp client consuming all the
network resources.

Re: wget default behavior [was Re: working on patch to limit to percent of bandwidth]

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256


 On 10/13/07, Tony Godshall [EMAIL PROTECTED] wrote:
 OK, so let's go back to basics for a moment.

 wget's default behavior is to use all available bandwidth.

 Is this the right thing to do?

 Or is it better to back off a little after a bit?

Heh. Well, some people are saying that Wget should support accelerated
downloads; several connections to download a single resource, which can
sometimes give a speed increase at the expense of nice-ness.

So you could say we're at a happy medium between those options! :)

Actually, Wget probably will get support for multiple simultaneous
connections; but number of connections to one host will be limited to a
max of two.

It's impossible for Wget to know how much is appropriate to back off,
and in most situations I can think of, backing off isn't appropriate.

In general, though, I agree that Wget's policy should be nice by default.

Josh Williams wrote:
 That's one of the reasons I believe this
 should be a module instead, because it's more or less a hack to patch
 what the environment should be doing for wget, not vice versa.

At this point, since it seems to have some demand, I'll probably put it
in for 1.12.x; but I may very well move it to a module when we have
support for that.

Of course, Tony G indicated that he would prefer it to be
conditionally-compiled, for concerns that the plugin architecture will
add overhead to the wget binary. Wget is such a lightweight app, though,
I'm not thinking that the plugin architecture is going to be very
significant. It would be interesting to see if we can add support for
some modules to be linked in directly, rather than dynamically; however,
it'd still probably have to use the same mechanisms as the normal
modules in order to work. Anyway, I'm sure we'll think about those
things more when the time comes.

Or you could be proactive and start work on
http://wget.addictivecode.org/FeatureSpecifications/Plugins
(non-existent, but already linked to from FeatureSpecifications). :)

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHESNi7M8hyUobTrERCChSAJ90KmWelT0bH9qQMlArapEdn1ocSACfRHcK
JJmV8QaqcnKTRYam/v0/lwg=
=TPsw
-END PGP SIGNATURE-

Re: working on patch to limit to percent of bandwidth

[Tony Godshall]

On 10/12/07, Hrvoje Niksic [EMAIL PROTECTED] wrote:
 Tony Godshall [EMAIL PROTECTED] writes:

  My point remains that the maximum initial rate (however you define
  initial in a protocol as unreliable as TCP/IP) can and will be
  wrong in a large number of cases, especially on shared connections.
 
  Again, would an algorithm where the rate is re-measured periodically
  and the initial-rate-error criticism were therefore addressed reduce
  your objection to the patch?

 Personally I don't see the value in attempting to find out the
 available bandwidth automatically.

You keep saying that.

 It seems too error prone, no
 matter how much heuristics you add into it.

But like someone said- the error is always in the nice direction.

  --limit-rate works
 because reading the data more slowly causes it to (eventually) also be
 sent more slowly.  --limit-percentage is impossible to define in
 precise terms, there's just too much guessing.

My patch --limit-percent does exactly the same thing except without
requiring foreknowledge.


-- 
Best Regards.
Please keep in touch.

Re: wget default behavior [was Re: working on patch to limit to percent of bandwidth]

[Tony Godshall]

On 10/13/07, Josh Williams [EMAIL PROTECTED] wrote:
 On 10/13/07, Tony Godshall [EMAIL PROTECTED] wrote:
  OK, so let's go back to basics for a moment.
 
  wget's default behavior is to use all available bandwidth.
 
  Is this the right thing to do?
 
  Or is it better to back off a little after a bit?
 
  Tony

 IMO, this should be handled by the operating system, not the
 individual applications. That's one of the reasons I believe this
 should be a module instead, because it's more or less a hack to patch
 what the environment should be doing for wget, not vice versa.

 In my experience, GNU/Linux tends to consume all the resources
 unbiasedly, seemingly on a first come first serve *until you're...

Well, you may have such problems but you are very much reaching in
thinking that my --linux-percent has anything to do with any failing
in linux.

It's about dealing with unfair upstream switches, which, I'm quite
sure, were not running Linux.

Let's not hijack this into a linux-bash.

-- 
Best Regards.
Tony

Re: wget default behavior [was Re: working on patch to limit to percent of bandwidth]

[Josh Williams]

On 10/13/07, Tony Godshall [EMAIL PROTECTED] wrote:
 Well, you may have such problems but you are very much reaching in
 thinking that my --linux-percent has anything to do with any failing
 in linux.

 It's about dealing with unfair upstream switches, which, I'm quite
 sure, were not running Linux.

 Let's not hijack this into a linux-bash.

I really don't know what you were trying to say here. I use GNU/Linux.

Re: wget default behavior [was Re: working on patch to limit to percent of bandwidth]

[Tony Godshall]

On 10/13/07, Micah Cowan [EMAIL PROTECTED] wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256


  On 10/13/07, Tony Godshall [EMAIL PROTECTED] wrote:
  OK, so let's go back to basics for a moment.
 
  wget's default behavior is to use all available bandwidth.
 
  Is this the right thing to do?
 
  Or is it better to back off a little after a bit?

 Heh. Well, some people are saying that Wget should support accelerated
 downloads; several connections to download a single resource, which can
 sometimes give a speed increase at the expense of nice-ness.

 So you could say we're at a happy medium between those options! :)

 Actually, Wget probably will get support for multiple simultaneous
 connections; but number of connections to one host will be limited to a
 max of two.

 It's impossible for Wget to know how much is appropriate to back off,
 and in most situations I can think of, backing off isn't appropriate.

 In general, though, I agree that Wget's policy should be nice by default.

Yeah, thanks, that's what I was trying to get at.

Wget should be aggressive iff you tell it to be, and otherwise should be nice.

In the presense of bad upstream switches I've found that even a
--limit-rate of 95% is way more tolerable to others than the default
100% utilization.

 Josh Williams wrote:
  That's one of the reasons I believe this
  should be a module instead, because it's more or less a hack to patch
  what the environment should be doing for wget, not vice versa.

 At this point, since it seems to have some demand, I'll probably put it
 in for 1.12.x; but I may very well move it to a module when we have
 support for that.

 Of course, Tony G indicated that he would prefer it to be
 conditionally-compiled, for concerns that the plugin architecture will
 add overhead to the wget binary. Wget is such a lightweight app, though,
 I'm not thinking that the plugin architecture is going to be very
 significant. It would be interesting to see if we can add support for
 some modules to be linked in directly, rather than dynamically; however,
 it'd still probably have to use the same mechanisms as the normal
 modules in order to work. Anyway, I'm sure we'll think about those
 things more when the time comes.

Good point.  I guess someone who wants an ultralightweight wget will
use the one in busybox instead of the normal one.

 Or you could be proactive and start work on
 http://wget.addictivecode.org/FeatureSpecifications/Plugins
 (non-existent, but already linked to from FeatureSpecifications). :)

Interesting.  I'll take a look.

 - --
 Micah J. Cowan
 Programmer, musician, typesetting enthusiast, gamer...
 http://micah.cowan.name/

 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.6 (GNU/Linux)
 Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

 iD8DBQFHESNi7M8hyUobTrERCChSAJ90KmWelT0bH9qQMlArapEdn1ocSACfRHcK
 JJmV8QaqcnKTRYam/v0/lwg=
 =TPsw
 -END PGP SIGNATURE-



-- 
Best Regards.
Please keep in touch.

Re: working on patch to limit to percent of bandwidth

[Hrvoje Niksic]

Tony Godshall [EMAIL PROTECTED] writes:

  available bandwidth and adjusts to that.  The usefullness is in
  trying to be unobtrusive to other users.

 The problem is that Wget simply doesn't have enough information to be
 unobtrusive.  Currently available bandwidth can and does change as new
 downloads are initiated and old ones are turned off.  Measuring
 initial bandwidth is simply insufficient to decide what bandwidth is
 really appropriate for Wget; only the user can know that, and that's
 what --limit-rate does.

 My patch (and the doc change in my patch) don't claim to be totally
 unobtrusive [...] Obviously people who the level of unobtrusiveness
 you define shouldn't be using it.

It was never my intention to define a particular level of
unobtrusiveness; the concept of being unobtrusive to other users was
brought up by Jim and I was responding to that.  My point remains that
the maximum initial rate (however you define initial in a protocol
as unreliable as TCP/IP) can and will be wrong in a large number of
cases, especially on shared connections.  Not only is it impossible to
be totally unobtrusive, but any *automated* attempts at being nice
to other users are doomed to failure, either by taking too much (if
the download starts when you're alone) or too little (if the download
starts with shared connection).

Re: working on patch to limit to percent of bandwidth

[Josh Williams]

On 10/12/07, Tony Godshall [EMAIL PROTECTED] wrote:
 Again, I do not claim to be unobtrusive.  Merely to reduce
 obtrusiveness.  I do not and cannot claim to be making wget *nice*,
 just nicER.

 You can't deny that dialing back is nicer than not.

Personally, I think this is a great idea. But I do agree that the
documentation is a bit messy right now (as well as the code). If this
doesn't make it into the current trunk, I think it'd make a great
module in version 2.

Re: working on patch to limit to percent of bandwidth

[Tony Godshall]

On 10/12/07, Hrvoje Niksic [EMAIL PROTECTED] wrote:
 Tony Godshall [EMAIL PROTECTED] writes:

   available bandwidth and adjusts to that.  The usefullness is in
   trying to be unobtrusive to other users.
 
  The problem is that Wget simply doesn't have enough information to be
  unobtrusive.  Currently available bandwidth can and does change as new
  downloads are initiated and old ones are turned off.  Measuring
  initial bandwidth is simply insufficient to decide what bandwidth is
  really appropriate for Wget; only the user can know that, and that's
  what --limit-rate does.
 
  My patch (and the doc change in my patch) don't claim to be totally
  unobtrusive [...] Obviously people who the level of unobtrusiveness
  you define shouldn't be using it.

 It was never my intention to define a particular level of
 unobtrusiveness; the concept of being unobtrusive to other users was
 brought up by Jim and I was responding to that.  My point remains that
 the maximum initial rate (however you define initial in a protocol
 as unreliable as TCP/IP) can and will be wrong in a large number of
 cases, especially on shared connections.

Again, would an algorithm where the rate is re-measured periodically
and the initial-rate-error criticism were therefore addressed reduce
your objection to the patch?  Perhaps you can answer each idea I gave
separately:

a) full speed downloads (which re-measure channel capacity) followed
by long sleeps

b) speed ramps up to peak and then back down

 Not only is it impossible to
 be totally unobtrusive, but any *automated* attempts at being nice
 to other users are doomed to failure, either by taking too much (if
 the download starts when you're alone) or too little (if the download
 starts with shared connection).

Again, I do not claim to be unobtrusive.  Merely to reduce
obtrusiveness.  I do not and cannot claim to be making wget *nice*,
just nicER.

You can't deny that dialing back is nicer than not.

-- 
Best Regards.
Please keep in touch.

Re: working on patch to limit to percent of bandwidth

[Tony Godshall]

On 10/12/07, Josh Williams [EMAIL PROTECTED] wrote:
 On 10/12/07, Tony Godshall [EMAIL PROTECTED] wrote:
  Again, I do not claim to be unobtrusive.  Merely to reduce
  obtrusiveness.  I do not and cannot claim to be making wget *nice*,
  just nicER.
 
  You can't deny that dialing back is nicer than not.

 Personally, I think this is a great idea. But I do agree that the
 documentation is a bit messy right now (as well as the code). If this
 doesn't make it into the current trunk, I think it'd make a great
 module in version 2.

Thanks for the support

Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

[Jim Wright]

I don't want this to spiral down to Micah bashing.  He has brought a lot
of good energy to the project, and gotten things moving forward nicely.
Thanks.

I know of instances where this option would be useful for me, and others
have chipped in.  I think we all agree it isn't perfect and there is
no perfect solution for the situation.  But it is better than what
exists now.

How about one last hypothetical situation, and then I'll bow out of this.
(Yes, I can live a happy life if this option isn't included!)

Joe Random User can type

wget --limit-percent 50% ftp://site/BigImage.iso

and then happily play his online game without giving up all his
bandwidth, and without having to have a clue about networking.  A simple
--limit-percent replaces trying to explain to someone how to determine
their bandwidth and then specifying some amount which is less than the
total but still leaves enough for other activity.

Jim

Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

[Tony Godshall]

...
  I guess I'd like to see compile-time options so people could make a
  tiny version for their embedded system, with most options and all
  documentation stripped out, and a huge kitchen-sink all-the-bells
  version and complete documentation for the power user version.  I
  don't think you have to go to a totally new (plug in) architecture or
  make the hard choices.

[Jim]
 Well, we need the plugin architecture anyway. There are some planned
 features (JavaScript and MetaLink support being the main ones) that have
 no business in Wget proper, as far as I'm concerned, but are inarguably
 useful.

  I know when I put an app into an embedded app, I'd rather not even
  have the overhead of the plug-in mechanism, I want it smaller than
  that.

 You have a good point regarding customized compilation, though I think
 that most of the current features in Wget belong as core features. There
 are some small exceptions (egd sockets).

Thanks.

Well, when I'm building an embedded device, I look at the invocations
wget that are actually being called in the scripts.  Since the end
product has no interactive shell, I don't need to have all those extra
options enabled!  In fact, in wget's case, one can often dispense with
the tool entirely- the busybox version suffices.

  ... And when I'm running the gnu version of something I expect it
  to have verbose man pages and lots of double-dash options, that's what
  tools like less and grep are for.

 Well... many GNU tools actually lack verbose man pages, particularly
 since info is the preferred documentation system for GNU software.

Well, I guess I'm spoiled by Debian.  If it ain't broke, don't fix it.
 Debian makes man pages because tools should have manpages.  IIRC,
that was one of the divorce issues.

 Despite the fact that many important GNU utilities are very
 feature-packed, they also tend not to have options that are only useful
 to a relatively small number of people--particularly when equivalent
 effects are possible with preexisting options.

 As to the overhead of the plugin mechanism, you're right, and I may well
 decide to make that optionally compiled.

Well, I'd rather have rate-limiting things be optionally compiled than
plugged-in, since they'd be useful for embedded devices.

[Micah]
  It's not really about this option, it's about a class of options. I'm in
  the unenviable position of having to determine whether small patches
  that add options are sufficiently useful to justify the addition of the
  option. Adding one new option/rc command is not a problem. But when,
  over time, fifty people suggest little patches that offer options with
  small benefits, we've suddenly got fifty new options cluttering up the
  documentation and --help output.

[Jim]
  I would posit that the vast majority of wget options are used in some
  extremely small percentage of wget invocations.  Should they be removed?

[Micah]
 Such as which ones?

 I don't think we're talking about the same extremely small percentages.

OK, so so far there are three of us, I think, that find it potentially
useful.  And you have not addressed the use cases I brought up.  So I
think your extremely small percentages assumption may be faulty.

 Looking through the options listed with --help, I can find very few
 options that I've never used or would not consider vital in some
 situations I (or someone else) might encounter.

 This doesn't look to me like a vital function, one that a large number
 of users will find mildly useful, or one that a mild number of users
 will find extremely useful. This looks like one that a mild number of
 users will find mildly useful. Only slightly more useful, in fact, than
 what is already done.

You keep saying that.  You seem to think unknown upstream bandwidth is
a rare thing.  Or that wanting to be nice to other bandwidth users in
such a circumstance is a rare thing.  I wish I lived in your universe.
 Mine's a lot more sloppy.

 It's also one of those fuzzy features that addresses a scenario that
 has no right solution (JavaScript support is in that domain). These
 sorts of features tend to invite a gang of friends to help get a little
 bit closer to the unreachable target. For instance, if we include this
 option, then the same users will find another option to control the
 period of time spent full-bore just as useful. A pulse feature might
 be useful, but then you'll probably want an option to control the
 spacing between those, too. And someone else may wish to introduce an
 option that saves bandwidth information persistently, and uses this to
 make a good estimate from the beginning.

Ah, finally, some meat.  You see this as opening a door.  Especially
as I inquire as too whether anyone has feedback on my implementation,
you see it mushrooming into a plethora of options.

 And all of this would amount to a very mild improvement over what
 already exists.

Your universe crisp, mine sloppy (above) ;-)

  In my view, wget is a useful and flexible 

Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Tony Godshall wrote:
 [Jim]
 Well, we need the plugin architecture anyway. There are some planned
 features (JavaScript and MetaLink support being the main ones) that have
 no business in Wget proper, as far as I'm concerned, but are inarguably
 useful.
 
 I know when I put an app into an embedded app, I'd rather not even
 have the overhead of the plug-in mechanism, I want it smaller than
 that.
 
 You have a good point regarding customized compilation, though I think
 that most of the current features in Wget belong as core features. There
 are some small exceptions (egd sockets).
 
 Thanks.

(You've misattributed this: it's me talking here, not Jim.)

 OK, so so far there are three of us, I think, that find it potentially
 useful.

One of whom, you'll note, was happy to see it as a module, which is what
I had also been suggesting.

 This doesn't look to me like a vital function, one that a large number
 of users will find mildly useful, or one that a mild number of users
 will find extremely useful. This looks like one that a mild number of
 users will find mildly useful. Only slightly more useful, in fact, than
 what is already done.
 
 You keep saying that.  You seem to think unknown upstream bandwidth is
 a rare thing.  Or that wanting to be nice to other bandwidth users in
 such a circumstance is a rare thing.

I do not think it's a particularly rare thing. I think it's a fairly
easily-dealt-with thing.

 Like I said when I submitted the patch, this
 essentially automates what I do manually:
   wget somesite
   ctrl-c
   wget -c --limit-rate nnK somesite

What I've been trying to establish, is whether automating such a thing
(directly within Wget), is a useful-enough thing to justify the patch.

 But they mainly fall into the category of features that a large number
 of users will use occasionally, and a small number of users will find
 indispensable much of the time. Will you find this feature
 indispensable, or can you pretty much use --limit-rate with a reasonable
 value to do the same thing?
 
 Horse dead.  Parts rolling in the freeway.

Is it? I was talking to Jim, not you. He actually hadn't said very much
until this point.

 If, on the other hand, it is really, just a pretty minor improvement
 that happens to be mildly useful to you, could we please drop using this
 as a platform to predict what my future reactions to new features in
 general are likely to be? :p
 
 Well, when a guy first joins the list and submits his first patch and gets...

Gets what? One should not expect that all patches are automatically
accepted. Jim knows this, and has also seen other people come with
patches I've accepted, which is why it's just silly to accuse me of
something there's already ample proof I don't.

And what is it you got? Did I ever say, no, it's not going in? Did I
ever say I'm against it? What I repeatedly said was, I need convincing.

 Anyhow, perhaps I did the wrong thing in bringing it here- perhaps I
 should have provided it as a wishlist bug in debian and seen how many
 ordinary people find it useful before taking it to the source...
 perhaps I should have vetted it or whatever.

Sure, vetting it is entirely helpful. Getting feedback from a larger
community of users is very helpful. And, lamentably, the current
activity level of this list is not sufficient that I can gauge how
useful a feature is to the community as a whole from the five-or-so
people that participate on this list.

I cannot gauge how useful a feature is from how loudly the contributor
proclaims it's useful. I already _know_ you find it useful, as you cared
enough to bother writing a patch. What I was hoping to hear, but hadn't
heard much of until just now, was more support from the rest of the
community. Jim had spoken up, but not particularly strongly. Rather than
waiting for people to have the chance to speak up, though, you just got
louder.

What is most interesting to me, is your reaction to my statements, which
were never I'm not putting it in, but I think it should wait and live
as an accessory. And to this you get upset, and both defensive and
offensive. This does not make it likelier for me to include your changes.

In this specific case, there's probably a good chance it'll go in (not
for 1.11 though), as I'm clearer now on exactly how useful Jim finds it,
and we've also had another speak up. In the future, though, if you've
got something you'd like me to consider including, you might consider
just a bit more patience than you've exhibited this time around.

Hopefully this thread can go away now, unless someone has something
truly new to contribute.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHD/mZ7M8hyUobTrERCLCrAJ9wApBrS12uqTJ/5pNDLxNI9zbJkACgkdBr
gCwWMhPZ/kzzY1ynR8aof+g=
=t9s1

Re: working on patch to limit to percent of bandwidth

[Hrvoje Niksic]

Tony Godshall [EMAIL PROTECTED] writes:

 My point remains that the maximum initial rate (however you define
 initial in a protocol as unreliable as TCP/IP) can and will be
 wrong in a large number of cases, especially on shared connections.

 Again, would an algorithm where the rate is re-measured periodically
 and the initial-rate-error criticism were therefore addressed reduce
 your objection to the patch?

Personally I don't see the value in attempting to find out the
available bandwidth automatically.  It seems too error prone, no
matter how much heuristics you add into it.  --limit-rate works
because reading the data more slowly causes it to (eventually) also be
sent more slowly.  --limit-percentage is impossible to define in
precise terms, there's just too much guessing.

Re: working on patch to limit to percent of bandwidth

[Josh Williams]

On 10/12/07, Hrvoje Niksic [EMAIL PROTECTED] wrote:
 Personally I don't see the value in attempting to find out the
 available bandwidth automatically.  It seems too error prone, no
 matter how much heuristics you add into it.  --limit-rate works
 because reading the data more slowly causes it to (eventually) also be
 sent more slowly.  --limit-percentage is impossible to define in
 precise terms, there's just too much guessing.

Yeah, that is a good point. Hence, I vote for it to become a module.

Re: working on patch to limit to percent of bandwidth

[Tony Godshall]

On 10/10/07, Micah Cowan [EMAIL PROTECTED] wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 Tony Godshall wrote:
  The scenario I was picturing was where you'd want to make sure some
  bandwidth was left available so that unfair routers wouldn't screw
  your net-neighbors.  I really don't see this as an attempt to be
  unobtrusive at all.  This is not an attempt to hide one's traffic,
  it's an attempt to not overwhelm in the presence of unfair switching.
  If I say --limit-pct 75% and the network is congested, yes, what I
  want is to use no more than 75% of the available bandwidth, not the
  total bandwidth.  So, yes, if the network is more congensted just now,
  then let this download get a lower bitrate, that's fine.

 I'm pretty sure that's what Jim meant by being unobtrusive; it surely
 had nothing to do with traffic-hiding.

 My current impression is that this is a useful addition for some limited
 scenarios, but not particularly more useful than --limit-rate already
 is. That's part of what makes it a good candidate as a plugin.

I guess I don't see how picking a reasonable rate automatically is
less useful then having to know what the maximum upstream bandwidth is
ahead of time.  (If the argument is about the rare case where maxing
out the download even briefly is unacceptable, then the whole
technique wget uses is really not appropriate- even limit-rate does
not back off till it has retrieved enough bytes to start measuring and
then they come in as a burst- that's the nature of starting a TCP
connection.)

Re: working on patch to limit to percent of bandwidth

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Tony Godshall wrote:
 On 10/10/07, Micah Cowan [EMAIL PROTECTED] wrote:
 My current impression is that this is a useful addition for some limited
 scenarios, but not particularly more useful than --limit-rate already
 is. That's part of what makes it a good candidate as a plugin.
 
 I guess I don't see how picking a reasonable rate automatically is
 less useful then having to know what the maximum upstream bandwidth is
 ahead of time.

I never claimed it was less useful. In fact, I said it was more useful.
My doubt is as to whether it is _significantly_ more useful.

I'm still open, just need more convincing.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDprc7M8hyUobTrERCHN6AJ4/+GhluN5a4ejkMZJN9dt+tHqM2ACfQ0I9
8LvCCfCaxdvH0PfHWHiN4MQ=
=m7+Y
-END PGP SIGNATURE-

anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

[Tony Godshall]

On 10/11/07, Micah Cowan [EMAIL PROTECTED] wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 Tony Godshall wrote:
  On 10/10/07, Micah Cowan [EMAIL PROTECTED] wrote:
  My current impression is that this is a useful addition for some limited
  scenarios, but not particularly more useful than --limit-rate already
  is. That's part of what makes it a good candidate as a plugin.
 
  I guess I don't see how picking a reasonable rate automatically is
  less useful then having to know what the maximum upstream bandwidth is
  ahead of time.

 I never claimed it was less useful. In fact, I said it was more useful.
 My doubt is as to whether it is _significantly_ more useful.

For me, yes.  For you, apparently not.  It's a small patch, really.
Did you even look at it?

 I'm still open, just need more convincing.

Well, I've said my piece.

Anyone want to comment on the actual code?

Anybody try it?

Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Tony Godshall wrote:
 On 10/11/07, Micah Cowan [EMAIL PROTECTED] wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 Tony Godshall wrote:
 On 10/10/07, Micah Cowan [EMAIL PROTECTED] wrote:
 My current impression is that this is a useful addition for some limited
 scenarios, but not particularly more useful than --limit-rate already
 is. That's part of what makes it a good candidate as a plugin.
 I guess I don't see how picking a reasonable rate automatically is
 less useful then having to know what the maximum upstream bandwidth is
 ahead of time.
 I never claimed it was less useful. In fact, I said it was more useful.
 My doubt is as to whether it is _significantly_ more useful.
 
 For me, yes.  For you, apparently not.  It's a small patch, really.
 Did you even look at it?

I have, yes. And yes, it's a very small patch. The issue isn't so much
about the extra code or code maintenance; it's more about extra
documentation, and avoiding too much clutter of documentation and lists
of options/rc-commands. I'm not very picky about adding little
improvements to Wget; I'm a little pickier about adding new options.

It's not really about this option, it's about a class of options. I'm in
the unenviable position of having to determine whether small patches
that add options are sufficiently useful to justify the addition of the
option. Adding one new option/rc command is not a problem. But when,
over time, fifty people suggest little patches that offer options with
small benefits, we've suddenly got fifty new options cluttering up the
documentation and --help output. If the benefits are such that only a
handful of people will ever use any of them, then they may not have been
worth the addition, and I'm probably not doing my job properly.

Particularly since a plugin architecture is planned, it seems ideal to
me to recommend that such things be implemented as plugins at that
point. In the meantime, people who find the feature sufficiently useful
can easily apply the patch to Wget themselves (that's part of what makes
Free Software great!), and even offer patched binaries up if there's
call for it.

If a number of people bother to download and install the patch, or fetch
 patched binaries in preference to the official binaries, that'd be a
good indicator that it's worth pulling in.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDsBj7M8hyUobTrERCIMPAJ9z936EGkfx7b/1sKAt3zw6OcPMIgCaAi2Y
qtNxSlmy09JSvtaWgZ42M7o=
=iRGw
-END PGP SIGNATURE-

Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

[Tony Godshall]

...
 I have, yes. And yes, it's a very small patch. The issue isn't so much
 about the extra code or code maintenance; it's more about extra
 documentation, and avoiding too much clutter of documentation and lists
 of options/rc-commands. I'm not very picky about adding little
 improvements to Wget; I'm a little pickier about adding new options.

 It's not really about this option, it's about a class of options. I'm in
 the unenviable position of having to determine whether small patches
 that add options are sufficiently useful to justify the addition of the
 option. Adding one new option/rc command is not a problem. But when,
 over time, fifty people suggest little patches that offer options with
 small benefits, we've suddenly got fifty new options cluttering up the
 documentation and --help output.

Would it be better, then, if I made it --limit-rate nn% instead of
limit-percent nn?
And made the descrip briefer?

  If the benefits are such that only a
 handful of people will ever use any of them, then they may not have been
 worth the addition, and I'm probably not doing my job properly. ...

I guess I'd like to see compile-time options so people could make a
tiny version for their embedded system, with most options and all
documentation stripped out, and a huge kitchen-sink all-the-bells
version and complete documentation for the power user version.  I
don't think you have to go to a totally new (plug in) architecture or
make the hard choices.

I know when I put an app into an embedded app, I'd rather not even
have the overhead of the plug-in mechanism, I want it smaller than
that.  And when I'm running the gnu version of something I expect it
to have verbose man pages and lots of double-dash options, that's what
tools like less and grep are for.

Tony

Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

[Tony Godshall]

On 10/11/07, Tony Godshall [EMAIL PROTECTED] wrote:
 ...
  I have, yes. And yes, it's a very small patch. The issue isn't so much
  about the extra code or code maintenance; it's more about extra
  documentation, and avoiding too much clutter of documentation and lists
  of options/rc-commands. I'm not very picky about adding little
  improvements to Wget; I'm a little pickier about adding new options.
 
  It's not really about this option, it's about a class of options. I'm in
  the unenviable position of having to determine whether small patches
  that add options are sufficiently useful to justify the addition of the
  option. Adding one new option/rc command is not a problem. But when,
  over time, fifty people suggest little patches that offer options with
  small benefits, we've suddenly got fifty new options cluttering up the
  documentation and --help output.

 Would it be better, then, if I made it --limit-rate nn% instead of
 limit-percent nn?
 And made the descrip briefer?

Also would it help if the behavior was changed so it pulsed
occasionally and therefore wouldn't suffer from the
initial-measurement-error case.

I'm trying to judge whether I should spend more time touching it up
into something acceptable or just let it remain a personal hack.

   If the benefits are such that only a
  handful of people will ever use any of them, then they may not have been
  worth the addition, and I'm probably not doing my job properly. ...

 I guess I'd like to see compile-time options so people could make a
 tiny version for their embedded system, with most options and all
 documentation stripped out, and a huge kitchen-sink all-the-bells
 version and complete documentation for the power user version.  I
 don't think you have to go to a totally new (plug in) architecture or
 make the hard choices.

 I know when I put an app into an embedded app, I'd rather not even
 have the overhead of the plug-in mechanism, I want it smaller than
 that.  And when I'm running the gnu version of something I expect it
 to have verbose man pages and lots of double-dash options, that's what
 tools like less and grep are for.

 Tony



-- 
Best Regards.
Please keep in touch.

Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Tony Godshall wrote:
 ...
 I have, yes. And yes, it's a very small patch. The issue isn't so much
 about the extra code or code maintenance; it's more about extra
 documentation, and avoiding too much clutter of documentation and lists
 of options/rc-commands. I'm not very picky about adding little
 improvements to Wget; I'm a little pickier about adding new options.

 It's not really about this option, it's about a class of options. I'm in
 the unenviable position of having to determine whether small patches
 that add options are sufficiently useful to justify the addition of the
 option. Adding one new option/rc command is not a problem. But when,
 over time, fifty people suggest little patches that offer options with
 small benefits, we've suddenly got fifty new options cluttering up the
 documentation and --help output.
 
 Would it be better, then, if I made it --limit-rate nn% instead of
 limit-percent nn?

The thought occurred to me; but then it's no longer such a small patch,
and we've just introduced a new argument parser function, so we still
get to is it justified?.

 And made the descrip briefer?

It's not really the length of the description that I'm concerned about,
it's just the number of little options.

  If the benefits are such that only a
 handful of people will ever use any of them, then they may not have been
 worth the addition, and I'm probably not doing my job properly. ...
 
 I guess I'd like to see compile-time options so people could make a
 tiny version for their embedded system, with most options and all
 documentation stripped out, and a huge kitchen-sink all-the-bells
 version and complete documentation for the power user version.  I
 don't think you have to go to a totally new (plug in) architecture or
 make the hard choices.

Well, we need the plugin architecture anyway. There are some planned
features (JavaScript and MetaLink support being the main ones) that have
no business in Wget proper, as far as I'm concerned, but are inarguably
useful.

You have a good point regarding customized compilation, though I think
that most of the current features in Wget belong as core features. There
are some small exceptions (egd sockets).

 I know when I put an app into an embedded app, I'd rather not even
 have the overhead of the plug-in mechanism, I want it smaller than
 that.  And when I'm running the gnu version of something I expect it
 to have verbose man pages and lots of double-dash options, that's what
 tools like less and grep are for.

Well... many GNU tools actually lack verbose man pages, particularly
since info is the preferred documentation system for GNU software.

Despite the fact that many important GNU utilities are very
feature-packed, they also tend not to have options that are only useful
to a relatively small number of people--particularly when equivalent
effects are possible with preexisting options.

As to the overhead of the plugin mechanism, you're right, and I may well
decide to make that optionally compiled.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDs2U7M8hyUobTrERCOYoAJ9bIfGbztes0MEfKxAPwpQY/bjJAQCeOAXn
8M6Kj1vLploBN+qENpF2gu8=
=K9Sb
-END PGP SIGNATURE-

Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

[Jim Wright]

On Thu, 11 Oct 2007, Micah Cowan wrote:

 It's not really about this option, it's about a class of options. I'm in
 the unenviable position of having to determine whether small patches
 that add options are sufficiently useful to justify the addition of the
 option. Adding one new option/rc command is not a problem. But when,
 over time, fifty people suggest little patches that offer options with
 small benefits, we've suddenly got fifty new options cluttering up the
 documentation and --help output.

I would posit that the vast majority of wget options are used in some
extremely small percentage of wget invocations.  Should they be removed?
In my view, wget is a useful and flexible tool largely because there
are a lot of options.  The internet is a messy place, and wget can cope.

I have a handful of options I've added to wget which are mandatory for
my use.  Mostly dealing with timeouts and retries.  Useful features which
would not commonly be used.  Am I correct in reading in to this discussion
that submitting new features is not encouraged?

Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jim Wright wrote:
 On Thu, 11 Oct 2007, Micah Cowan wrote:
 
 It's not really about this option, it's about a class of options. I'm in
 the unenviable position of having to determine whether small patches
 that add options are sufficiently useful to justify the addition of the
 option. Adding one new option/rc command is not a problem. But when,
 over time, fifty people suggest little patches that offer options with
 small benefits, we've suddenly got fifty new options cluttering up the
 documentation and --help output.
 
 I would posit that the vast majority of wget options are used in some
 extremely small percentage of wget invocations.  Should they be removed?

Such as which ones?

I don't think we're talking about the same extremely small percentages.

Looking through the options listed with --help, I can find very few
options that I've never used or would not consider vital in some
situations I (or someone else) might encounter.

This doesn't look to me like a vital function, one that a large number
of users will find mildly useful, or one that a mild number of users
will find extremely useful. This looks like one that a mild number of
users will find mildly useful. Only slightly more useful, in fact, than
what is already done.

It's also one of those fuzzy features that addresses a scenario that
has no right solution (JavaScript support is in that domain). These
sorts of features tend to invite a gang of friends to help get a little
bit closer to the unreachable target. For instance, if we include this
option, then the same users will find another option to control the
period of time spent full-bore just as useful. A pulse feature might
be useful, but then you'll probably want an option to control the
spacing between those, too. And someone else may wish to introduce an
option that saves bandwidth information persistently, and uses this to
make a good estimate from the beginning.

And all of this would amount to a very mild improvement over what
already exists.

 In my view, wget is a useful and flexible tool largely because there
 are a lot of options.  The internet is a messy place, and wget can cope.

Sure. But what does --limit-percent allow wget to cope with that it
cannot currently?

 I have a handful of options I've added to wget which are mandatory for
 my use.  Mostly dealing with timeouts and retries.  Useful features which
 would not commonly be used.

But they mainly fall into the category of features that a large number
of users will use occasionally, and a small number of users will find
indispensable much of the time. Will you find this feature
indispensable, or can you pretty much use --limit-rate with a reasonable
value to do the same thing?

 Am I correct in reading in to this discussion
 that submitting new features is not encouraged?

To be honest, I'm shocked to get this sort of reaction over what is, as
far as I can tell, an extremely small improvement. If you really care
that much about it, I really don't mind putting it in. But if it's
really that useful to you, then I don't think your previous comments
really conveyed the degree to which that was the case. I repeatedly
asked for people to sell it for me, and got very little actual
case-making other than impressions that it was a very minor convenience
improvement. If it's more than a very minor improvement to you, then I
wish you'd have made that clearer from the start.

If, on the other hand, it is really, just a pretty minor improvement
that happens to be mildly useful to you, could we please drop using this
as a platform to predict what my future reactions to new features in
general are likely to be? :p

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDwIc7M8hyUobTrERCDJRAJ91SkMNlTc0ssUpejnyEuGp7MqvIwCgir9U
9t7oOJ8y40VerzlnhysFSXw=
=u5oK
-END PGP SIGNATURE-

Re: working on patch to limit to percent of bandwidth

[Hrvoje Niksic]

Jim Wright [EMAIL PROTECTED] writes:

 - --limit-rate will find your version handy, but I want to hear from
 them. :)

 I would appreciate and have use for such an option.  We often access
 instruments in remote locations (think a tiny island in the Aleutians)
 where we share bandwidth with other organizations.

A limitation in percentage doesn't make sense if you don't know
exactly how much bandwidth is available.  Trying to determine full
bandwidth and backing off from there is IMHO doomed to failure because
the initial speed Wget gets can be quite different from the actual
link bandiwdth, at least in a shared link scenario.  A --limit-percent
implemented as proposed here would only limit the retrieval speed to
the specified fraction of the speed Wget happened to get at the
beginning of the download.  That is not only incorrect, but also quite
non-deterministic.

If there were way to query the network for the connection speed, I
would support the limit-percent idea.  But since that's not
possible, I think it's better to stick with the current --limit-rate,
where we give the user an option to simply tell Wget how much
bandwidth to consume.

Re: working on patch to limit to percent of bandwidth

[Jim Wright]

I think there is still a case for attempting percent limiting.  I agree
with your point that we can not discover the full bandwidth of the
link and adjust to that.  The approach discovers the current available
bandwidth and adjusts to that.  The usefullness is in trying to be
unobtrusive to other users.




On Wed, 10 Oct 2007, Hrvoje Niksic wrote:

 Jim Wright [EMAIL PROTECTED] writes:
 
  - --limit-rate will find your version handy, but I want to hear from
  them. :)
 
  I would appreciate and have use for such an option.  We often access
  instruments in remote locations (think a tiny island in the Aleutians)
  where we share bandwidth with other organizations.
 
 A limitation in percentage doesn't make sense if you don't know
 exactly how much bandwidth is available.  Trying to determine full
 bandwidth and backing off from there is IMHO doomed to failure because
 the initial speed Wget gets can be quite different from the actual
 link bandiwdth, at least in a shared link scenario.  A --limit-percent
 implemented as proposed here would only limit the retrieval speed to
 the specified fraction of the speed Wget happened to get at the
 beginning of the download.  That is not only incorrect, but also quite
 non-deterministic.
 
 If there were way to query the network for the connection speed, I
 would support the limit-percent idea.  But since that's not
 possible, I think it's better to stick with the current --limit-rate,
 where we give the user an option to simply tell Wget how much
 bandwidth to consume.
 

Re: working on patch to limit to percent of bandwidth

[Tony Godshall]

   - --limit-rate will find your version handy, but I want to hear from
   them. :)

   I would appreciate and have use for such an option.  We often access
   instruments in remote locations (think a tiny island in the Aleutians)
   where we share bandwidth with other organizations.

  A limitation in percentage doesn't make sense if you don't know
  exactly how much bandwidth is available.  Trying to determine full
  bandwidth and backing off from there is IMHO doomed to failure because
  the initial speed Wget gets can be quite different from the actual
  link bandiwdth, at least in a shared link scenario.  A --limit-percent
  implemented as proposed here would only limit the retrieval speed to
  the specified fraction of the speed Wget happened to get at the
  beginning of the download.  That is not only incorrect, but also quite
  non-deterministic.

  If there were way to query the network for the connection speed, I
  would support the limit-percent idea.  But since that's not
  possible, I think it's better to stick with the current --limit-rate,
  where we give the user an option to simply tell Wget how much
  bandwidth to consume.

 I think there is still a case for attempting percent limiting.  I agree
 with your point that we can not discover the full bandwidth of the
 link and adjust to that.  The approach discovers the current available
 bandwidth and adjusts to that.  The usefullness is in trying to be
 unobtrusive to other users.

Network conditions do change and my initial all-out estimate is
certainly not ideal but it works in many situations.

Another way would be to transfer in a more bursty mode rather than
meter at small granularity- that way one could measure the rate of
each burst an one could take the channel capacity.  I worry that that
might be more harmful to those sharing channel in cases like Hvroje's
than the initial-burst measurement, and in fact have am thinking that
one could cache the initial-measure value and use it for future
connections.

An alternative to a bursty mode would be to start at full speed and
then ramp down till we hit the desired rate and then ramp back up till
we bump the limit and down again.  That way we could update the max
rate estimate periodically and recover from any error that might have
occurred in the initial estimate.  Any thoughts on this behavior?
Less harmful?  More?

Tony

Re: working on patch to limit to percent of bandwidth

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jim Wright wrote:
 I think there is still a case for attempting percent limiting.  I agree
 with your point that we can not discover the full bandwidth of the
 link and adjust to that.  The approach discovers the current available
 bandwidth and adjusts to that.  The usefullness is in trying to be
 unobtrusive to other users.

Does it really fit that description, though? Given that it runs
full-bore for 15 seconds (not that that's very long)...

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDTRD7M8hyUobTrERCM44AJ9rwWoavX1aqHIw8i3MR4nNabbNQgCeLdKW
gA2VuaUGbBRhlaexlhOn+TE=
=t4lv
-END PGP SIGNATURE-

Re: working on patch to limit to percent of bandwidth

[Tony Godshall]

 ... I worry that that might be more harmful to those sharing channel in cases 
 like Hvroje's ...

Sorry, Hvroje, Jim, I meant Jim's case.

Tony

Re: working on patch to limit to percent of bandwidth

[Tony Godshall]

 Jim Wright wrote:
  I think there is still a case for attempting percent limiting.  I agree
  with your point that we can not discover the full bandwidth of the
  link and adjust to that.  The approach discovers the current available
  bandwidth and adjusts to that.  The usefullness is in trying to be
  unobtrusive to other users.

 Does it really fit that description, though? Given that it runs
 full-bore for 15 seconds (not that that's very long)...

I guess it depends on the type of users you are sharing with and the
upstream switches and routers.

My experience is that with some routers and switches a single user
wget'ing an iso can cause web-browsing people to experience slow
response.  That kind of application is not one where 15sec will make
much difference, and in fact there's a big backoff after that first
15sec.

OTOH if you are sharing with latency-sensitive apps (VOIP, realtime
control, etc.) and a wget bogs your app down, you better fix your
switches and routers- you will be affected by anybody in an
interactive web browser streaming youtube or whatever too.  This patch
is not a solution for that use case, and I agree that there really
isn't one that an app like wget can reasonably implement (without
delving into nonportable OS stuff).

Tony

Re: working on patch to limit to percent of bandwidth

[Hrvoje Niksic]

Jim Wright [EMAIL PROTECTED] writes:

 I think there is still a case for attempting percent limiting.  I
 agree with your point that we can not discover the full bandwidth of
 the link and adjust to that.  The approach discovers the current
 available bandwidth and adjusts to that.  The usefullness is in
 trying to be unobtrusive to other users.

The problem is that Wget simply doesn't have enough information to be
unobtrusive.  Currently available bandwidth can and does change as new
downloads are initiated and old ones are turned off.  Measuring
initial bandwidth is simply insufficient to decide what bandwidth is
really appropriate for Wget; only the user can know that, and that's
what --limit-rate does.

RE: working on patch to limit to percent of bandwidth

[Tony Lewis]

Hrvoje Niksic wrote:

 Measuring initial bandwidth is simply insufficient to decide what
 bandwidth is really appropriate for Wget; only the user can know
 that, and that's what --limit-rate does.

The user might be able to make a reasonable guess as to the download rate if
wget reported its average rate at the end of a session. That way the user
can collect rates over time and try to give --limit-rate a reasonable value.

Tony

Re: working on patch to limit to percent of bandwidth

[Tony Godshall]

On 10/10/07, Tony Lewis [EMAIL PROTECTED] wrote:
 Hrvoje Niksic wrote:

  Measuring initial bandwidth is simply insufficient to decide what
  bandwidth is really appropriate for Wget; only the user can know
  that, and that's what --limit-rate does.

 The user might be able to make a reasonable guess as to the download rate if
 wget reported its average rate at the end of a session. That way the user
 can collect rates over time and try to give --limit-rate a reasonable value.

 Tony [L]

It reports rate during, and on my connections it's constant enough to
make a good estimate for --limit-rate.  I just automated that, really.

Tony G

Re: working on patch to limit to percent of bandwidth

[Tony Godshall]

Indeed.
On 10/10/07, Hrvoje Niksic [EMAIL PROTECTED] wrote:
 Jim Wright [EMAIL PROTECTED] writes:

  I think there is still a case for attempting percent limiting.  I
  agree with your point that we can not discover the full bandwidth of
  the link and adjust to that.  The approach discovers the current
  available bandwidth and adjusts to that.  The usefullness is in
  trying to be unobtrusive to other users.

 The problem is that Wget simply doesn't have enough information to be
 unobtrusive.  Currently available bandwidth can and does change as new
 downloads are initiated and old ones are turned off.  Measuring
 initial bandwidth is simply insufficient to decide what bandwidth is
 really appropriate for Wget; only the user can know that, and that's
 what --limit-rate does.

My patch (and the doc change in my patch) don't claim to be totally
unobtrusive - it has a particular behavior that's documented, which is
to try to be less obtrusive than your typical
get-it-for-me-right-now-as-fast-as-you-can download.

Obviously people who the level of unobtrusiveness you define shouldn't
be using it.  Then again, people who require that level probably need
to get routers implement a little more fairness or QOS- that don't
let one TCP connection lock out other users.

My patch just does automatically what I used to do manually- start
obtrusive and then scale back to less obtrusive for the rest of the
download.

Even competent non-sys-admin people often are not appraised of the
technical details of the networks they use, but they may still want to
be reasonably nice, for example, to the other people using the wifi at
the cybercafe.

It's certainly a step above the naive behavior- the naive user doesn't
even know then (their typical tools, like MSIE don't even tell them!)

Tony G

Re: working on patch to limit to percent of bandwidth

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hrvoje Niksic wrote:
 Jim Wright [EMAIL PROTECTED] writes:
 
 I think there is still a case for attempting percent limiting.  I
 agree with your point that we can not discover the full bandwidth of
 the link and adjust to that.  The approach discovers the current
 available bandwidth and adjusts to that.  The usefullness is in
 trying to be unobtrusive to other users.
 
 The problem is that Wget simply doesn't have enough information to be
 unobtrusive.  Currently available bandwidth can and does change as new
 downloads are initiated and old ones are turned off.  Measuring
 initial bandwidth is simply insufficient to decide what bandwidth is
 really appropriate for Wget; only the user can know that, and that's
 what --limit-rate does.

So far, I'm inclined to agree.

For instance, if one just sticks limit_percent = 25 in their wgetrc,
then on some occasions, Wget will limit to far too _low_ a rate, when
most of the available bandwidth is already being consumed by other things.

Regardless of what we decide on this, though, I like Tony L's suggestion
of some summary data at completion. He had already suggested something
similar to this for a proposed interactive prompt at interrupt.

I'm thinking that there are a lot of other little nice-to-haves
related to such a feature, too: someone might want Wget to save previous
download rates to a file, and average them up across invocations, and
base a percentage on this when called upon to do so. Basically, it
smells like one of many possible hacks to perform on --limit-rate, which
makes it a good candidate for a plugin (once we have that infrastructure
in place), rather than Wget proper.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDUlH7M8hyUobTrERCLbrAJ0eKAHly5C46dGpJVuGsJLseYSU9gCfW9H7
C2e8R7BF9rcQ7UXHKHLV6bk=
=I/eA
-END PGP SIGNATURE-

Re: working on patch to limit to percent of bandwidth

[Tony Godshall]

  I think there is still a case for attempting percent limiting.  I
  agree with your point that we can not discover the full bandwidth of
  the link and adjust to that.  The approach discovers the current
  available bandwidth and adjusts to that.  The usefullness is in
  trying to be unobtrusive to other users.

  The problem is that Wget simply doesn't have enough information to be
  unobtrusive.  Currently available bandwidth can and does change as new
  downloads are initiated and old ones are turned off.  Measuring
  initial bandwidth is simply insufficient to decide what bandwidth is
  really appropriate for Wget; only the user can know that, and that's
  what --limit-rate does.

 So far, I'm inclined to agree.

 For instance, if one just sticks limit_percent = 25 in their wgetrc,
 then on some occasions, Wget will limit to far too _low_ a rate, when
 most of the available bandwidth is already being consumed by other things.
...

Well, if you are using 25% then you are trying to be *really* nice and
there's no such thing as far too low, is there.

The scenario I was picturing was where you'd want to make sure some
bandwidth was left available so that unfair routers wouldn't screw
your net-neighbors.  I really don't see this as an attempt to be
unobtrusive at all.  This is not an attempt to hide one's traffic,
it's an attempt to not overwhelm in the presence of unfair switching.
If I say --limit-pct 75% and the network is congested, yes, what I
want is to use no more than 75% of the available bandwidth, not the
total bandwidth.  So, yes, if the network is more congensted just now,
then let this download get a lower bitrate, that's fine.

Tony

Re: working on patch to limit to percent of bandwidth

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Tony Godshall wrote:
 The scenario I was picturing was where you'd want to make sure some
 bandwidth was left available so that unfair routers wouldn't screw
 your net-neighbors.  I really don't see this as an attempt to be
 unobtrusive at all.  This is not an attempt to hide one's traffic,
 it's an attempt to not overwhelm in the presence of unfair switching.
 If I say --limit-pct 75% and the network is congested, yes, what I
 want is to use no more than 75% of the available bandwidth, not the
 total bandwidth.  So, yes, if the network is more congensted just now,
 then let this download get a lower bitrate, that's fine.

I'm pretty sure that's what Jim meant by being unobtrusive; it surely
had nothing to do with traffic-hiding.

My current impression is that this is a useful addition for some limited
scenarios, but not particularly more useful than --limit-rate already
is. That's part of what makes it a good candidate as a plugin.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDWs07M8hyUobTrERCEkbAJ9lbnva+Xtk8rv9S1AYOZ7yjZ2VuQCcDgOQ
hZKEjD4qZy/BwgDmchCDT1k=
=jN50
-END PGP SIGNATURE-

Re: working on patch to limit to percent of bandwidth

[Jim Wright]

On Mon, 8 Oct 2007, Micah Cowan wrote:

 As to whether or not it will be included in mainline Wget, that depends
 on the answer to your question, does this seem like something others of
 you could use? I, personally, wouldn't find it very useful (I rarely
 use even --limit-rate), so I'd be interested in knowing who would. I
 suspect that it may well be the case that most folks who have need of
 - --limit-rate will find your version handy, but I want to hear from
 them. :)

I would appreciate and have use for such an option.  We often access
instruments in remote locations (think a tiny island in the Aleutians)
where we share bandwidth with other organizations.  As the other
organizations are often involved in health and safety (Coast Guard,
Alaska Tsunami Warning Center, etc.) we can not dominate the bandwidth.
While --limit-rate might well be preferred in specific cases, having a
--limit-percent option available as well would be nice.

Jim

Re: working on patch to limit to percent of bandwidth

[A. P. Godshall]

On 10/9/07, Jim Wright [EMAIL PROTECTED] wrote:
 On Mon, 8 Oct 2007, Micah Cowan wrote:

  As to whether or not it will be included in mainline Wget, that depends
  on the answer to your question, does this seem like something others of
  you could use? I, personally, wouldn't find it very useful (I rarely
  use even --limit-rate), so I'd be interested in knowing who would. I
  suspect that it may well be the case that most folks who have need of
  - --limit-rate will find your version handy, but I want to hear from
  them. :)

 I would appreciate and have use for such an option.  We often access
 instruments in remote locations (think a tiny island in the Aleutians)
 where we share bandwidth with other organizations.  As the other
 organizations are often involved in health and safety (Coast Guard,
 Alaska Tsunami Warning Center, etc.) we can not dominate the bandwidth.
 While --limit-rate might well be preferred in specific cases, having a
 --limit-percent option available as well would be nice.

[private response to limit list clutter]

Cool.  Good to hear of another user.  See the Works For Me version
(patch) in the other message.

Note though that my patch *does* dominate the bandwidth for about 15 seconds
to measure the available bandwidth before it falls back.  On my
network, it seemed
to take a few seconds before enough bytes were transferred to get a reasonable
measure.

-- 
Best Regards.
Tony

Re: working on patch to limit to percent of bandwidth

[A. P. Godshall]

 Go ahead and send it on here so we can comment on the code :-)
...

I sent it to wget-patches; if you are not subscribed to that, you can
find it at...

  http://permalink.gmane.org/gmane.comp.web.wget.patches/2190

Discussion here or there, I don't care

-- 
Best Regards.
Please keep in touch.

Re: working on patch to limit to percent of bandwidth

[Josh Williams]

On 10/8/07, A. P. Godshall [EMAIL PROTECTED] wrote:
 Anyhow, does this seem like something others of you could use?  Should
 I submit the patch to the submit list or should I post it here for
 people to hash out any parameterization niceties etc first?

Go ahead and send it on here so we can comment on the code :-)

Re: working on patch to limit to percent of bandwidth

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

A. P. Godshall wrote:
 Hi
 
 New to the list.

Welcome!

 Wrote a patch that Works For Me to limit to a percent of measured
 bandwidth.  This is useful, like --limit-rate, in cases where an
 upstream switch is poorly made and interactive users get locked out
 when a single box does a wget, but limit-pct is more automatic in
 the sense that you don't have to know ahead of time how big your
 downstream pipe is.
 
 I.e., what I used to do is (1) wget, look at the bandwidth I was
 getting, and then (2) Ctrl-C,  (3) Ctrl-P and edit the line to add -c
 --limit-rate nnK (where nn is a bit less than I was getting).
 
 Now I can wget --limit-pct 50 and it will go full-speed for a bit and
 then back off till till the average speed is 50% of what the we saw
 during that time.
 
 The heuristic I'm using is to download full-speed for 15 seconds and
 then back off- that seems to work on my connection (too much less and
 measured rate is erratic, too much more and the defective upstream
 switch locks interactive folks out long enough that they notice and
 complain).  Does that seem reasonable to folks or should it be
 parameterized.  I'm not sure I can spend much time on complex
 parameter handling etc. right now.
 
 Anyhow, does this seem like something others of you could use?  Should
 I submit the patch to the submit list or should I post it here for
 people to hash out any parameterization niceties etc first?

The best place to submit patches is to [EMAIL PROTECTED]
Discussions can also happen there, too. Please submit patches against
relatively recent development code, as opposed to the latest release
(which happened ~two years ago), as otherwise the work involved in
bringing it up-to-date may be a disincentive to include it. :)
Information on obtaining the latest development version of Wget is at
http://wget.addictivecode.org/RepositoryAccess.

As to whether or not it will be included in mainline Wget, that depends
on the answer to your question, does this seem like something others of
you could use? I, personally, wouldn't find it very useful (I rarely
use even --limit-rate), so I'd be interested in knowing who would. I
suspect that it may well be the case that most folks who have need of
- --limit-rate will find your version handy, but I want to hear from
them. :)

Also, I'd like a little more understanding about what the use case is:
is it just to use N% less bandwidth than you seem to have available, so
that other connections you may open won't be taxed as much?

A couple notes off the bat: I'd prefer --limit-percent to --limit-pct,
as apparently more recent naming conventions seem to prefer
unabbreviated terms to abbreviated ones. And, I would desire a wgetrc
command to complement the long option version. Don't break your back
working these things into your patch, though: let's first see what
you've got and whether folks want it (lest you waste effort for nothing).

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHCqBJ7M8hyUobTrERCFXFAJ9/BQrl9B+SajnuXgxCtZWRyPITUQCeKSl/
oQzSzJuGNDZhSP6GNKq9wkI=
=AgXg
-END PGP SIGNATURE-

Re: working on patch to limit to percent of bandwidth

[Micah Cowan]

And here's another post that apparently got sent with an erroneous
signature. I think I may have figured out what was wrong; I specifically
remember that I was still holding shift down when I typed one of the
spaces in my passphrase... maybe that results in some screwiness...

-- 
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/