Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

2007-10-12 Thread Jim Wright
I don't want this to spiral down to Micah bashing.  He has brought a lot
of good energy to the project, and gotten things moving forward nicely.
Thanks.

I know of instances where this option would be useful for me, and others
have chipped in.  I think we all agree it isn't perfect and there is
no perfect solution for the situation.  But it is better than what
exists now.

How about one last hypothetical situation, and then I'll bow out of this.
(Yes, I can live a happy life if this option isn't included!)

Joe Random User can type

wget --limit-percent 50% ftp://site/BigImage.iso

and then happily play his online game without giving up all his
bandwidth, and without having to have a clue about networking.  A simple
--limit-percent replaces trying to explain to someone how to determine
their bandwidth and then specifying some amount which is less than the
total but still leaves enough for other activity.

Jim


Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

2007-10-12 Thread Tony Godshall
...
  I guess I'd like to see compile-time options so people could make a
  tiny version for their embedded system, with most options and all
  documentation stripped out, and a huge kitchen-sink all-the-bells
  version and complete documentation for the power user version.  I
  don't think you have to go to a totally new (plug in) architecture or
  make the hard choices.

[Jim]
 Well, we need the plugin architecture anyway. There are some planned
 features (JavaScript and MetaLink support being the main ones) that have
 no business in Wget proper, as far as I'm concerned, but are inarguably
 useful.

  I know when I put an app into an embedded app, I'd rather not even
  have the overhead of the plug-in mechanism, I want it smaller than
  that.

 You have a good point regarding customized compilation, though I think
 that most of the current features in Wget belong as core features. There
 are some small exceptions (egd sockets).

Thanks.

Well, when I'm building an embedded device, I look at the invocations
wget that are actually being called in the scripts.  Since the end
product has no interactive shell, I don't need to have all those extra
options enabled!  In fact, in wget's case, one can often dispense with
the tool entirely- the busybox version suffices.

  ... And when I'm running the gnu version of something I expect it
  to have verbose man pages and lots of double-dash options, that's what
  tools like less and grep are for.

 Well... many GNU tools actually lack verbose man pages, particularly
 since info is the preferred documentation system for GNU software.

Well, I guess I'm spoiled by Debian.  If it ain't broke, don't fix it.
 Debian makes man pages because tools should have manpages.  IIRC,
that was one of the divorce issues.

 Despite the fact that many important GNU utilities are very
 feature-packed, they also tend not to have options that are only useful
 to a relatively small number of people--particularly when equivalent
 effects are possible with preexisting options.

 As to the overhead of the plugin mechanism, you're right, and I may well
 decide to make that optionally compiled.

Well, I'd rather have rate-limiting things be optionally compiled than
plugged-in, since they'd be useful for embedded devices.

[Micah]
  It's not really about this option, it's about a class of options. I'm in
  the unenviable position of having to determine whether small patches
  that add options are sufficiently useful to justify the addition of the
  option. Adding one new option/rc command is not a problem. But when,
  over time, fifty people suggest little patches that offer options with
  small benefits, we've suddenly got fifty new options cluttering up the
  documentation and --help output.

[Jim]
  I would posit that the vast majority of wget options are used in some
  extremely small percentage of wget invocations.  Should they be removed?

[Micah]
 Such as which ones?

 I don't think we're talking about the same extremely small percentages.

OK, so so far there are three of us, I think, that find it potentially
useful.  And you have not addressed the use cases I brought up.  So I
think your extremely small percentages assumption may be faulty.

 Looking through the options listed with --help, I can find very few
 options that I've never used or would not consider vital in some
 situations I (or someone else) might encounter.

 This doesn't look to me like a vital function, one that a large number
 of users will find mildly useful, or one that a mild number of users
 will find extremely useful. This looks like one that a mild number of
 users will find mildly useful. Only slightly more useful, in fact, than
 what is already done.

You keep saying that.  You seem to think unknown upstream bandwidth is
a rare thing.  Or that wanting to be nice to other bandwidth users in
such a circumstance is a rare thing.  I wish I lived in your universe.
 Mine's a lot more sloppy.

 It's also one of those fuzzy features that addresses a scenario that
 has no right solution (JavaScript support is in that domain). These
 sorts of features tend to invite a gang of friends to help get a little
 bit closer to the unreachable target. For instance, if we include this
 option, then the same users will find another option to control the
 period of time spent full-bore just as useful. A pulse feature might
 be useful, but then you'll probably want an option to control the
 spacing between those, too. And someone else may wish to introduce an
 option that saves bandwidth information persistently, and uses this to
 make a good estimate from the beginning.

Ah, finally, some meat.  You see this as opening a door.  Especially
as I inquire as too whether anyone has feedback on my implementation,
you see it mushrooming into a plethora of options.

 And all of this would amount to a very mild improvement over what
 already exists.

Your universe crisp, mine sloppy (above) ;-)

  In my view, wget is a useful and flexible 

Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

2007-10-12 Thread Micah Cowan
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Tony Godshall wrote:
 [Jim]
 Well, we need the plugin architecture anyway. There are some planned
 features (JavaScript and MetaLink support being the main ones) that have
 no business in Wget proper, as far as I'm concerned, but are inarguably
 useful.
 
 I know when I put an app into an embedded app, I'd rather not even
 have the overhead of the plug-in mechanism, I want it smaller than
 that.
 
 You have a good point regarding customized compilation, though I think
 that most of the current features in Wget belong as core features. There
 are some small exceptions (egd sockets).
 
 Thanks.

(You've misattributed this: it's me talking here, not Jim.)

 OK, so so far there are three of us, I think, that find it potentially
 useful.

One of whom, you'll note, was happy to see it as a module, which is what
I had also been suggesting.

 This doesn't look to me like a vital function, one that a large number
 of users will find mildly useful, or one that a mild number of users
 will find extremely useful. This looks like one that a mild number of
 users will find mildly useful. Only slightly more useful, in fact, than
 what is already done.
 
 You keep saying that.  You seem to think unknown upstream bandwidth is
 a rare thing.  Or that wanting to be nice to other bandwidth users in
 such a circumstance is a rare thing.

I do not think it's a particularly rare thing. I think it's a fairly
easily-dealt-with thing.

 Like I said when I submitted the patch, this
 essentially automates what I do manually:
   wget somesite
   ctrl-c
   wget -c --limit-rate nnK somesite

What I've been trying to establish, is whether automating such a thing
(directly within Wget), is a useful-enough thing to justify the patch.

 But they mainly fall into the category of features that a large number
 of users will use occasionally, and a small number of users will find
 indispensable much of the time. Will you find this feature
 indispensable, or can you pretty much use --limit-rate with a reasonable
 value to do the same thing?
 
 Horse dead.  Parts rolling in the freeway.

Is it? I was talking to Jim, not you. He actually hadn't said very much
until this point.

 If, on the other hand, it is really, just a pretty minor improvement
 that happens to be mildly useful to you, could we please drop using this
 as a platform to predict what my future reactions to new features in
 general are likely to be? :p
 
 Well, when a guy first joins the list and submits his first patch and gets...

Gets what? One should not expect that all patches are automatically
accepted. Jim knows this, and has also seen other people come with
patches I've accepted, which is why it's just silly to accuse me of
something there's already ample proof I don't.

And what is it you got? Did I ever say, no, it's not going in? Did I
ever say I'm against it? What I repeatedly said was, I need convincing.

 Anyhow, perhaps I did the wrong thing in bringing it here- perhaps I
 should have provided it as a wishlist bug in debian and seen how many
 ordinary people find it useful before taking it to the source...
 perhaps I should have vetted it or whatever.

Sure, vetting it is entirely helpful. Getting feedback from a larger
community of users is very helpful. And, lamentably, the current
activity level of this list is not sufficient that I can gauge how
useful a feature is to the community as a whole from the five-or-so
people that participate on this list.

I cannot gauge how useful a feature is from how loudly the contributor
proclaims it's useful. I already _know_ you find it useful, as you cared
enough to bother writing a patch. What I was hoping to hear, but hadn't
heard much of until just now, was more support from the rest of the
community. Jim had spoken up, but not particularly strongly. Rather than
waiting for people to have the chance to speak up, though, you just got
louder.

What is most interesting to me, is your reaction to my statements, which
were never I'm not putting it in, but I think it should wait and live
as an accessory. And to this you get upset, and both defensive and
offensive. This does not make it likelier for me to include your changes.

In this specific case, there's probably a good chance it'll go in (not
for 1.11 though), as I'm clearer now on exactly how useful Jim finds it,
and we've also had another speak up. In the future, though, if you've
got something you'd like me to consider including, you might consider
just a bit more patience than you've exhibited this time around.

Hopefully this thread can go away now, unless someone has something
truly new to contribute.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHD/mZ7M8hyUobTrERCLCrAJ9wApBrS12uqTJ/5pNDLxNI9zbJkACgkdBr
gCwWMhPZ/kzzY1ynR8aof+g=
=t9s1

Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

2007-10-11 Thread Micah Cowan
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Tony Godshall wrote:
 On 10/11/07, Micah Cowan [EMAIL PROTECTED] wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 Tony Godshall wrote:
 On 10/10/07, Micah Cowan [EMAIL PROTECTED] wrote:
 My current impression is that this is a useful addition for some limited
 scenarios, but not particularly more useful than --limit-rate already
 is. That's part of what makes it a good candidate as a plugin.
 I guess I don't see how picking a reasonable rate automatically is
 less useful then having to know what the maximum upstream bandwidth is
 ahead of time.
 I never claimed it was less useful. In fact, I said it was more useful.
 My doubt is as to whether it is _significantly_ more useful.
 
 For me, yes.  For you, apparently not.  It's a small patch, really.
 Did you even look at it?

I have, yes. And yes, it's a very small patch. The issue isn't so much
about the extra code or code maintenance; it's more about extra
documentation, and avoiding too much clutter of documentation and lists
of options/rc-commands. I'm not very picky about adding little
improvements to Wget; I'm a little pickier about adding new options.

It's not really about this option, it's about a class of options. I'm in
the unenviable position of having to determine whether small patches
that add options are sufficiently useful to justify the addition of the
option. Adding one new option/rc command is not a problem. But when,
over time, fifty people suggest little patches that offer options with
small benefits, we've suddenly got fifty new options cluttering up the
documentation and --help output. If the benefits are such that only a
handful of people will ever use any of them, then they may not have been
worth the addition, and I'm probably not doing my job properly.

Particularly since a plugin architecture is planned, it seems ideal to
me to recommend that such things be implemented as plugins at that
point. In the meantime, people who find the feature sufficiently useful
can easily apply the patch to Wget themselves (that's part of what makes
Free Software great!), and even offer patched binaries up if there's
call for it.

If a number of people bother to download and install the patch, or fetch
 patched binaries in preference to the official binaries, that'd be a
good indicator that it's worth pulling in.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDsBj7M8hyUobTrERCIMPAJ9z936EGkfx7b/1sKAt3zw6OcPMIgCaAi2Y
qtNxSlmy09JSvtaWgZ42M7o=
=iRGw
-END PGP SIGNATURE-


Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

2007-10-11 Thread Tony Godshall
...
 I have, yes. And yes, it's a very small patch. The issue isn't so much
 about the extra code or code maintenance; it's more about extra
 documentation, and avoiding too much clutter of documentation and lists
 of options/rc-commands. I'm not very picky about adding little
 improvements to Wget; I'm a little pickier about adding new options.

 It's not really about this option, it's about a class of options. I'm in
 the unenviable position of having to determine whether small patches
 that add options are sufficiently useful to justify the addition of the
 option. Adding one new option/rc command is not a problem. But when,
 over time, fifty people suggest little patches that offer options with
 small benefits, we've suddenly got fifty new options cluttering up the
 documentation and --help output.

Would it be better, then, if I made it --limit-rate nn% instead of
limit-percent nn?
And made the descrip briefer?

  If the benefits are such that only a
 handful of people will ever use any of them, then they may not have been
 worth the addition, and I'm probably not doing my job properly. ...

I guess I'd like to see compile-time options so people could make a
tiny version for their embedded system, with most options and all
documentation stripped out, and a huge kitchen-sink all-the-bells
version and complete documentation for the power user version.  I
don't think you have to go to a totally new (plug in) architecture or
make the hard choices.

I know when I put an app into an embedded app, I'd rather not even
have the overhead of the plug-in mechanism, I want it smaller than
that.  And when I'm running the gnu version of something I expect it
to have verbose man pages and lots of double-dash options, that's what
tools like less and grep are for.

Tony


Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

2007-10-11 Thread Tony Godshall
On 10/11/07, Tony Godshall [EMAIL PROTECTED] wrote:
 ...
  I have, yes. And yes, it's a very small patch. The issue isn't so much
  about the extra code or code maintenance; it's more about extra
  documentation, and avoiding too much clutter of documentation and lists
  of options/rc-commands. I'm not very picky about adding little
  improvements to Wget; I'm a little pickier about adding new options.
 
  It's not really about this option, it's about a class of options. I'm in
  the unenviable position of having to determine whether small patches
  that add options are sufficiently useful to justify the addition of the
  option. Adding one new option/rc command is not a problem. But when,
  over time, fifty people suggest little patches that offer options with
  small benefits, we've suddenly got fifty new options cluttering up the
  documentation and --help output.

 Would it be better, then, if I made it --limit-rate nn% instead of
 limit-percent nn?
 And made the descrip briefer?

Also would it help if the behavior was changed so it pulsed
occasionally and therefore wouldn't suffer from the
initial-measurement-error case.

I'm trying to judge whether I should spend more time touching it up
into something acceptable or just let it remain a personal hack.

   If the benefits are such that only a
  handful of people will ever use any of them, then they may not have been
  worth the addition, and I'm probably not doing my job properly. ...

 I guess I'd like to see compile-time options so people could make a
 tiny version for their embedded system, with most options and all
 documentation stripped out, and a huge kitchen-sink all-the-bells
 version and complete documentation for the power user version.  I
 don't think you have to go to a totally new (plug in) architecture or
 make the hard choices.

 I know when I put an app into an embedded app, I'd rather not even
 have the overhead of the plug-in mechanism, I want it smaller than
 that.  And when I'm running the gnu version of something I expect it
 to have verbose man pages and lots of double-dash options, that's what
 tools like less and grep are for.

 Tony



-- 
Best Regards.
Please keep in touch.


Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

2007-10-11 Thread Micah Cowan
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Tony Godshall wrote:
 ...
 I have, yes. And yes, it's a very small patch. The issue isn't so much
 about the extra code or code maintenance; it's more about extra
 documentation, and avoiding too much clutter of documentation and lists
 of options/rc-commands. I'm not very picky about adding little
 improvements to Wget; I'm a little pickier about adding new options.

 It's not really about this option, it's about a class of options. I'm in
 the unenviable position of having to determine whether small patches
 that add options are sufficiently useful to justify the addition of the
 option. Adding one new option/rc command is not a problem. But when,
 over time, fifty people suggest little patches that offer options with
 small benefits, we've suddenly got fifty new options cluttering up the
 documentation and --help output.
 
 Would it be better, then, if I made it --limit-rate nn% instead of
 limit-percent nn?

The thought occurred to me; but then it's no longer such a small patch,
and we've just introduced a new argument parser function, so we still
get to is it justified?.

 And made the descrip briefer?

It's not really the length of the description that I'm concerned about,
it's just the number of little options.

  If the benefits are such that only a
 handful of people will ever use any of them, then they may not have been
 worth the addition, and I'm probably not doing my job properly. ...
 
 I guess I'd like to see compile-time options so people could make a
 tiny version for their embedded system, with most options and all
 documentation stripped out, and a huge kitchen-sink all-the-bells
 version and complete documentation for the power user version.  I
 don't think you have to go to a totally new (plug in) architecture or
 make the hard choices.

Well, we need the plugin architecture anyway. There are some planned
features (JavaScript and MetaLink support being the main ones) that have
no business in Wget proper, as far as I'm concerned, but are inarguably
useful.

You have a good point regarding customized compilation, though I think
that most of the current features in Wget belong as core features. There
are some small exceptions (egd sockets).

 I know when I put an app into an embedded app, I'd rather not even
 have the overhead of the plug-in mechanism, I want it smaller than
 that.  And when I'm running the gnu version of something I expect it
 to have verbose man pages and lots of double-dash options, that's what
 tools like less and grep are for.

Well... many GNU tools actually lack verbose man pages, particularly
since info is the preferred documentation system for GNU software.

Despite the fact that many important GNU utilities are very
feature-packed, they also tend not to have options that are only useful
to a relatively small number of people--particularly when equivalent
effects are possible with preexisting options.

As to the overhead of the plugin mechanism, you're right, and I may well
decide to make that optionally compiled.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDs2U7M8hyUobTrERCOYoAJ9bIfGbztes0MEfKxAPwpQY/bjJAQCeOAXn
8M6Kj1vLploBN+qENpF2gu8=
=K9Sb
-END PGP SIGNATURE-


Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

2007-10-11 Thread Jim Wright
On Thu, 11 Oct 2007, Micah Cowan wrote:

 It's not really about this option, it's about a class of options. I'm in
 the unenviable position of having to determine whether small patches
 that add options are sufficiently useful to justify the addition of the
 option. Adding one new option/rc command is not a problem. But when,
 over time, fifty people suggest little patches that offer options with
 small benefits, we've suddenly got fifty new options cluttering up the
 documentation and --help output.

I would posit that the vast majority of wget options are used in some
extremely small percentage of wget invocations.  Should they be removed?
In my view, wget is a useful and flexible tool largely because there
are a lot of options.  The internet is a messy place, and wget can cope.

I have a handful of options I've added to wget which are mandatory for
my use.  Mostly dealing with timeouts and retries.  Useful features which
would not commonly be used.  Am I correct in reading in to this discussion
that submitting new features is not encouraged?



Re: anyone look at the actual patch? anyone try it? [Re: working on patch to limit to percent of bandwidth]

2007-10-11 Thread Micah Cowan
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jim Wright wrote:
 On Thu, 11 Oct 2007, Micah Cowan wrote:
 
 It's not really about this option, it's about a class of options. I'm in
 the unenviable position of having to determine whether small patches
 that add options are sufficiently useful to justify the addition of the
 option. Adding one new option/rc command is not a problem. But when,
 over time, fifty people suggest little patches that offer options with
 small benefits, we've suddenly got fifty new options cluttering up the
 documentation and --help output.
 
 I would posit that the vast majority of wget options are used in some
 extremely small percentage of wget invocations.  Should they be removed?

Such as which ones?

I don't think we're talking about the same extremely small percentages.

Looking through the options listed with --help, I can find very few
options that I've never used or would not consider vital in some
situations I (or someone else) might encounter.

This doesn't look to me like a vital function, one that a large number
of users will find mildly useful, or one that a mild number of users
will find extremely useful. This looks like one that a mild number of
users will find mildly useful. Only slightly more useful, in fact, than
what is already done.

It's also one of those fuzzy features that addresses a scenario that
has no right solution (JavaScript support is in that domain). These
sorts of features tend to invite a gang of friends to help get a little
bit closer to the unreachable target. For instance, if we include this
option, then the same users will find another option to control the
period of time spent full-bore just as useful. A pulse feature might
be useful, but then you'll probably want an option to control the
spacing between those, too. And someone else may wish to introduce an
option that saves bandwidth information persistently, and uses this to
make a good estimate from the beginning.

And all of this would amount to a very mild improvement over what
already exists.

 In my view, wget is a useful and flexible tool largely because there
 are a lot of options.  The internet is a messy place, and wget can cope.

Sure. But what does --limit-percent allow wget to cope with that it
cannot currently?

 I have a handful of options I've added to wget which are mandatory for
 my use.  Mostly dealing with timeouts and retries.  Useful features which
 would not commonly be used.

But they mainly fall into the category of features that a large number
of users will use occasionally, and a small number of users will find
indispensable much of the time. Will you find this feature
indispensable, or can you pretty much use --limit-rate with a reasonable
value to do the same thing?

 Am I correct in reading in to this discussion
 that submitting new features is not encouraged?

To be honest, I'm shocked to get this sort of reaction over what is, as
far as I can tell, an extremely small improvement. If you really care
that much about it, I really don't mind putting it in. But if it's
really that useful to you, then I don't think your previous comments
really conveyed the degree to which that was the case. I repeatedly
asked for people to sell it for me, and got very little actual
case-making other than impressions that it was a very minor convenience
improvement. If it's more than a very minor improvement to you, then I
wish you'd have made that clearer from the start.

If, on the other hand, it is really, just a pretty minor improvement
that happens to be mildly useful to you, could we please drop using this
as a platform to predict what my future reactions to new features in
general are likely to be? :p

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDwIc7M8hyUobTrERCDJRAJ91SkMNlTc0ssUpejnyEuGp7MqvIwCgir9U
9t7oOJ8y40VerzlnhysFSXw=
=u5oK
-END PGP SIGNATURE-