scan.coverity.com
Hey Just to let you know, it seems that coverity.com scans/scanned wget as part of their scan project, and I belive a wget person could get a signin and get to see the details from that: http://scan.coverity.com/rung0.html We got curl added as well and coverity did find a range of rather non-trivial and non-obvious flaws in the code that we could fix. (I have no relation to these guys other than that I'm enjoying being part of a(nother) project that they scan for us.)
Re: automake-ify wget [0/N]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ralf Wildenhues wrote: Hello Micah, * Micah Cowan wrote on Sat, Oct 06, 2007 at 12:04:53AM CEST: Ralf Wildenhues wrote: Remove all files that will be installed/governed by automake: (We keep deprecated mkinstalldirs for now, as po/Makefile.in.in still uses it. We keep config.rpath until the switch to a newer gettext infrastructure.) svn remove config.guess config.sub install-sh doc/version.texi As to the first three of these; wouldn't it be better to update them _once_, and then not use --force as part of the autoreconf stuff? I thought --force was for the occasion when you want to refresh a few things, and not for habitual use? Sure. But without --force, files installed by automake, such as install-sh, missing, config.guess etc., will not be updated. The rationale for this behavior is to allow for local changes to these files to be preserved. But yes, it is a free choice of yours to make, both approaches are viable. I'll probably remove autogen.sh altogether, and just replace its use in the instructions with autoreconf. And yes, configure.in will be renamed to configure.ac Fine with me. Should I post updated patches for all of these modifications? I've already applied most of your patches so far, some with my own tweaks/modifications. I've opted not to apply the patch to Makefile.in.in to make src writable, because I'd prefer to have a broken make distcheck than a falsely working one. :) I also removed the realclean target (may decide to add it back in at some point), as Automake has, IMO, more than enough *clean targets, and leaving it out keeps the Makefile.am's cleaner. I moved sample.wgetrc.munged_for_texi_inclusion from EXTRA_DIST to wget_TEXINFOS, as this allowed make doc to work (it accidentally worked for you with make distcheck, because make distcheck built it for the distribution; make wasn't working. I also took some liberties with the ChangeLogs, reducing much of the work done to, essentially, Makefile.am: converted from Makefile.in for automake, or something similar. You can see my current progress at http://hg.addictivecode.org/wget/automakify (get a copy of Mercurial and hg clone from there to get a repository). If you could produce further patches from that foundation, I'd appreciate it. If getting Mercurial is inconvenient, I've temporarily enabled tarballing at that location, so you can just fetch the tree by clicking a bz2 or gz link; however, Mercurial is recommended, as you'll be able to inspect the history of changes, and, if you use Mercurial to track your own modifications, it'll be that much easier for me to pull or import them from you. :) - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHCoQr7M8hyUobTrERCGSTAJ9ODT5QbtBRFgE+jI+jr5JQzNQjggCfWTO9 HNwBEnWFtW2PGSjjzx851x4= =2In3 -END PGP SIGNATURE-
working on patch to limit to percent of bandwidth
Hi New to the list. Wrote a patch that Works For Me to limit to a percent of measured bandwidth. This is useful, like --limit-rate, in cases where an upstream switch is poorly made and interactive users get locked out when a single box does a wget, but limit-pct is more automatic in the sense that you don't have to know ahead of time how big your downstream pipe is. I.e., what I used to do is (1) wget, look at the bandwidth I was getting, and then (2) Ctrl-C, (3) Ctrl-P and edit the line to add -c --limit-rate nnK (where nn is a bit less than I was getting). Now I can wget --limit-pct 50 and it will go full-speed for a bit and then back off till till the average speed is 50% of what the we saw during that time. The heuristic I'm using is to download full-speed for 15 seconds and then back off- that seems to work on my connection (too much less and measured rate is erratic, too much more and the defective upstream switch locks interactive folks out long enough that they notice and complain). Does that seem reasonable to folks or should it be parameterized. I'm not sure I can spend much time on complex parameter handling etc. right now. Anyhow, does this seem like something others of you could use? Should I submit the patch to the submit list or should I post it here for people to hash out any parameterization niceties etc first? -- Best Regards. Please keep in touch.
Re: working on patch to limit to percent of bandwidth
On 10/8/07, A. P. Godshall [EMAIL PROTECTED] wrote: Anyhow, does this seem like something others of you could use? Should I submit the patch to the submit list or should I post it here for people to hash out any parameterization niceties etc first? Go ahead and send it on here so we can comment on the code :-)
Re: working on patch to limit to percent of bandwidth
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 A. P. Godshall wrote: Hi New to the list. Welcome! Wrote a patch that Works For Me to limit to a percent of measured bandwidth. This is useful, like --limit-rate, in cases where an upstream switch is poorly made and interactive users get locked out when a single box does a wget, but limit-pct is more automatic in the sense that you don't have to know ahead of time how big your downstream pipe is. I.e., what I used to do is (1) wget, look at the bandwidth I was getting, and then (2) Ctrl-C, (3) Ctrl-P and edit the line to add -c --limit-rate nnK (where nn is a bit less than I was getting). Now I can wget --limit-pct 50 and it will go full-speed for a bit and then back off till till the average speed is 50% of what the we saw during that time. The heuristic I'm using is to download full-speed for 15 seconds and then back off- that seems to work on my connection (too much less and measured rate is erratic, too much more and the defective upstream switch locks interactive folks out long enough that they notice and complain). Does that seem reasonable to folks or should it be parameterized. I'm not sure I can spend much time on complex parameter handling etc. right now. Anyhow, does this seem like something others of you could use? Should I submit the patch to the submit list or should I post it here for people to hash out any parameterization niceties etc first? The best place to submit patches is to [EMAIL PROTECTED] Discussions can also happen there, too. Please submit patches against relatively recent development code, as opposed to the latest release (which happened ~two years ago), as otherwise the work involved in bringing it up-to-date may be a disincentive to include it. :) Information on obtaining the latest development version of Wget is at http://wget.addictivecode.org/RepositoryAccess. As to whether or not it will be included in mainline Wget, that depends on the answer to your question, does this seem like something others of you could use? I, personally, wouldn't find it very useful (I rarely use even --limit-rate), so I'd be interested in knowing who would. I suspect that it may well be the case that most folks who have need of - --limit-rate will find your version handy, but I want to hear from them. :) Also, I'd like a little more understanding about what the use case is: is it just to use N% less bandwidth than you seem to have available, so that other connections you may open won't be taxed as much? A couple notes off the bat: I'd prefer --limit-percent to --limit-pct, as apparently more recent naming conventions seem to prefer unabbreviated terms to abbreviated ones. And, I would desire a wgetrc command to complement the long option version. Don't break your back working these things into your patch, though: let's first see what you've got and whether folks want it (lest you waste effort for nothing). - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHCqBJ7M8hyUobTrERCFXFAJ9/BQrl9B+SajnuXgxCtZWRyPITUQCeKSl/ oQzSzJuGNDZhSP6GNKq9wkI= =AgXg -END PGP SIGNATURE-
Re: working on patch to limit to percent of bandwidth
And here's another post that apparently got sent with an erroneous signature. I think I may have figured out what was wrong; I specifically remember that I was still holding shift down when I typed one of the spaces in my passphrase... maybe that results in some screwiness... -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/
Initial draft- patch to limit bandwidth by percent of measured rate
Please find attached...
The quick test:
If you run wget with --limit-percent 50, you should see it run at full
blast for 15 seconds and then back off till it's downloading at 50%
the rate it acheived in the first 15 seconds.
This is only the initial Works For Me version of the patch. Comments
welcome, as anyone else who wants to run with it is welcome to do so.
Best Regards.
Tony
PS to Micah: Yes, I changed pct to percent.
diff --git a/src/init.c b/src/init.c
--- a/src/init.c
+++ b/src/init.c
@@ -179,6 +179,7 @@ static const struct {
#endif
{ input,opt.input_filename,cmd_file },
{ keepsessioncookies, opt.keep_session_cookies, cmd_boolean },
+ { limitpercent, opt.limit_percent, cmd_number },
{ limitrate,opt.limit_rate,cmd_bytes },
{ loadcookies, opt.cookies_input, cmd_file },
{ logfile, opt.lfilename, cmd_file },
diff --git a/src/main.c b/src/main.c
--- a/src/main.c
+++ b/src/main.c
@@ -189,6 +189,7 @@ static struct cmdline_option option_data
{ input-file, 'i', OPT_VALUE, input, -1 },
{ keep-session-cookies, 0, OPT_BOOLEAN, keepsessioncookies, -1 },
{ level, 'l', OPT_VALUE, reclevel, -1 },
+{ limit-percent, 0, OPT_VALUE, limitpercent, -1 },
{ limit-rate, 0, OPT_VALUE, limitrate, -1 },
{ load-cookies, 0, OPT_VALUE, loadcookies, -1 },
{ max-redirect, 0, OPT_VALUE, maxredirect, -1 },
@@ -453,6 +454,10 @@ Download:\n),
-Q, --quota=NUMBERset retrieval quota to NUMBER.\n),
N_(\
--bind-address=ADDRESSbind to ADDRESS (hostname or IP) on local host.\n),
+N_(\
+ --limit-percent=NUMBERlimit download rate to NUMBER percent of measured initial burst\n),
+N_(\
+ or rate specified by --limit-rate\n),
N_(\
--limit-rate=RATE limit download rate to RATE.\n),
N_(\
diff --git a/src/options.h b/src/options.h
--- a/src/options.h
+++ b/src/options.h
@@ -115,6 +115,8 @@ struct options
double waitretry; /* The wait period between retries. - HEH */
bool use_robots; /* Do we heed robots.txt? */
+ wgint limit_percent; /* Limit the download rate to this percentage
+ of initial measured burst rate. */
wgint limit_rate; /* Limit the download rate to this
many bps. */
SUM_SIZE_INT quota; /* Maximum file size to download and
diff --git a/src/retr.c b/src/retr.c
--- a/src/retr.c
+++ b/src/retr.c
@@ -86,14 +86,61 @@ limit_bandwidth (wgint bytes, struct pti
limit_bandwidth (wgint bytes, struct ptimer *timer)
{
double delta_t = ptimer_read (timer) - limit_data.chunk_start;
- double expected;
+ double expected= 0.0;
limit_data.chunk_bytes += bytes;
+ static wgint measured_limit= 0;
+
+ wgint limit= 0;
+
+ if ( opt.limit_rate )
+ {
+limit= opt.limit_rate;
+
+if ( opt.limit_percent )
+{
+ limit= limit * opt.limit_percent / 100;
+}
+DEBUGP((fixed limit governs: %lld bps\n, limit));
+ }
+ else
+ if ( opt.limit_percent )
+ {
+if ( ! measured_limit )
+{
+ static double total_sec= 0.0;
+ static wgint total_bytes= 0;
+
+ total_sec += delta_t;
+ total_bytes += bytes;
+ const double MEASURE_SEC= 15.0;
+
+ if ( total_sec MEASURE_SEC )
+ {
+DEBUGP((After %.3f seconds we saw %lld bytes so our measured limit is %lld bps\n,
+total_sec / 1000, total_bytes, measured_limit ));
+measured_limit = total_bytes / total_sec * opt.limit_percent / 100.0;
+ }
+}
+
+if ( measured_limit )
+{
+ if ( !limit || measured_limit limit )
+ {
+limit= measured_limit;
+ }
+}
+ }
+
+ if ( limit )
+ {
+
/* Calculate the amount of time we expect downloading the chunk
- should take. If in reality it took less time, sleep to
- compensate for the difference. */
- expected = (double) limit_data.chunk_bytes / opt.limit_rate;
+ should take at this fixed rate. If in reality it took less time,
+ sleep to compensate for the difference. */
+
+ expected = (double) limit_data.chunk_bytes / limit;
if (expected delta_t)
{
@@ -127,6 +174,8 @@ limit_bandwidth (wgint bytes, struct pti
limit_data.sleep_adjust = -0.5;
}
+ }
+
limit_data.chunk_bytes = 0;
limit_data.chunk_start = ptimer_read (timer);
}
@@ -229,13 +278,13 @@ fd_read_body (int fd, FILE *out, wgint t
progress_interactive = progress_interactive_p (progress);
}
- if (opt.limit_rate)
+ if (opt.limit_rate || opt.limit_percent)
limit_bandwidth_reset ();
/* A timer is needed for tracking progress, for throttling, and for
tracking elapsed time. If either of these are requested, start
the timer. */
- if (progress || opt.limit_rate || elapsed)
+ if (progress || opt.limit_rate || opt.limit_percent || elapsed)
{
timer = ptimer_new ();
last_successful_read_tm = 0;
@@ -286,7
