Re: Mailing list migration?
Micah Cowan wrote: What do y'all think? I think you should remember to let gmane.org know of any changes :-). Other than that, no objections to changing. I'm a bit partial to @gnu.org, as that's (unsurprisingly) the most widely used for GNU projects. -- Matthew First time I've gotten a programming job that required a drug test. I was worried they were going to say 'you don't have enough LSD in your system to do Unix programming'. -- Paul Tomblin (from cluefire.net)
Re: getpass alternative [Re: getpass documentation]
Micah Cowan wrote: Matthew Woehlke wrote: Micah Cowan wrote: Wget just added code to the dev repo that uses gnu_getpass to support password prompting. This was mainly because it was quick-and-easy, and because gnu_getpass doesn't suffer from many of the serious flaws plaguing alternative implementations. Hehe, earlier today I merged my old, lame-but-functional patch with 1.11.3 (I've changed systems since last time). Does this mean that when fedora picks up 1.12 (after there *is* a 1.12 obviously :-) ) that I won't need to roll my own any more? ;-) That's the plan. Great news, thanks! Now... if Fedora will just pick it up... :-) I guess you're quoting from my post to the gnulib list? :) Right. -- Matthew I don't see what C++ has to do with keeping people from shooting themselves in the foot. C++ will happily load the gun, offer you a drink to steady your nerves, and help you aim. -- Peter da Silva (from cluefire.net)
Re: getpass alternative [Re: getpass documentation]
Micah Cowan wrote: Matthew Woehlke wrote: Great news, thanks! Now... if Fedora will just pick it up... :-) I don't see why they wouldn't; they're up-to-date with 1.11.3, as of today, which seems like a pretty quick pick-up, to me. Maybe, but Fedora 8 is only on 1.10.2 (the patched Red Hat one)... I'm not sure what their policy on switching minor versions is; hopefully it isn't 'stick with 1.11.x until Fedora 10' :-). -- Matthew I don't see what C++ has to do with keeping people from shooting themselves in the foot. C++ will happily load the gun, offer you a drink to steady your nerves, and help you aim. -- Peter da Silva (from cluefire.net)
Re: getpass alternative [Re: getpass documentation]
Micah Cowan wrote: Wget just added code to the dev repo that uses gnu_getpass to support password prompting. This was mainly because it was quick-and-easy, and because gnu_getpass doesn't suffer from many of the serious flaws plaguing alternative implementations. Hehe, earlier today I merged my old, lame-but-functional patch with 1.11.3 (I've changed systems since last time). Does this mean that when fedora picks up 1.12 (after there *is* a 1.12 obviously :-) ) that I won't need to roll my own any more? ;-) -- Matthew I want to vote for a Conservative Democrat. Too bad they're about as rare as an Honest Politician. Maybe I'll get lucky and someone will come along that's both.
Re: Problems mirroring an asp website
rafael bonifaz wrote: The file is downloaded with no problem. So my hypothesis is that the MS Internet Informatation service is bloking me because wget is doing to many http requests at the time that it may be seen as an DoS attack. Is this reasoning right? I've no idea, but if it is you might RTFM. In particular, look for the text Wait the specified number of seconds (or just wait will do) to learn how to ask wget to reduce the request rate. -- Matthew A pool hall put up a sign in their front window that read: Profound language prohibited within. I could just imagine some people discussing the meaning of life and being told to take it outside. -- Scott Adams
Re: --limit-percent N versus --limit-rate N% ?
Micah Cowan wrote: Also: does the current proposed patch deal properly with situations such as where the first 15 seconds haven't been taken up by part of a single download, but rather several very small ones? I'm not very familiar yet with the rate-limiting stuff, so I really have no idea. If the point is to limit *your* bandwidth, well it's hard to say, although the consensus seems to be that overly conservative is the right thing to do, usually. Of course, if the point (as in one suggested use case) is to limit the amount of the /server's/ bandwidth consumed, then a new percent should be calculated for every host. Just some thoughts... -- Matthew Five is right out! -- Cleric (from Monty Python's Quest for the Holy Grail)
Re: Port range option in bind-address implemented?
Micah Cowan wrote: Yes, that appears to work quite well, as long as we seed it right; starting with a consistent Xâ‚€ would be just as bad as trying them sequentially, and choosing something that does not change several times a second (such as time()) still makes it likely that multiple invocations will choose the same first port. Probably, /dev/random as first choice, falling back to by gettimeofday() where that's available. I don't know what Windows would use. QueryPerformanceCounter, if it seems trustworthy (i.e. != 0), which might be what Hrvoje pointed out. Otherwise the UUID generator is theoretically sufficient (and there is always getpid() which can be used to salt time... which is likely what the UUID generator does, anyway). -- Matthew Five is right out! -- Cleric (from Monty Python's Quest for the Holy Grail)
Re: Wget on Mercurial!
Micah Cowan wrote: Since I do virtually all my work on a laptop, which is usually but not always connected to the Wired, I have begun experimenting with distributed SCMs. I have recently been using Mercurial for work on Wget, and then synching the work with Subversion, and am very happy with it so far. I would like to consider moving Wget development from Subversion to Mercurial at some point in the future, if it continues to work well. Have you already considered and rejected git? I'm not trying to start a VCS war, it's just that I've seen some other GNU projects (notably coreutils and gnulib) moving to git and want to know you didn't pick hg just because it's the first DVCS you liked. -- Matthew Nobody expects the traditional Bourne shell!
Re: changing url
Aram Wool wrote: hi, I'm using wget with this url: http://www.twis.org/audio/podpress_trac/web/147/0/TWIS_2007_09_11.mp3 the directory named 147 increases by 1 each week, corresponding to an mp3 with a new date. I can use macros to automatically deal with a changing date, but haven't been able to find out how to make wget go to directory 148 the following week, etc. without manually changing the url. I'd expect there's an easy solution to such a simple problem. I don't think this is a problem for wget to solve (how would it?). What you can do though is keep track of what URL you downloaded last, and use that. Something like this: $ mkdir /var/tmp/$(id -un) $ echo 147 /var/tmp/$(id -un)/teh_last_mp3 $ cat getnext.sh #!/bin/bash lastnum=$(cat /var/tmp/$(id -un)/teh_last_mp3) (( nextnum = lastnum + 1 )) wget url-using-$nextnum-and-date echo $nextnum /var/tmp/$(id -un)/teh_last_mp3 ...unless you wanted something based on the week number from the date, in which case 'info date' is your friend (hint: %U). -- Matthew Non sequitor. Your facts are out of order. -- Nomad
Re: trouble with -p
Micah Cowan wrote: Brian Keck wrote: Sometimes -p doesn't work. For instance: wget -p http://en.wikipedia.org/wiki/Herbig-Haro_object In this case, it appears that you've bumped into the fact that wget, by default, will refuse to cross hostname boundaries to download things, unless you tell it otherwise. You want the -H option. Hmm, an interesting observation from that... am I missing something, or is there not currently an easy way to tell wget to span hosts in the same domain, but not span domains? For example, spanning to upload.wikipedia.org makes sense when grabbing from en.wikipedia.org, but spanning to casa.colorado.edu, www.daviddarling.info or sparky.rice.edu (to steal the external references from the mentioned article) probably isn't desired. Might be a useful wish for some point in the unspecified future. -- Matthew So long, and thanks for all the fish -- the dolphins
Re: patch: prompt for password
Maciej W. Rozycki wrote:
On Tue, 24 Jul 2007, Matthew Woehlke wrote:
This rough patch adds a '--ask-password' option to wget. About all that can be
It's a good rule to send patches inlined rather than attached as that
makes them easy to comment on.
It's also a good rule to send patches as attachments so that dumb
mailers don't break them :-). (And some mailers, e.g. Thunderbird, will
inline text attachments when replying anyway.)
said for it is that it works; hopefully it will serve as a useful proof of
concept and possible a starting point for a fully-developed feature.
Hmm, what's there to prove? Anyway...
Demonstration implementation?
Index: src/http.c
===
--- src/http.c (revision 2295)
+++ src/http.c (working copy)
@@ -31,6 +31,7 @@
#include stdio.h
#include stdlib.h
#include string.h
+#include termios.h /* FIXME probably not portable? */
#ifdef HAVE_UNISTD_H
# include unistd.h
#endif
AC_SYS_POSIX_TERMIOS
@@ -1419,7 +1421,29 @@
passwd = u-passwd;
search_netrc (u-host, (const char **)user, (const char **)passwd, 0);
user = user ? user : (opt.http_user ? opt.http_user : opt.user);
+ if (opt.ask_passwd)
+{
+ int in_fd = fileno(stdin);
Standard input is used for URLs; you probably want to use stderr here.
Read from stderr? I admit I've heard stderr is bi-directional but I
can't say I've ever seen it used to read input.
Formatting (ditto all the function calls below).
I both dislike GNU formatting and am not familiar with it, so obviously
someone that understands it will have to fix this.
+ printf(URL \%s\\nPassword for user \%s\: , u-url, user);
Standard output is used for storing remote files; you definitely want to
use stderr here.
Good point.
+ /* TODO check error */
+ tcgetattr(in_fd, attr);
isatty() beforehand perhaps?
Hmm, good point #2. I did say it works not it's a good patch, right? ;-)
Though I am not sure whether it is
really needed given that many years have passed and nobody wanted such a
feature.
Which strikes me as odd, really, but I would guess not so many people
use wget for sites that need a password (I never did until recently), at
least not on non-private computers. I.e. if they do, they're ok with the
existing security risks.
Otherwise it looks OK, I think.
But the decision is up to the maintainer (once you sort out
technical problems).
I very much doubt this patch will be accepted, there are huge problems
with it yet (like, it should persist the password, work with ftp, etc).
It was meant more as a starting point, and for commentary... and you've
made some good comments, thanks!
--
Matthew
Microsoft: Expect the unexpected
Re: patch: prompt for password
Maciej W. Rozycki wrote: On Wed, 25 Jul 2007, Matthew Woehlke wrote: Read from stderr? I admit I've heard stderr is bi-directional but I can't say I've ever seen it used to read input. Well, `wget' can be used as a filter (and I did do so a couple of times in some scripts in the past; I would be surprised if others did not), so you better keep its standard input and output intact. You could try to fopen(ttyname(fileno(stderr)), r) or suchlike instead, but stderr should just work. You may want to have a look at what `ssh' does for example. Well, I tried, and obviously reading from stderr is complicated (if even do-able). Maybe you can improve the patch? Well, chances are nobody could be bothered to fix your patch if you submit it incorrectly formatted; I would certainly not if I were the maintainer. ...which I would be a lot more inclined to do if I had any expectation of the patch being accepted as-is, which I don't. I expect it to be used as a reference (if it is even needed) to DTRT. If Micah wants to ask for my help there, that's one thing. Otherwise I fail to see how this conversation is worth anyone's time. Which strikes me as odd, really, but I would guess not so many people use wget for sites that need a password (I never did until recently), at least not on non-private computers. I.e. if they do, they're ok with the existing security risks. You can store your password in the input file provided to `wget' so it will be concealed from all the other users of the system apart from the administrator. And if you do not trust your administrator, you are lost anyway (they could log all your keystrokes if they wanted to), so why bother? There is a difference between trusting my coworkers and leaving a major password sitting in plain text :-). -- Matthew Microsoft: Expect the unexpected
Re: patch: prompt for password
Josh Williams wrote: On 7/25/07, Matthew Woehlke [EMAIL PROTECTED] wrote: Any reason you're not replying to the list? (Unless there is, please direct replies to the list.) No, I was in a hurry at the time and forgot to change the e-mail address before I sent it. Reply All ;-) (Although if people want to drop me from being CC'd that's fine also; I know the general policy varies by list, but really, I personally can read the list fine and don't need to be copied :-).) If you're interested, we could use someone to help develop an *optional* user interface (browse the list archives if you need some ideas). If Micah asks for my help specifically, that's fine. Otherwise I've got wget workingfor the particular use case that I absolutely had to have so that I could get back to other work in my day job :-). -- Matthew Microsoft: Expect the unexpected
patch: prompt for password
This rough patch adds a '--ask-password' option to wget. About all that
can be said for it is that it works; hopefully it will serve as a useful
proof of concept and possible a starting point for a fully-developed
feature.
--
Matthew
Resistance is futile -- Borg
No, resistance is the reciprocal of conductance -- Dave Korn
Index: src/options.h
===
--- src/options.h(revision 2295)
+++ src/options.h(working copy)
@@ -81,6 +81,7 @@
char *user; /* Generic username */
char *passwd;/* Generic password */
+ bool ask_passwd; /* Ask for password? */
bool always_rest;/* Always use REST. */
char *ftp_user; /* FTP username */
Index: src/init.c
===
--- src/init.c (revision 2295)
+++ src/init.c (working copy)
@@ -112,6 +112,7 @@
{ accept, opt.accepts, cmd_vector },
{ addhostdir, opt.add_hostdir, cmd_boolean },
{ alwaysrest, opt.always_rest, cmd_boolean }, /* deprecated */
+ { askpassword, opt.ask_passwd,cmd_boolean },
{ background, opt.background,cmd_boolean },
{ backupconverted, opt.backup_converted, cmd_boolean },
{ backups, opt.backups, cmd_number },
Index: src/http.c
===
--- src/http.c (revision 2295)
+++ src/http.c (working copy)
@@ -31,6 +31,7 @@
#include stdio.h
#include stdlib.h
#include string.h
+#include termios.h /* FIXME probably not portable? */
#ifdef HAVE_UNISTD_H
# include unistd.h
#endif
@@ -1302,6 +1303,7 @@
char *type;
char *user, *passwd;
+ char buffer[1024]; /* for ask_passwd */
char *proxyauth;
int statcode;
int write_error;
@@ -1419,7 +1421,29 @@
passwd = u-passwd;
search_netrc (u-host, (const char **)user, (const char **)passwd, 0);
user = user ? user : (opt.http_user ? opt.http_user : opt.user);
+ if (opt.ask_passwd)
+{
+ int in_fd = fileno(stdin);
+ struct termios attr;
+ tcflag_t old_flags;
+ int n;
+ printf(URL \%s\\nPassword for user \%s\: , u-url, user);
+ /* TODO check error */
+ tcgetattr(in_fd, attr);
+ old_flags = attr.c_lflag;
+ attr.c_lflag = ~ECHO;
+ tcsetattr(in_fd, TCSANOW, attr);
+ fgets(buffer, sizeof(buffer), stdin);
+ attr.c_lflag = old_flags;
+ tcsetattr(in_fd, TCSANOW, attr);
+ n = strlen(buffer) - 1;
+ if (n = 0 buffer[n] == '\n') buffer[n] = 0;
+ passwd = buffer;
+}
+ else
+{
passwd = passwd ? passwd : (opt.http_passwd ? opt.http_passwd : opt.passwd);
+}
if (user passwd)
{
Index: src/main.c
===
--- src/main.c (revision 2295)
+++ src/main.c (working copy)
@@ -129,6 +129,7 @@
{
{ accept, 'A', OPT_VALUE, accept, -1 },
{ append-output, 'a', OPT__APPEND_OUTPUT, NULL, required_argument },
+{ ask-password, 0, OPT_BOOLEAN, askpassword, -1 },
{ background, 'b', OPT_BOOLEAN, background, -1 },
{ backup-converted, 'K', OPT_BOOLEAN, backupconverted, -1 },
{ backups, 0, OPT_BOOLEAN, backups, -1 },
@@ -464,6 +465,8 @@
--user=USER set both ftp and http user to USER.\n),
N_(\
--password=PASS set both ftp and http password to PASS.\n),
+N_(\
+ --ask-passwordprompt for passwords.\n),
\n,
N_(\
Re: Bug update notifications
Micah Cowan wrote: Matthew Woehlke wrote: Micah Cowan wrote: ...any reason to not CC bug updates here also/instead? That's how e.g. kwrite does thing (also several other lists AFAIK), and seems to make sense. This is 'bug-wget' after all :-). It is; but it's also 'wget'. Hmm, so it is; my bad :-). While I agree that it probably makes sense to send it to a bugs discussion list, this list is a combination bugs/development/support/general discussion list, and I'm not certain it's appropriate to bump up the traffic level for this. Still, if there are enough folks that would like to get these updates (without also seeing commit notifications), perhaps we could craft a second list for this (or, alternatively, split off the main discussion/support list from the bugs list)? I guess a common pattern is: foo-help foo-devel foo-commits ...but of course you're the maintainer, it's your call :-). (The above aren't necessarily actual names of course, just the categories it seems like I'm most used to seeing. e.g. the GNU convention is of course bug-foo, not foo-devel.) -- Matthew This .sig is false
how to feed password to wget securely?
http://www.mail-archive.com/wget@sunsite.dk/msg06979.html Did this patch make it into the soon-to-be-released version (1.10.3?)? I need to wget a webpage that wants authentication, and I don't want to have to put it on disk or have it show up in 'ps'. -- Matthew ESIG: .sig file not available
Re: how to feed password to wget securely?
Micah Cowan wrote: Matthew Woehlke wrote: http://www.mail-archive.com/wget@sunsite.dk/msg06979.html Did this patch make it into the soon-to-be-released version (1.10.3?)? I need to wget a webpage that wants authentication, and I don't want to have to put it on disk or have it show up in 'ps'. I can't find the message (with accompanying patch) that is referred to by the message in that link. However, Mauro's response seemed to indicate that he didn't like that particular method for telling wget to prompt for password. I'd agree with Mauro that the option should be --prompt-passwd or some such. Ah, I see now, the first patch must have been something else. At any rate, I don't believe the upcoming 1.11 release is going to include a secure password prompt. I'm somewhat surprised that there hasn't been one in all these years. Adding such a feature will be one of my top priorities for the following release. Ok, thanks. I'll try to watch CVS (may even send you a patch), I want/need it rather urgently (for a script that talks to a website that didn't need authentication last time I used said script). However, I won't hold the next release for such a change, as we're just too close. Understood. :-) Thanks for the quick reply! I'm already going to be delaying the release a bit for a couple of (higher-priority) security fixes that I simply would not be comfortable releasing without--for instance, if you are uncomfortable with the idea of putting your password on disk or in the process table, how comfortable are you with the idea that in every version of wget up until now, it sends that password in the clear, regardless of whether or not the remote server is using cleartext password authentication (only applies to http, not https, situations)? Yes, I saw that thread. Alas, I often have to connect via telnet to boxes that want the same password, so in that sense I'm rather already screwed. -- Matthew ESIG: .sig file not available
