Re: Cookie options
On Tue, 4 Nov 2003, Hrvoje Niksic wrote: Interestingly enough, curl has a `--junk-session-cookies', which indicates that it keeps them by default (?). Daniel, are you still listening? :-) I am, and that is correct! I never thought that single-shot curl invokes could be seen as single sessions and therefore I think removing (or not saving) session cookies is plain wrong and will only make things harder and more confusing to users. But then, I believe the average usage of wget and curl differ a bit. Many users that use curl to script actions and that use cookies, operate with multiple curl command lines. The option named --junk-session-cookies is a rather new baby in curl land, as previously curl never ignored session cookies at load-time. Also, curl doesn't have any option to send out-of-date cookies such as has been proposed here as an option for wget, and I've never got a request from a single soul to add it to curl. I'd say it is an indication that such an option would not be widely used. (Given that curl has been used for scripting POSTs, logins and cookie stuff for years already.) -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
RE: About termination of wget and -T option
Hrvoje, This is different from what I observed: - I execute a cgi by wget. The cgi should return a file of about 7MB. - Data start coming, and after 15 minutes, while the file is not complete, wget starts a new trial. Luigi -Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Tuesday, November 04, 2003 9:35 PM To: Luigi Stefano Sona (lsona) Cc: [EMAIL PROTECTED] Subject: Re: About termination of wget and -T option Luigi Stefano Sona (lsona) [EMAIL PROTECTED] writes: About -T, you confirm that anyway, the timeout is for the start of the answer, not for the finish ? Almost. In fact, the timeout applies whenever the download stalls, at any point when Wget waits for data to arrive. You can think of it this way: Wget reads data in a loop like this one: while data_pending: read_chunk_from_network write_chunk_to_disk If a read_chunk_from_network step takes more than 15min, the download is interrupted (and retried). But the whole download can take as long as it takes.
Re: Cookie options
Daniel Stenberg [EMAIL PROTECTED] writes: I never thought that single-shot curl invokes could be seen as single sessions and therefore I think removing (or not saving) session cookies is plain wrong and will only make things harder and more confusing to users. Good point, although I somewhat disagree about the plain wrong part. From what I gather, the `cookies.txt' file to which Wget and curl dump their cookies is meant to hold permanent cookies. Putting session cookies inside might confuse other browsers. That notwithstanding, another problem is that, since session cookies have no expiry time, saving them by default would make them effectively immortal, which sounds quite wrong to me. I may be missing curl-specific details, though. The option named --junk-session-cookies is a rather new baby in curl land, as previously curl never ignored session cookies at load-time. Hmm. Wget will not ignore session cookies at load-time (any more), regardless of options. `--keep-session-cookies' causes it not to discard them at save time. Also, curl doesn't have any option to send out-of-date cookies such as has been proposed here as an option for wget, and I've never got a request from a single soul to add it to curl. I'd say it is an indication that such an option would not be widely used. (Given that curl has been used for scripting POSTs, logins and cookie stuff for years already.) Thanks for the insight. If someone *really* wants to send stale cookies, he can still hand-hack the cookies.txt file.
Re: About termination of wget and -T option
Luigi Stefano Sona (lsona) [EMAIL PROTECTED] writes: Hrvoje, This is different from what I observed: - I execute a cgi by wget. The cgi should return a file of about 7MB. - Data start coming, and after 15 minutes, while the file is not complete, wget starts a new trial. I don't doubt your observation, but it might well be the result of a different problem. The debug output might help narrow it down.
Re: FTP time-stamping and time zones
Jochen Hayek [EMAIL PROTECTED] writes: But as FTP directory listings do not include the time zone, the files live in, (I think) wget just assumes the local time zone to be identical to the remote one. Am I right with this? As long as this concept is being kept to entirely, this is probably the best, that can be done. You're right. Wget assumes local time zone and sets the local time stamp. Even if the time stamp is not correct (because of time zone difference), it will be *the same* incorrect time stamp each time around and mirroring will work. But it leads to a situation, that wget creates files with time stamps much older, than the files actually are. Or newer -- yes. Until MLST/MDTM are supported, there is no good cure for this (that I can think of). I wouldn't mind making wget believe (maybe through setting the environment variable TZ) it actually lives in New-England, although it lives here in Europe. That might work. Wget uses the library functions for time conversions, and they should respect TZ.
Re: Cookie options
On Wed, 5 Nov 2003, Hrvoje Niksic wrote: Good point, although I somewhat disagree about the plain wrong part. From what I gather, the `cookies.txt' file to which Wget and curl dump their cookies is meant to hold permanent cookies. Putting session cookies inside might confuse other browsers. Not necessarily, since they are stored without any expire time (using zero) and ordinary non-session cookies always have a number different than zero in that field. But sure, they might still confuse a browser that reads the file. I've never considered that a problem and I've never got any problems with that behavior reported. I believe very few people actually try that stunt. That notwithstanding, another problem is that, since session cookies have no expiry time, saving them by default would make them effectively immortal, which sounds quite wrong to me. I may be missing curl-specific details, though. Yes, it makes them immortal and lets the user decide when the session ends (by using the correct option). There is no one but the user who can tell when one session ends and another starts. In most cases, people will make a fresh cookie file each time they start a new session and then just keep session cookies around until they're done for this time. (Based on what I've done myself and seen others do with automated curl operations using cookies.) Hmm. Wget will not ignore session cookies at load-time (any more), regardless of options. `--keep-session-cookies' causes it not to discard them at save time. Right, but then again people who use wget are more likely (I'm guessing here) to use one single wget command line and consider that to be the full session. In curl's case, when using cookies you will very often enter multiple command lines to sort out a single session's business. -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
retrieving only index.html
Hi, we have a problem downloading files using wget version 1.9 without explicitly providing the file names. With --accept we have specified the files we want to receive. Version 1.7 works as expected. Version 1.9 is retrieving the index-file only. We have searched the internet for any clues. But no luck. We want to get all files that have the extensions zip and ini. Version 1.7 provides the index file but also the other files. The shell script is listed below. Thanks for your attention #!/bin/sh VIRPATH=/usrhome/ftp/pub/antivir CHKPATH=/usrhome/ftp/antivir.new SERVERS=ftp.nai.com ftpeur.nai.com ftpde.nai.com URL=pub/antivirus/datfiles/4.x/ # autoreorg: TMPLOG=/tmp/$P.log.$$ http_proxy=x.x.x.x:x ftp_proxy=x.x.x.x: export ftp_proxy export http_proxy for server in $SERVERS ; do echo +++ Pruefe ftp://$server/$URL; rm -f index.html ./wget \ --no-directories \ --no-parent \ --recursive \ --level=1 \ --no-clobber \ --cache=off \ --proxy=on \ --accept=.zip,.ini \ ftp://$server/$URL rc=$? [ $rc = 0 ] break done Output using wget version 1.7 This is actualiy the second call. Just to make the output shorter. All files have been downloaded with the first call. +++ Testing ftp://ftp.nai.com/pub/antivirus/datfiles/4.x/ --12:36:47-- ftp://ftp.nai.com/pub/antivirus/datfiles/4.x/ = `index.html' Proxy request sent, awaiting response... 200 OK Length: unspecified [text/html] 0K .. @ 11.18 KB/s 12:36:50 (11.18 KB/s) - `index.html' saved [2748] File `dat-4301.zip' already there, will not retrieve. File `delta.ini' already there, will not retrieve. File `update.ini' already there, will not retrieve. FINISHED --12:36:50-- Downloaded: 2,748 bytes in 1 files Output using wget version 1.9. The script ist the same. +++ Testing ftp://ftp.nai.com/pub/antivirus/datfiles/4.x/ --12:51:35-- ftp://ftp.nai.com/pub/antivirus/datfiles/4.x/ = `index.html' Resolving wgetproxy... x.x.x.x Proxy request sent, awaiting response... 200 OK Length: unspecified [text/html] [ =] 2,748 12.20K/s 12:51:37 (12.20 KB/s) - `index.html' saved [2748] FINISHED --12:51:37-- Downloaded: 2,748 bytes in 1 files Mit freundlichen Gren / Best regards Thomas Wagner KORDOBA GmbH Co. KG Tel.: +49 69 797 - 6539 Fax: +49 69 797 - 6597 [EMAIL PROTECTED] http://www.kordoba.de
GNU TLS vs. OpenSSL
Does anyone know what the situation is regarding the use of GNU TLS vs. OpenSSL? Last time I checked (several years ago), GNU TLS was still in beta; also, it was incompatible with OpenSSL. Is GNU TLS usable now? Should Wget try to use it in preference to OpenSSL and thus get rid of the GPL exception?
RE: About termination of wget and -T option
This is the end of debug output... === ... 6250K .. .. .. .. .. 12.11 KB/s 6300K .. .. .. .. .. 11.36 KB/s 6350K .. .. .. .. .. 10.96 KB/s 6400K .. .. .. .. ..8.33 KB/s 6450K .. .. .. .. ..2.02 KB/s 6500K .. .. .. .. ..5.90 KB/s 6550K .. .. .. .. ..7.76 KB/s 6600K .. .. .. .. ..7.03 KB/s 6650K .. .. .. .. . 4.56 KB/s Closing fd 5 15:05:13 (6.74 KB/s) - `pages/OM2_report_21705.txt' saved [6852406] === The file received is not complete. Luigi -Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 05, 2003 12:36 PM To: Luigi Stefano Sona (lsona) Cc: [EMAIL PROTECTED] Subject: Re: About termination of wget and -T option Luigi Stefano Sona (lsona) [EMAIL PROTECTED] writes: Hrvoje, This is different from what I observed: - I execute a cgi by wget. The cgi should return a file of about 7MB. - Data start coming, and after 15 minutes, while the file is not complete, wget starts a new trial. I don't doubt your observation, but it might well be the result of a different problem. The debug output might help narrow it down.
Re: About termination of wget and -T option
Luigi Stefano Sona (lsona) [EMAIL PROTECTED] writes: ---request end--- HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Wed, 05 Nov 2003 13:48:56 GMT Server: Apache/1.3.26 (Unix) PHP/4.2.2 mod_perl/1.27 Content-Type: text/plain Via: 1.1 Application and Content Networking System Software 5.0.5 Connection: Close [...] 6650K .. .. .. .. . 4.56 KB/s Closing fd 5 15:05:13 (6.74 KB/s) - `pages/OM2_report_21705.txt' saved [6852406] As far as I can tell: * Either your server is not sending Content-Length for some reason (such as the file being dynamically generated), or a proxy server is stripping it. * The download is interrupted without a clear indication that this has happened. Wget's network read indicates end-of-transfer. * Since Wget doesn't know the length of the file, it cannot know that end-of-transfer is an error and considers the file fully retrieved. A question: are you absolutely certain that the file is not truncated on the server? Can you download it using a web browser?
Re: gettext and charsets
Hrvoje Niksic [EMAIL PROTECTED] writes: Perhaps you could try posting to [EMAIL PROTECTED]? Failing that, you might want to try at the address of the Free Translation Project and/or the Norwegian national team near you. [EMAIL PROTECTED] is the preferred list for reporting gettext bugs. Implementation issues can be discussed at [EMAIL PROTECTED] (also read by Bruno Haible).
RE: About termination of wget and -T option
The file is dinamically generated. Most of times the download goes well, but in specific hours it doesn't complete. I've tried from Netscape directly, and even in this case the download is not completed sometimes. Thanks Luigi -Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 05, 2003 3:40 PM To: Luigi Stefano Sona (lsona) Cc: [EMAIL PROTECTED] Subject: Re: About termination of wget and -T option Luigi Stefano Sona (lsona) [EMAIL PROTECTED] writes: ---request end--- HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Wed, 05 Nov 2003 13:48:56 GMT Server: Apache/1.3.26 (Unix) PHP/4.2.2 mod_perl/1.27 Content-Type: text/plain Via: 1.1 Application and Content Networking System Software 5.0.5 Connection: Close [...] 6650K .. .. .. .. . 4.56 KB/s Closing fd 5 15:05:13 (6.74 KB/s) - `pages/OM2_report_21705.txt' saved [6852406] As far as I can tell: * Either your server is not sending Content-Length for some reason (such as the file being dynamically generated), or a proxy server is stripping it. * The download is interrupted without a clear indication that this has happened. Wget's network read indicates end-of-transfer. * Since Wget doesn't know the length of the file, it cannot know that end-of-transfer is an error and considers the file fully retrieved. A question: are you absolutely certain that the file is not truncated on the server? Can you download it using a web browser?
Hello, new feature added!
Heya people! I am new to the list so hello. I sended a patch to add the --preserve-permissions option to wget this option can be used to preserve permissions when retrieving dirlistings from a remote site. The new standard behavior is not copying permissions this is because users often don't want the permissions a server had set.
RE: About termination of wget and -T option
DEBUG output created by Wget 1.8.2 on solaris2.8. --14:48:31-- http://xxx = `pages/OM2_report_21705.txt' Resolving yyy... done. Caching = NN.MM.NN.MM Connecting to yyy[NN.MM.NN.MM]:80... connected. Created socket 5. Releasing nn (new refcount 1). ---request begin--- GET xxx HTTP/1.0^M User-Agent: Wget/1.8.2^M Host: x^M Accept: */*^M Connection: Keep-Alive^M Authorization: Basic x=^M ^M ---request end--- HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Wed, 05 Nov 2003 13:48:56 GMT Server: Apache/1.3.26 (Unix) PHP/4.2.2 mod_perl/1.27 Content-Type: text/plain Via: 1.1 Application and Content Networking System Software 5.0.5 Connection: Close Length: unspecified [text/plain] 0K .. .. .. .. ..3.69 KB/s 50K .. .. .. .. .. 39.68 KB/s 100K .. .. .. .. .. 21.19 KB/s 150K .. .. .. .. .. 26.58 KB/s 200K .. .. .. .. .. 31.67 KB/s 250K .. .. .. .. .. 33.78 KB/s .. 6250K .. .. .. .. .. 12.11 KB/s 6300K .. .. .. .. .. 11.36 KB/s 6350K .. .. .. .. .. 10.96 KB/s 6400K .. .. .. .. ..8.33 KB/s 6450K .. .. .. .. ..2.02 KB/s 6500K .. .. .. .. ..5.90 KB/s 6550K .. .. .. .. ..7.76 KB/s 6600K .. .. .. .. ..7.03 KB/s 6650K .. .. .. .. . 4.56 KB/s Closing fd 5 15:05:13 (6.74 KB/s) - `pages/OM2_report_21705.txt' saved [6852406] -Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 05, 2003 3:17 PM To: Luigi Stefano Sona (lsona) Cc: [EMAIL PROTECTED] Subject: Re: About termination of wget and -T option Luigi Stefano Sona (lsona) [EMAIL PROTECTED] writes: This is the end of debug output... Why not the whole output? Some crucial portions are missing. If you have confidential information in the output, feel free to delete it first. From seeing this portion, it seems that there is no timeout, and that Wget believes that the whole file has been retrieved. Additionally, the server does not appear to be sending the `Content-Type' header, so Wget cannot tell that the file is incomplete.
Re: About termination of wget and -T option
Luigi Stefano Sona (lsona) [EMAIL PROTECTED] writes: The file is dinamically generated. Most of times the download goes well, but in specific hours it doesn't complete. I've tried from Netscape directly, and even in this case the download is not completed sometimes. This could be a bug in the dynamic page that generates the file, or possibly a problem with the proxy server or with your network configuration. Wget is designed to deal with this issue, but if the sevrer doesn't provide Content-Length, its hands are tied. If you can convince the author of the dynamic page to send Content-Length, Wget will detect the problem and retry. Otherwise, there is simply no indication that anything is wrong, and the browser cannot help you.
Re: The libtool situation
On Wed, 5 Nov 2003, Hrvoje Niksic wrote: Is it me, or is there something really strange going on with libtool? I tried updating Wget's libtool, but libtool 1.5 is *still* unavailable from the GNU FTP site and its mirrors. It points to a BACK-RSN type of file that explains how they've been compromised, and so on. It must be something that has happened recently, as the 1.5 version used to be around on all the GNU mirrors. In case it helps, I happened to find my own downloaded package lying around and I can offer it for download from here: http://www.contactor.se/~dast/libtool-1.5.tar.gz -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
Windows msvc 1.9-stable bugfix release
Binary for current 1.9 cvs branch at http://xoomer.virgilio.it/hherold/ Heiko -- -- PREVINET S.p.A. www.previnet.it -- Heiko Herold [EMAIL PROTECTED] -- +39-041-5907073 ph -- +39-041-5907472 fax
RE: GNU TLS vs. OpenSSL
From Various Licenses and Comments about Them http://www.gnu.ctssn.com/licenses/license-list.html The OpenSSL license. The license of OpenSSL is a conjunction of two licenses, One of them being the license of SSLeay. You must follow both. The combination results in a copyleft free software license that is incompatible with the GNU GPL. It also has an advertising clause like the original BSD license and the Apache license. We recommend using GNUTLS instead of OpenSSL in software you write. However, there is no reason not to use OpenSSL and applications that work with OpenSSL. shouldn't the above say say non-copyleft? /a On Wed, 5 Nov 2003, Post, Mark K wrote: I'm a little confused. OpenSSL is licensed pretty much the same as Apache. What's the GPL issue with that style of license? Mark Post -Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 05, 2003 8:27 AM To: [EMAIL PROTECTED] Subject: GNU TLS vs. OpenSSL Does anyone know what the situation is regarding the use of GNU TLS vs. OpenSSL? Last time I checked (several years ago), GNU TLS was still in beta; also, it was incompatible with OpenSSL. Is GNU TLS usable now? Should Wget try to use it in preference to OpenSSL and thus get rid of the GPL exception?
Re: The libtool situation
Daniel Stenberg [EMAIL PROTECTED] writes: On Wed, 5 Nov 2003, Hrvoje Niksic wrote: Is it me, or is there something really strange going on with libtool? I tried updating Wget's libtool, but libtool 1.5 is *still* unavailable from the GNU FTP site and its mirrors. It points to a BACK-RSN type of file that explains how they've been compromised, and so on. It must be something that has happened recently, as the 1.5 version used to be around on all the GNU mirrors. I'm not sure this is that recent, unless there was a new compromise I never heard about. As for the mirrors, well, they're mirrors -- they just picked up the new file, which happens to be a symlink. (Which is just as well because they would have happily picked up the compromised version too.) In case it helps, I happened to find my own downloaded package lying around and I can offer it for download from here: [...] Thanks, but I got it off Debian. Although I suppose I could pick it up and compare it with what I got just in case.
WGET Unable to Get Through Proxy Server with HTTPS
I'm unable to get through the proxy server when downloading via WGET HTTPS. The following is the message. Can you help? [] C:\WGET\wget-1.9\srcwget.exe -dS --proxy-user=censored --proxy-p asswd=censored https:// :@momentumw1.boh.com/WebDownloads/bankahttps://censored id:censored pwd@momentumw1.boh.com/WebDownloads/banka _an/andrept.txt030919105214 wget: debug support not compiled in. --13:21:50-- https://banka_an:[EMAIL PROTECTED]/WebDownloads/banka_ an/andrept.txt030919105214 = `andrept.txt030919105214.1' Resolving censored proxy.bankofamerica.com... censored ip Connecting to censored proxy[censored ip]:8080... connected. Proxy request sent, awaiting response... 1 HTTP/1.0 502 internal error - server connection terminated 2 Date: Wed, 05 Nov 2003 21:21:50 GMT 3 Via: HTTPS/1.1 nup-ctc (Traffic-Server/4.0.9 [c sEf ]) 4 Cache-Control: no-store 5 Content-Type: text/html 6 Content-Language: en 7 Content-Length: 368 13:21:50 ERROR 502: internal error - server connection terminated. Thank you.
Re: GNU TLS vs. OpenSSL
Post, Mark K [EMAIL PROTECTED] writes: I'm a little confused. OpenSSL is licensed pretty much the same as Apache. What's the GPL issue with that style of license? I don't speak for the FSF, but from reading the license, it imposes additional constraints, specifically the advertising clause. That makes it incompatible with the GPL, despite of it clearly being free software. There might be other incompatibilities I'm not aware of. The exception, phrased by Eben Moglen and approved by RMS, allows distributors such as Debian to legally distribute SSL-enabled Wget binaries.
