problem use accept/reject rules
I use that rule in .wgetrc: accept = *[?]* and reject = *\.[zZ][iI][pP]* I think that rule exclude all *.zip* from download, but in test url like http://domain.com/price/source/10-20030915.zip?PHPSESSID=0cd4eb0801c656a292e 33c9b8134c899 downloaded. Whats wrong wget or my regexp?
no-clobber add more suffix
`--no-clobber' is very usfull option, but i retrive document not only with .html/.htm suffix. Make addition option that like -A/-R define all allowed/rejected rules for -nc option.
problem with 302 server respose parsing
I use Wget 1.8.2. When I try receive page with '-nc' option and server return 302 and new url, wget not test that url on rules in '-nc' and download and rewrite existing file. I think wget not used command line option rules when parse server response header! It is a bug?
Wget 1.8.2 bug
I use wget 1.8.2. When I try recursive download site site.com where site.com/ first page redirect to site.com/xxx.html that have first link in the page to site.com/ then Wget download only xxx.html and stop. Other links from xxx.html not followed!
Question about url convert
Have wget any rules to convert retrive url to store url? Or may be in future? For example: Get - site.com/index.php?PHPSESSID=123124324 Filter - /PHPSESSID=[a-z0-9]+//i Save as - site.com/index.php
Problem recursive download
I use wget 1.8.2 Try recursive downdload www.map-by.info/index.html, but wget stop in first page. Why? index.html have links to another page. /usr/local/bin/wget -np -r -N -nH --referer=http://map-by.info -P /tmp/www.map-by.info -D map-by.info http://map-by.info http://www.map-by.info --10:09:25-- http://map-by.info/ = `/p4/poisk/spider/resource/www.map-by.info/index.html' Resolving proxy.open.by... done. Connecting to proxy.open.by[193.232.92.3]:8080... connected. Proxy request sent, awaiting response... 200 OK Length: ignored [text/html] Server file no newer than local file `/p4/poisk/spider/resource/www.map-by.info/index.html' -- not retrieving. --10:09:25-- http://www.map-by.info/ = `/p4/poisk/spider/resource/www.map-by.info/index.html' Connecting to proxy.open.by[193.232.92.3]:8080... connected. Proxy request sent, awaiting response... 200 OK Length: ignored [text/html] Server file no newer than local file `/p4/poisk/spider/resource/www.map-by.info/index.html' -- not retrieving. FINISHED --10:09:26-- Downloaded: 0 bytes in 0 files
RE: Problem recursive download
I think wget strong verify link syntax:
a href=about_rus.html onMouseOver=img_on('main21');
onMouseOut=img_off('main21')
That link have incorrect symbol ';' not quoted in a
-Original Message-
From: Sergey Vasilevsky [mailto:[EMAIL PROTECTED]
Sent: Thursday, October 16, 2003 10:15 AM
To: [EMAIL PROTECTED]
Subject: Problem recursive download
I use wget 1.8.2
Try recursive downdload www.map-by.info/index.html, but wget stop in first
page.
Why?
index.html have links to another page.
/usr/local/bin/wget -np -r -N -nH --referer=http://map-by.info -P
/tmp/www.map-by.info -D map-by.info http://map-by.info
http://www.map-by.info
--10:09:25-- http://map-by.info/
= `/p4/poisk/spider/resource/www.map-by.info/index.html'
Resolving proxy.open.by... done.
Connecting to proxy.open.by[193.232.92.3]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: ignored [text/html]
Server file no newer than local file
`/p4/poisk/spider/resource/www.map-by.info/index.html' -- not retrieving.
--10:09:25-- http://www.map-by.info/
= `/p4/poisk/spider/resource/www.map-by.info/index.html'
Connecting to proxy.open.by[193.232.92.3]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: ignored [text/html]
Server file no newer than local file
`/p4/poisk/spider/resource/www.map-by.info/index.html' -- not retrieving.
FINISHED --10:09:26--
Downloaded: 0 bytes in 0 files
RE: Wget 1.8.2 bug
Thanks for explain this reasons. And I have anoter problem: in .wgetrc I use reject = *.[zZ][iI][pP]*,*.[rR][aA][rR]*,*.[gG][iI][fF]*,*.[jJ][pP][gG]*,*.[Ee][xX][E e]*,*[=]http* accept = *.yp*,*.pl*,*.dll*,*.nsf*,*.[hH][tT][mM]*,*.[pPsSjJ][hH][tT][mM]*,*.[pP][hH] [pP]*,*.[jJ][sS][pP]*,*.[tT][xX][tT],*.[cC][gG][iI]*,*.[cC][sS][pP]*,*.[aA][ sS][pP]*,*[?]* In command line add some more rules '-R xxx' - I think it joined with previos rules. And use recursive download. In result I found *.zip and *.exe ... files! What I do wrong? -Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Friday, October 17, 2003 7:18 PM To: Tony Lewis Cc: Wget List Subject: Re: Wget 1.8.2 bug Tony Lewis [EMAIL PROTECTED] writes: Hrvoje Niksic wrote: Incidentally, Wget is not the only browser that has a problem with that. For me, Mozilla is simply showing the source of http://www.minskshop.by/cgi-bin/shop.cgi?id=1cookie=set, because the returned content-type is text/plain. On the other hand, Internet Explorer will treat lots of content types as HTML if the content starts with html. I know. But so far noone has asked for this in Wget. Perhaps we can add an option to wget so that it will look for an html tag in plain text files? If more people clamor for the option, I suppose we could overload `--force-html' to perform such detection.
follow_ftp not work
Wget 1.9.1 .wgetrc: reject = *.[Ee][xX][Ee]* follow_ftp = off Command line: wget -np -nv -r -N -nH --referer=http://www.orion.by -P /tmp/www.orion.by -D orion.by http://www.orion.by Output: Last-modified header missing -- time-stamps turned off. 13:15:08 URL:http://www.orion.by/index.php?mode=main [24703] - /tmp/www.orion.by/index.php?mode=main [1] http://www.orion.by/robots.txt: 13:15:09 ERROR 404: Not Found. 20 redirections exceeded. 20 redirections exceeded. 13:15:18 URL: ftp://62.118.248.95/cyberfight/q3/utils/Seismovision222light.exe [882] - /tmp/www.orion.by/cyberfight/q3/utils/.listing [1] ^C Question: 1. How I can see what parameters wget use at run time? You may add some option for print it. 2. Reject rules require more help with examples!
