I'd find it useful to guide wget by using regular expressions to
control which links get followed. For example, to avoid
following links based on embedded css styles or link text.
I've needed this several times, but the most recent was when I
wanted to avoid following any add to cart or buy
Hi,
I have difficulties in getting the files from another server
with 403 ERROR. I am trying to run the following command and I get the errors
shown below:
wget -x -N -p -i upload.txt -P dir --http-user=xxx
--http-password=xxx -o upload.log
--17:03:03--
It would be great if wget had a way of limiting the amount of time it
took to run so it won't accidentally hammer on someone's web server for
an indefinate amount of time. I'm often needing to let a crawler run
for a while on an unknown site, and I have to manually kill wget after a
few hours
