On Tue, May 15, 2001 at 06:21:06PM -0700, J Scott Jaderholm wrote:
> Hi,
>
> I'd like to recursively get a directory being served by an apache
> server. The problem is that about 1 out of 4 requests for a file or
> directory are answered with a "file not found" (404) error.
>
> Is there a way to have wget reget the files until it succeeds? I'd
> just fix the problem on the server but 1) I don't run it 2) no one
> that I've talked to (which includes a lot of apache users) has seen
> this problem before.
>
> Any suggestions for using wget as a solution to this problem would be
> _greatly_ appreciated.
>
> Sincerely,
> jsj
> --
> the sky is tired of being blue
Something like the attached?
--
Always hardwire the explosives
-- Fiona Dexter quoting Monkey, J. Gregory Keyes, Dark Genesis
#!/bin/bash
END=2;
START=1;
export HTTP_PROXY=127.0.0.1:3128
rm -f ~/1.txt.mwgeta
while ((END > START)); do
cat ~/1.txt.mwgeta 2>/dev/null
echo trying...
START=`date +%s`
echo Starting WGET
echo wget $@
if wget $@ > ~/1.txt.mwgeta 2>&1; then echo WGET done successfully;
else echo WGET lost something; END=`date +%s`;
if grep "unsupported protocol" ~/1.txt.mwgeta; then echo UNSUPPORTED PROTOCOL; END=1;
fi
continue; fi
echo WGET ended
END=`cat /var/log/connected.log`
echo START=$START
echo END=$END
done
rm ~/1.txt.mwgeta
