On Mon, Jun 21, 2010 at 17:08, Andrew Farnsworth <[email protected]> wrote: > Hey everyone, > I am having problems using wget to crawl a site (by request of the owners) > that is behind a login. I have tried using wget and and have it save and > use cookies and it does appear to save them. However, I cannot seem to > crawl the site while logged in. > > Any suggestions from experience? > > Thanks, > > Andy Farnsworth > > -- > You received this message because you are subscribed to the Google Groups > "NLUG" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nlug-talk?hl=en >
wget -mc is for mirror and continue if stopped.. Is there a robots.txt file in Apache I'm assuming stopping your robot. -- -- Remember it's not that we have something to hide, we just have nothing we want to show. ---Keep tunnelling. -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en
