Folks, I'm trying to index a site that is dynamically creating the same web page. Thus the web server will have multiple URL links to the same page, but each time uses a different auto-generated URL link to reference the page. Is there a way to avoid or catch that this is happening? One obvious solution is to limit the depth to 1 or 2, but some pages will be excluded, which is no good. The some of the sites typically generates from /cgi-bin/, but I don't want to necessarily exclude these links either and it's not always the case. Are there any combination of wget options that could help?? Thanks
-- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm
