Jukka, >On 8/5/06, Chris Schneider <[EMAIL PROTECTED]> wrote: >>Given this, shouldn't the default URL normalizer just add a slash to >>the end of a URL that doesn't have a file extension?
At 8:41 AM +0300 8/5/06, Jukka Zitting wrote: >Section 6.2.4 of RFC 3986 suggests that a crawler could do such a >normalization if it detects that >http://mail.python.org/mailman/listinfo redirects to >http://mail.python.org/mailman/listinfo/. I think just blindly adding >the slash without knowing about the redirection is incorrect. I wasn't thinking about redirection. You are correct; dedup is the way to handle this problem. - Chris -- ------------------------ Chris Schneider TransPac Software, Inc. [EMAIL PROTECTED] ------------------------ ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
