Meghna Kukreja wrote: > Hi, > > I checked out the latest Nutch trunk to get the fix for Issues 273 > (http://issues.apache.org/jira/browse/NUTCH-273) and 322 > (http://issues.apache.org/jira/browse/NUTCH-322) and it looks like if > http.redirect.max is 0, then the redirected url is inserted with the > same status as the original url of CrawlDatum.STATUS_FETCH_REDIR_PERM > (Fetcher.java line 201) which is converted to > CrawlDatum.STATUS_DB_REDIR_PERM. This prevents the redirected url from > being selected by the generator in the next round (Generator.java > lines 140-142).
Indeed, this looks wrong - the target of the redirect should be inserted with another fetch status (we could re-use status code LINKED, which is then turned into DB_UNFETCHED if a page didn't exist yet). > My initial seed url was a RSS feed where all the items were being > redirected. I tried to do a crawl with depth 3 but at the last depth > the Generator was not able to select any urls, printed the message "0 > records selected for fetching, exiting ..." and the crawl ended with a > NullPointerException as the fetcher was trying to fetch an empty > segment. Yup, another place to fix so that it doesn't throw an NPE. > I was wondering if any one else had seen this behaviour and has been > able to find a way to fix it. Thanks for testing this, I'll apply these fixes soon. -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-developers mailing list Nutch-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nutch-developers
