[ https://issues.apache.org/jira/browse/NUTCH-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660382#action_12660382 ]
Todd Lipcon commented on NUTCH-669: ----------------------------------- Here's a further report on my progress: - It turns out the change in NUTCH-676 caused things to break - there's some behavior in nutch's MapWritable that differs from Hadoop's, so it was spending all of its time in output.collect - I think the writables were accruing lots of key/value pairs that they weren't sposed to. So, this doesn't depend on NUTCH-676. - I implemented adaptive crawl delay (NUTCH-475) in the new fetcher. - Also implemented early termination as discussed in this mailing list thread: http://www.nabble.com/proposal:-fetcher-performance-improvements-td20939872.html Results so far are looking good. I was able to run a 1M url fetch with 5000 urls per host at a sustained rate of 25 pages/second (total around 11 hours). About 60% of the URLs ended up parsed, which isn't significantly worse than I usually see without early termination, but past attempts to run 1M fetches have taken several days because of some slow hosts. I'm running a 2M+ URL fetch right now and have been sustaining 40-60mbit inbound from 8 fetchers for the last couple hours. - I did experience one GC error - I think I need to add some cleanup of empty queues out of the FetchQueue structure when the number of unique hosts is very high. Complete history is here: http://github.com/toddlipcon/nutch/tree/nutch-669 > Consolidate code for Fetcher and Fetcher2 > ----------------------------------------- > > Key: NUTCH-669 > URL: https://issues.apache.org/jira/browse/NUTCH-669 > Project: Nutch > Issue Type: Improvement > Components: fetcher > Affects Versions: 0.9.0 > Reporter: Todd Lipcon > Fix For: 1.0.0 > > > I'd like to consolidate a lot of the common code between Fetcher and > Fetcher2.java. > It seems to me like there are the following differences: > - Fetcher relies on the Protocol to obey robots.txt and crawl delay > settings whereas Fetcher2 implements them itself > - Fetcher2 uses a different queueing model (queue per crawl host) to > accomplish the per-host limiting without making the Protocol do it. > I've begun work on this but want to check with people on the following: > - What reason is there for Fetcher existing at all since Fetcher2 seems to be > a superset of functionality? > - Is it on the road map to remove the robots/delay logic from the Http > protocol and make Fetcher2's delegation of duties the standard? > - Any other improvements wanted for Fetcher while I am in and around the code? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.