Guys, This sounds awesome. Even I could understand it, which is saying something! :)
My only question: why introduce a new data structure called “Markers” when all that seems to be is a Metadata object. Let’s use o.a.tika.metadata.Metadata to represent that? My only comment then would be, aren’t we still doing something you mentioned you wanted to get rid of below, where you said: “For example, during parsing we don't have access to a URL's fetch status. So we copy fetch status into content metadata.” Aren’t we just doing the same thing with Markers? Cheers, Chris On 7/3/10 3:01 AM, "Doğacan Güney" <[email protected]> wrote: Hello everyone, I am attaching first draft of a complete nutchbase design document. There are parts missing and parts not yet explained clearly but I would like to get everyone's opinion on what they think so far. Please let me know which parts are unclear, which parts make no sense etc, and I will improve the draft. ----------------------------------------------------------------------------------------------------------------------------------------- Nutchbase ========= 1) Rationale * All your data in a central location (at least, nutch gives you the illusion of a centralized storage) * No more segment/crawldb/linkdb merges. * No more "missing" data in a job. There are a lot of places where we copy data from one structure to another just so that it is available in a later job. For example, during parsing we don't have access to a URL's fetch status. So we copy fetch status into content metadata. This will no longer be necessary after nutchbase. When writing a job or a new plugin, programmer only needs to specify which fields she wants to read and they will be available to plugin / job. * A much simpler data model. If you want to update a small part in a single record, now you have to write a MR job that reads the relevant directory, change the single record, remove old directory and rename new directory. With nutchbase, you can just update that record. 2) Design As mentioned above, all data for a URL is stored in a WebPage object. This object is accessed by a key that is the reverse form of a URL. For example, http://bar.foo.com:8983/to/index.html?a=b becomes com.foo.bar:http:8983/to/index.html?a=b If URLs are stored lexicographically, this means that URLs from same domain and host are stored closer together. This will hopefully make developing statistics tools easier for hosts and domains. Writing a MapReduce job that uses Gora for storage does not take much effort. There is a new class called StorageUtils that has a number of static methods to make setting mappers/reducers/etc easier. Here is an example (from GeneratorJob.java): Job job = new NutchJob(getConf(), "generate: " + crawlId); StorageUtils.initMapperJob(job, FIELDS, SelectorEntry.class, WebPage.class, GeneratorMapper.class, URLPartitioner.class); StorageUtils.initReducerJob(job, GeneratorReducer.class); An important argument is the second argument to #initMapperJob. This specifies all the fields that this job will be reading. If plugins will run during a job (for example, during ParserJob, several plugins will be active), then before job is run, those plugins must be initialized and FieldPluggable#getFields must be called for all plugins to figure out which fields they want to read. During map or reduce phase, modifying WebPage object is as simple as using the built-in setters. All changes will be persisted. Even though some of these objects are still there, most of the CrawlDatum, Content, ParseData or similar objects are removed (or are slated to be removed). In most cases, plugins will simply take (String key, WebPage page) as arguments and modify WebPage object in-place. 3) Jobs Nutchbase uses the concept of a marker to identify what has been processed and what will be processed from now on. WebPage object contains a map<String, String> called markers. For example, when GeneratorJob generates a URL, it puts a unique string and a unique crawl id to this marker map. Then FetcherJob only fetches a given URL if it contains this marker. At the end of the crawl cycle, DbUpdaterJob clears all markers (except markers placed by IndexerJob). a) InjectorJob: This phase consists of two different jobs. First job reads from a text file (as before) then puts a special inject marker. Second job goes over all URLs then if it finds a WebPage with inject marker but nothing else (so, it is a new URL), then this URL is injected (and marker is deleted). Otherwise, marker is just deleted (since URL is already injected). b) GeneratorJob: GeneratorJob is similar to what it was before, but it is now a single job. During map phase: if FetchSchedule indicates given URL is to be fetched then, Calculate generator score using scoring filters Output <SelectorEntry<URL, score>, WebPage> SelectorEntry is sorted according to given score so highest scoring entries will be processed first during reduce. Then URLPartitioner partitions according to choices specified in config files (by host, by ip or by domain). Reduce phase counts URL according to topN and host/domain limits and marks all URLs if limits are not yet reached. GeneratorJob markes all generated URLs with a unique crawl id. Fetcher, parser and indexer jobs can then use this crawl id to process only URLs that are in that particular crawl cycle. Alternatively, they can also work on all URLs (though, again, FetcherJob will only work on a URL which has been marked by GenerateJob. So even if FetcherJob is instructed to work on "all" URLs, it will skip those without generator markers). GeneratorJob will also print crawl id to console. c) FetcherJob: During map phase, FetcherJob outputs <Integer, FetcherEntry<key, WebPage>> pairs. The first member of the pair (integer) is a random integer between 0 and 65536. After map, these pairs are partitioned into reduces according to URL's hosts. Reduce works exactly like old Fetcher's map. FetcherJob can also continue interrupted fetches now (by giving "-continue" switch on command line). The random integer may seem pointless but it is actually quite important for performance. Let's say instead of a random integer we were to just output <key, WebPage> pairs from map. Again, let's say you have 1000 URLs from host a.com <http://a.com> , and 500 URLs from b.com <http://b.com> . Let's also say that we have 10 fetcher threads so maximum queue size is 500. In this case, since all URLs from host a.com <http://a.com> will be processed by reduce before all URLs from b.com <http://b.com> , during reduce phase, only one thread will fetch URLs from a.com <http://a.com> while every other thread will be spin-waiting. However, with randomization, URLs from a.com <http://a.com> and b.com <http://b.com> will be processed in a random order thus bandwidth utilization will be higher. d) ParserJob: ParserJob is straightforward. It is only a map (i.e., has 0 reducers). Simple parses all URLs with active parse plugins. e) DbUpdaterJob: This is a combination of updatedb and invertlinks jobs. If a URL is successfully parsed (which means, it will contain a parse marker), DbUpdaterJob will put its own marker. Note: It may make more sense to put a marker even if a URL is not successfully parsed. DbUpdaterJob also cleans all other markers. f) IndexerJob: Goes over all URLs with a db update marker (again, you can specify ALL URLs with update markers, or a crawl id), and indexes them. 4) What's missing Most of the core functionality and plugins have been ported. However, some tools and features are still missing: arc segment tools, PageRank scoring, field indexing API, etc.... ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
