[EMAIL PROTECTED] wrote:
Did anyone else hear that "click" sound, or was it just something in my head?
:) Yeah, I don't think this is properly documented ...
I was missing the info about the 7 day lockout. Now that makes sense. It makes even more sense when I look at machines running jobs with N tasks total, out of which N-1 are completed, and the remaining one is draaaaging. In a system where jobs are submitted sequentially, this means underutilized nodes. So does the following type of scheduling for Nutch jobs make sense: 0) imagine a cluster with M max maps and R max reduces (say M=R=8)
Just a note: this is a very untypical cluster. Usually Hadoop clusters are configured to run M >> numNodes and R ~= numNodes - see http://wiki.apache.org/hadoop/HowManyMapsAndReduces .
1) run generate job with -numFetchers equal to M-2 2) run a fetcher job (uses M-2 maps and later all R reduces) 3) at this point there are 2 open map slots for something else to run, say the updatedb job for the previously fetched/parsed segment 4) when updatedb job is done the cluster can take on more jobs. Any completed tasks (C) from the running fetcher job represent "open work slots" 5) start another fetch job. This will be able to use only C tasks, but C will grow as the first job opens up more slots, eventually hitting M-2 open slots. 6) at some point, the fetch job from 2) above will complete, opening up 2 map slots, so updatedb can be run, even in the background, allowing the execution to go back to 1) Is this all correct? If it is, or when it is, I'll stick it on the Wiki. Without overlapping jobs and getting the procedure right, people running Nuthc must not be utilizing their clusters fully.
Yes, this looks correct.
Did I get the numbers (M-2) right?
Yes, if that's the number you wish to use. :) I mean, this is an arbitrary decision of the operator using the cluster - he may prefer to use e.g. M/2 for the fetcher, and the rest for other short-lived maintenance jobs.
-- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com
