yes, the property is fetcher.timelimit.mins. if you not set this property, the QueueFeeder will not filter the url and log output may like this
QueueFeeder finished: total 36651 records + hit by time limit :0 Do you use bin/crawl command script. it will set the time limit for fetching to 180. the code like this # time limit for fetching timeLimitFetch=180 # fetching the segment echo "Fetching : $SEGMENT" $bin/nutch fetch $commonOptions -D fetcher.timelimit.mins=$timeLimitFetch $CRAWL_PATH/segments/$SEGMENT -noParsing -threads $numThreads On Sun, Mar 17, 2013 at 7:47 PM, Amit Sela <[email protected]> wrote: > By fetcher.limit property,do you mean fetcher.timelimit.mins ? because I > have it set on default (-1) - no time limit. > > > On Sat, Mar 16, 2013 at 5:12 PM, feng lu <[email protected]> wrote: > > > Hi Amit > > > > << > > I also note > > that the total hit by time limit here is 50927 but the job counters show > > 7493. > > >> > > > > This two time limits are all set bye fetcher.limit property. One is used > in > > QueueFeeder class, indicate that the QueueFeeder should finish load data > if > > current time is larger than time limit. So the total hit by time limit is > > 50927. Another is used in FetchItemQueues class, indicate that check the > > time if current time is larger than time limit and feeder has stopped , > > emptying the queues, So here job counters of time limit is 7493. There > are > > not equal. > > > > > > << > > Summing all of theses numbers does equal the total map input. > > >> > > > > do you set the property of "fetcher.follow.outlinks.depth", when > > fetcher.parse is true and this value is greater than 0 the fetcher will > > extract outlinks > > and follow until the desired depth is reached. > > > > Another reason is that when this page is redirect to another page , fetch > > will add new redirect page to fetch queues, so you can see that map input > > is not equal to numbers of all sum. > > > > > > > > > > > > > > On Sat, Mar 16, 2013 at 8:03 AM, Lewis John Mcgibbney < > > [email protected]> wrote: > > > > > Hi Amit, > > > > > > I know this thread is a bit old now, however it is also something which > > > bugged me when I was looking into something else (InjectorJob > counters). > > > > > > On Tue, Mar 5, 2013 at 3:16 AM, Amit Sela <[email protected]> wrote: > > > > > > > > > > > And summing all counters does not equal the total map input... > > > > > > > > Summing all of theses numbers does equal the total map input. I also > > note > > > > that the total hit by time limit here is 50927 but the job counters > > show > > > > 7493. > > > > > > > > > > > Basically, the easiest way to see and generally understand counters is > to > > > run the Nutch application within your Hadoop cluster (if no cluster > > > available then use psudo mode) and use the web application interface to > > > Hadoop. You will clearly see all counters associated with the job and > you > > > can take it from there. > > > I like the notion of creating custom counters to obtain specific > metrics > > > but this is solely driven by user requirements. > > > Do you want to learn more about counters? Look into the code. > > > Do you want to know more about Nutch counters, or make the counters > more > > > explicit? Then consider opening a Jira issue and we can discuss this in > > > more detail. > > > With regards to the Fetcher, there are many possible areas where > counters > > > are (and could be) really useful... as I said though this s only driven > > by > > > user requirements. > > > > > > > > > > > -- > > Don't Grow Old, Grow Up... :-) > > > -- Don't Grow Old, Grow Up... :-)
