(Better late than never... I forgot I didn't yet respond to your posting).
Doug Cutting wrote:
I think all that you're saying is that we should not run two CrawlDB
updates at once, right? But there are lots of reasons we cannot do
that besides the OPIC calculation.
When we used WebDB it was possible to overlap generate / fetch / update
cycles, because we would "lock" pages selected by FetchListTool for a
period of time.
Now we don't do this. The advantage is that we don't have to rewrite
CrawlDB. But operations on CrawlDB are considerably faster than on
WebDB, perhaps we should consider going back to this method?
Also, the "cash value" of those outlinks that point to URLs not in
the current fetchlist will be dropped, because they won't be
collected anywhere.
No, every cash value is used. The input to a crawl db update includes a
CrawlDatum for every known url, including those just linked to. If
the only CrawlDatum for a url is a new outlink from a page crawled,
then the score for the page is 1.0 + the score of that outlink.
Of course, you are right, I missed this.
And a final note: CrawlDB.update() uses the initial score value
recorded in the segment, and NOT the value that is actually found in
CrawlDB at the time of the update. This means that if there was
another update in the meantime, your new score in CrawlDB will be
overwritten with the score based on an older initial value. This is
counter-intuitive - I think CrawlDB.update() should always use the
latest score value found in the current CrawlDB. I.e. in
CrawlDBReducer instead of doing:
result.setScore(result.getScore() + scoreIncrement);
we should do:
result.setScore(old.getScore() + scoreIncrement);
The change is not quite that simple, since 'old' is sometimes null.
So perhaps we need to add an 'score' variable that is set to old.score
when old!=null and to 1.0 otherwise (for newly linked pages).
The reason I didn't do it that way was to permit the Fetcher to modify
scores, since I was thinking of the Fetcher as the actor whose actions
are being processed here, and of the CrawlDb as the passive thing
acted on. But indeed, if you have another process that's updating a
CrawlDb while a Fetcher is running, this may not be the case. So if
we want to switch things so that the Fetcher is not permitted to
adjust scores, then this seems like a reasonable change.
I would vote for implementing this change. The reason is that the active
actor that computes new scores is CrawlDb.update(). Fetcher may provide
additional information to affect the score, but IMHO the logic to
calculate new scores should be concentrated in the update() method.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
