[
https://issues.apache.org/jira/browse/NUTCH-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970764#comment-16970764
]
ASF GitHub Bot commented on NUTCH-2750:
---------------------------------------
sebastian-nagel commented on issue #483: Fix for NUTCH-2750
URL: https://github.com/apache/nutch/pull/483#issuecomment-552085300
Hi @sju,
I've verified that the readers are refreshed if the CrawlDb has changed -
tested in local mode on a local file and in distributed mode with a CrawlDb on
HDFS.
Will commit soon. Thanks!
Actually, I'm really impressed how fast look-ups are once the readers are
initialized: I've got 20 look-ups per second on a CrawlDb containing 1 billion
records in 200 partitions hold on HDFS. That's pretty fast given that it's not
a "hot" index but only files on HDFS. Opened
[NUTCH-2753](https://issues.apache.org/jira/browse/NUTCH-2753) to add this
great option to the command-line help of readdb/readlinkdb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> improve CrawlDbReader & LinkDbReader reader handling
> ----------------------------------------------------
>
> Key: NUTCH-2750
> URL: https://issues.apache.org/jira/browse/NUTCH-2750
> Project: Nutch
> Issue Type: Improvement
> Components: crawldb, linkdb
> Affects Versions: 1.16
> Reporter: Jurian Broertjes
> Priority: Minor
>
> The current implementation in the CrawlDbReader re-opens readers for every
> URL. This is not very efficient. I've implemented a modification time check
> that only re-opens readers on updated crawlDB.
> PR: https://github.com/apache/nutch/pull/483
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
