[
https://issues.apache.org/jira/browse/NUTCH-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205724#comment-15205724
]
ASF GitHub Bot commented on NUTCH-2222:
---------------------------------------
GitHub user lewismc opened a pull request:
https://github.com/apache/nutch/pull/99
NUTCH-2222 re-fetch deletes all metadata except _csh_ and _rs_
Hi Folks,
This pull request formalizes the contribution from Adnane B. over on
[NUTCH-2222](https://issues.apache.org/jira/browse/NUTCH-2222). I am not sure
that this is catching the bug however and it seems like the bug only rears it's
head when we use a persistent datastore.
Right now this patch uses the MemStore implementation which is what we
agreed tests would be like. I think we need to do add to the tests in order to
have a persistent datastore setup and that data is persisted into the
datastore.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lewismc/nutch NUTCH-2222
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nutch/pull/99.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #99
----
commit fd478448bb4a68ad53520ae7b325204a7834782a
Author: Lewis John McGibbney <[email protected]>
Date: 2016-03-22T03:46:18Z
NUTCH-2222 re-fetch deletes all metadata except _csh_ and _rs_
----
> re-fetch deletes all metadata except _csh_ and _rs_
> ----------------------------------------------------
>
> Key: NUTCH-2222
> URL: https://issues.apache.org/jira/browse/NUTCH-2222
> Project: Nutch
> Issue Type: Bug
> Components: crawldb
> Affects Versions: 2.3.1
> Environment: Centos 6, mongodb 2.6 and mongodb 3.0 and
> hbase-0.98.8-hadoop2
> Reporter: Adnane B.
> Assignee: Lewis John McGibbney
> Fix For: 2.3.2
>
> Attachments: TestReFetch.java, index.html
>
>
> This problem happens at the the second time I crawl a page
> {code}
> bin/nutch inject urls/
> bin/nutch generate -topN 1000
> bin/nutch fetch -all
> bin/nutch parse -force -all
> bin/nutch updatedb -all
> {code}
> seconde time (re-fetch) :
> {code}
> bin/nutch generate -topN 1000 --> batchid changes for all existing pages
> bin/nutch fetch -all --> *** metadatas are delete for all pages already
> crawled **
> bin/nutch parse -force -all
> bin/nutch updatedb -all
> {code}
> I reproduce it with mongodb 2.6, mongodb 3.0, and hbase-0.98.8-hadoop2
> It happens only if the page has not changed
> To reproduce easily, please add to nutch-site.xml :
> {code}
> <property>
> <name>db.fetch.interval.default</name>
> <value>60</value>
> <description>The default number of seconds between re-fetches of a page (1
> minute)
> </description>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
