[
https://issues.apache.org/jira/browse/NUTCH-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18088729#comment-18088729
]
Hudson commented on NUTCH-1732:
-------------------------------
SUCCESS: Integrated in Jenkins build Nutch ยป Nutch-trunk #242 (See
[https://ci-builds.apache.org/job/Nutch/job/Nutch-trunk/242/])
NUTCH-1732: allow deleting non-parsable documents (#891) (github:
[https://github.com/apache/nutch/commit/87483c650091e3e156114b49d8ec8a3f0753da93])
* (delete) src/test/org/apache/nutch/crawl/TestCrawlDbStatesExtended.java
* (edit) src/java/org/apache/nutch/parse/ParseSegment.java
* (edit) src/test/org/apache/nutch/indexer/TestIndexerMapReduce.java
* (edit) src/java/org/apache/nutch/crawl/CrawlDatum.java
* (add) src/testresources/fetch-parse-failure/index.html
* (edit) src/java/org/apache/nutch/fetcher/FetcherThread.java
* (edit) src/java/org/apache/nutch/metadata/Nutch.java
* (edit) src/test/org/apache/nutch/crawl/TestCrawlDbStates.java
* (edit) src/java/org/apache/nutch/crawl/CrawlDbReducer.java
* (edit) src/java/org/apache/nutch/indexer/IndexerMapReduce.java
* (edit) src/java/org/apache/nutch/metrics/NutchMetrics.java
* (edit) src/java/org/apache/nutch/parse/ParseStatus.java
* (edit) conf/nutch-default.xml
* (edit) src/test/org/apache/nutch/crawl/CrawlDBTestUtil.java
* (add) src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java
* (add) src/testresources/fetch-parse-failure/test.html
* (add) src/testresources/fetch-parse-failure/robots.txt
* (edit) src/java/org/apache/nutch/fetcher/Fetcher.java
> IndexerMapReduce to delete explicitly not indexable documents
> -------------------------------------------------------------
>
> Key: NUTCH-1732
> URL: https://issues.apache.org/jira/browse/NUTCH-1732
> Project: Nutch
> Issue Type: Bug
> Components: indexer
> Affects Versions: 1.8
> Reporter: Sebastian Nagel
> Assignee: Isabelle Giguere
> Priority: Critical
> Fix For: 1.23
>
>
> In a continuous crawl a previously successfully indexed document (identified
> by a URL) can become "not indexable" for a couple of reasons and must then
> explicitly deleted from the index. Some cases are handled in IndexerMapReduce
> (duplicates, gone documents or redirects, cf. NUTCH-1139) but others are not:
> * failed to parse (but previously successfully parsed): e.g., the document
> became larger and is now truncated
> * rejected by indexing filter (but previously accepted)
> In both cases (maybe there are more) the document should be explicitly
> deleted (if {{-deleteGone}} is set). Note that this cannot be done in
> CleaningJob because data from segments is required.
> We should also update/add a description for {{-deleteGone}}: it does not only
> trigger deletion of gone documents but also of redirects and duplicates (and
> unparseable and skipped docs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
