[jira] [Commented] (NUTCH-1732) IndexerMapReduce to delete explicitly not indexable documents

[ASF GitHub Bot (Jira)]

    [ 
https://issues.apache.org/jira/browse/NUTCH-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18088712#comment-18088712
 ] 

ASF GitHub Bot commented on NUTCH-1732:
---------------------------------------

sonarqubecloud[bot] commented on PR #891:
URL: https://github.com/apache/nutch/pull/891#issuecomment-4698325531

   ## [![Quality Gate 
Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png
 'Quality Gate 
Failed')](https://sonarcloud.io/dashboard?id=apache_nutch&pullRequest=891) 
**Quality Gate failed**  
   Failed conditions  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png
 '') [48.0% Coverage on New 
Code](https://sonarcloud.io/component_measures?id=apache_nutch&pullRequest=891&metric=new_coverage&view=list)
 (required ≥ 80%)  
     
   <!

> IndexerMapReduce to delete explicitly not indexable documents
> -------------------------------------------------------------
>
>                 Key: NUTCH-1732
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1732
>             Project: Nutch
>          Issue Type: Bug
>          Components: indexer
>    Affects Versions: 1.8
>            Reporter: Sebastian Nagel
>            Priority: Critical
>             Fix For: 1.23
>
>
> In a continuous crawl a previously successfully indexed document (identified 
> by a URL) can become "not indexable" for a couple of reasons and must then 
> explicitly deleted from the index. Some cases are handled in IndexerMapReduce 
> (duplicates, gone documents or redirects, cf. NUTCH-1139) but others are not:
> * failed to parse (but previously successfully parsed): e.g., the document 
> became larger and is now truncated
> * rejected by indexing filter (but previously accepted)
> In both cases (maybe there are more) the document should be explicitly 
> deleted (if {{-deleteGone}} is set). Note that this cannot be done in 
> CleaningJob because data from segments is required. 
> We should also update/add a description for {{-deleteGone}}: it does not only 
> trigger deletion of gone documents but also of redirects and duplicates (and 
> unparseable and skipped docs).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)