[
https://issues.apache.org/jira/browse/CONNECTORS-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431222#comment-13431222
]
Karl Wright commented on CONNECTORS-501:
----------------------------------------
I have confirmed that most if not all deletions are the result of hopcount
delete code being triggered. Furthermore, I have a scenario that would account
for the deletions.
The scenario looks like this:
- Start with two documents, a and b
- There are two paths from a to b, one longer than the other
- There are two paths from the seed to a, one longer than the other
- If we arrive at b via the longer path from seed to a and the shorter path
from a to b, then b may be removed along with the (shorter) link from a to b
- The system will not recover because only the longer link from a to b will be
discoverable after the shorter link has been removed
Basically this means that we cannot remove intrinsic links even though job
queue entries have been removed.
> Medium-scale web crawl with hopcount-based filtering fails to find correct
> number of documents
> ----------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-501
> URL: https://issues.apache.org/jira/browse/CONNECTORS-501
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework agents process, Web connector
> Affects Versions: ManifoldCF 0.6
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.7
>
>
> The new web crawler Postgresql load test, which uses hopcount-based
> filtering, does not discover all 11110 documents it is supposed to. It only
> discovered 10603 when I ran it just now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
