dpol1 opened a new issue, #2304:
URL: https://github.com/apache/stormcrawler/issues/2304
The Solr `StatusUpdaterBolt` sends every status as a plain add, so a
DISCOVERED tuple for a URL already in the `status` collection replaces its
record. For DISCOVERED, the OpenSearch bolt uses `create` and the SQL one
`INSERT IGNORE`, which keep what is there.
A test with two bolt instances, the second standing for a restarted
topology, gives:
```
before:
{"url":"https://www.url.net/missing.html","status":"FETCH_ERROR","metadata.fetch.error.count":["1"],"nextFetchDate":"2026-11-05T11:50:02Z",...}
after:
{"url":"https://www.url.net/missing.html","status":"DISCOVERED","nextFetchDate":"2026-11-05T09:50:03Z",...}
before:
{"url":"https://www.url.net/denied.html","status":"ERROR","nextFetchDate":"2026-12-06T09:55:57Z",...}
after:
{"url":"https://www.url.net/denied.html","status":"DISCOVERED","nextFetchDate":"2026-11-05T09:55:58Z",...}
```
The test conf has no `fetchInterval.error`, hence the date in the ERROR
record. The FETCH_ERROR test against the OpenSearch module keeps the record.
The status updater cache hides this as long as the URL is in it. After a
topology restart, or once the entry has expired or been evicted, the DISCOVERED
tuple reaches Solr. I first noticed this in a crawl with the Solr archetype: in
a second crawl topology, the 8 outlinks of `index.html` were fetched again
within a minute although none was due, and URLs that failed twice ended with
`fetch.error.count: 1`.
From the code, an ERROR URL would also become due at once with
`fetchInterval.error: -1`. For a URL denied by robots.txt the fetcher checks
again, so the URL itself would get no request. The code at 3.7.0 sends the same
add. I only ran this on main.
Solr's optimistic concurrency could give the same behaviour as OpenSearch: a
negative `_version_` means the document must not exist, and
`failOnVersionConflicts=false`
([SOLR-13320](https://issues.apache.org/jira/browse/SOLR-13320)) lets the rest
of a batch through. @mvolikas, would that fit the async batches of
`SolrConnection`? I can open a PR with the test.
Separately, the bolt acks right after `addAsync`, before the update reaches
Solr, which is why the test has to wait for the record to appear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
