[
https://issues.apache.org/jira/browse/SOLR-8709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132101#comment-18132101
]
Nick Shanin commented on SOLR-8709:
-----------------------------------
>From code reading against apache/main b73f03976fa, TopicStream
>(solr/solrj-streaming/src/java/org/apache/solr/client/solrj/io/stream/TopicStream.java:67)
> has no version checksum or retention window, so the proposed sliding-window
>redelivery is not present. Do you still need checksum-based redelivery, or can
>this close with the limitation documented?
🤖 AI-generated 🤖
> Add checksum to the TopicStream to ensure delivery of all documents within a
> Topic
> ----------------------------------------------------------------------------------
>
> Key: SOLR-8709
> URL: https://issues.apache.org/jira/browse/SOLR-8709
> Project: Solr
> Issue Type: Bug
> Reporter: Joel Bernstein
> Priority: Major
>
> Currently the TopicStream can miss documents if version numbers are received
> out-of-order. The TopicStream sorts on version number so it will only miss
> out-of-order versions that span commit boundaries. *Stress testing was not
> able to create a missed document scenario* (see comment below), but code
> review points to the possibility of this happening.
> In order to resolve this issue we can adopt an approach that keeps a checksum
> of the version numbers for a sliding time window. This checksum can be
> checked each run and if the checksums don't match the documents from the time
> window can be resent. As long as the time window is longer then the
> softCommit interval, this will guarantee delivery of all documents for the
> Topic. This won't guarantee *one time delivery* but should be provide a
> reasonable expectation of one time delivery.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
