We have a cluster of hosts running Solr 8.4 Each host has an application which listens to an external source for updated documents. When it gets a document we care about, it indexes that document into the correct Solr core (we are not running cloud).
In our API service, when we get a request to put this type of document, we first query Solr to see if the document exists. If it does not, we then create a new document in our database and the document is sent to the application to be indexed into Solr. If the document we are trying to *put* (in the API Service) exists in Solr, then we throw an exception back to the user if they have not specified the existing version (not the _version_ field from Solr, rather an increasing counter). As part of our write logic, after we put the document into the database, we query the Solr stack until we get the response containing the newly written document. So we know the document was written at this point. Some time later (maybe 5-10 minutes), we get another put request for the same document id. We query Solr, and in some cases, we get no documents returned, even though just before we actually found the document. The document has not been deleted in the interim. We use the same query for both checking for existence at the beginning of the logic, and for checking for eventual consistency after writing to the database, I could add a retry to the first part of the logic (retry if we don't find a document), but the question is why we don't find it the first time (but for the second put). If I query for the document (using the same query), I find the document on each host. Why are we not seeing documents which are actually there?
