Hi Bruno,
I have reproduced the issue, below are 2 scenarios -
1. On a 128 GB L3 distributed system (data size 12 million) replica
recovery fails after all Solr DS restart restart.
2. On a 256 GB L3 distributed system (data size 12 million) replica
recovery does not fail on all Solr DS restart. I deleted one active replica
then it went to recovery mode and recovery failed. Getting below error log
[2025-01-17T12:26:11.472] WARN [host]
org.apache.solr.handler.ReplicationHandler Exception while writing response
for params:
generation=82&qt=/replication&file=_6p.kdd&offset=8388608&checksum=true&wt=filestream&command=filecontent
java.io.EOFException: Read beyond EOF (position=8388608,
arrayLength=424066, fileLength=8812650) in Decrypting
MemorySegmentIndexInput(path="/data/1155855_shard1_replica_n26/data/index/_6p.kdd")
at
org.apache.solr.encryption.crypto.DecryptingIndexInput.readBytes(DecryptingIndexInput.java:223)
~[solr-encryption-plugin-9.0.0-SNAPSHOT.jar:?]
at
org.apache.solr.handler.ReplicationHandler$DirectoryFileStream.write(ReplicationHandler.java:1635)
~[solr-core-9.6.0.jar:9.6.0 f8e5a93c11267e13b7b43005a428bfb910ac6e57 - gus
- 2024-04-22 23:20:52]
at org.apache.solr.core.SolrCore$3.write(SolrCore.java:3056)
~[solr-core-9.6.0.jar:9.6.0 f8e5a93c11267e13b7b43005a428bfb910ac6e57 - gus
- 2024-04-22 23:20:52]
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:59)
~[solr-core-9.6.0.jar:9.6.0 f8e5a93c11267e13b7b43005a428bfb910ac6e57 - gus
- 2024-04-22 23:20:52]
at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:1031)
~[solr-core-9.6.0.jar:9.6.0 f8e5a93c11267e13b7b43005a428bfb910ac6e57 - gus
- 2024-04-22 23:20:52]
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:621)
~[solr-core-9.6.0.jar:9.6.0 f8e5a93c11267e13b7b43005a428bfb910ac6e57 - gus
- 2024-04-22 23:20:52]
at
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:262)
~[solr-core-9.6.0.jar:9.6.0 f8e5a93c11267e13b7b43005a428bfb910ac6e57 - gus
- 2024-04-22 23:20:52]
at
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:219)
~[solr-core-9.6.0.jar:9.6.0 f8e5a93c11267e13b7b43005a428bfb910ac6e57 - gus
- 2024-04-22 23:20:52]
at
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:249)
~[solr-core-9.6.0.jar:9.6.0 f8e5a93c11267e13b7b43005a428bfb910ac6e57 - gus
- 2024-04-22 23:20:52]
at
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:215)
~[solr-core-9.6.0.jar:9.6.0 f8e5a93c11267e13b7b43005a428bfb910ac6e57 - gus
- 2024-04-22 23:20:52]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213)
~[solr-core-9.6.0.jar:9.6.0 f8e5a93c11267e13b7b43005a428bfb910ac6e57 - gus
- 2024-04-22 23:20:52]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
~[solr-core-9.6.0.jar:9.6.0
3. On small-size encrypted (data) this issue is not producing. and
4. On large-size unencrypted data issues are not coming
Any help or direction to fix this issue will be invaluable.
Regards,
Manish Kumar
On Sun, Jan 5, 2025 at 6:33 PM Bruno Roustant <bruno.roust...@gmail.com>
wrote:
> Hi Manish Kumar,
>
> Can you open a git issue in the Solr sandbox repo?
> https://github.com/apache/solr-sandbox/issues
>
> What is the replica type? The default NRT?
> If you can contribute, it would be great to propose a PR with a Solr test
> reproducing the problem.
>
> Thanks,
> Bruno
>
> Le sam. 4 janv. 2025 à 07:02, Manish Kumar singh <mksmksi...@gmail.com> a
> écrit :
>
> > Hello,
> >
> > I am using the Solr encryption plugin for data and index encryption. It
> is
> > working fine for single-tenant systems. On a distributed system with two
> or
> > more tenants, the follower replica fails to start replication when a
> > collection has two or more replicas in a shard,, Replica recovery fails,
> > and it continuously retries and fails.
> >
> > I have tested this behaviour in a multi-sharded Solr collection with two
> > replicas
> > per shard.
> >
> > On the Solr log getting this error - org.apache.solr.update.processor.D
> > istributedUpdateProcessor Ignoring commit while not ACTIVE - state:
> > BUFFERING replay: false
> >
> > Any suggestions or directions to resolve the issue are helpful, and I am
> > willing to contribute to the necessary development to fix this issue if
> > needed.
> >
> >
> > Regards,
> > Manish Kumar
> >
>
