bharath-techie commented on PR #15330: URL: https://github.com/apache/lucene/pull/15330#issuecomment-3398230727
Hi @rmuir @msokolov , I'm yet to review this PR. But I see your points as the hard limit check should be enough as it accounts for byteBlockPool as well. For context , I originally created this issue https://github.com/apache/lucene/issues/15152 - where an opensearch user encountered the byteblockpool overflow during recovery. ``` message [shard failure, reason [index id[3458764570588151359] origin[LOCAL_TRANSLOG_RECOVERY] seq#[53664468]]], failure [NotSerializableExceptionWrapper[arithmetic_exception: integer overflow]], markAsStale [true]] NotSerializableExceptionWrapper[arithmetic_exception: integer overflow] at java.lang.Math.addExact(Math.java:883) at org.apache.lucene.util.ByteBlockPool.nextBuffer(ByteBlockPool.java:199) at org.apache.lucene.index.ByteSlicePool.allocKnownSizeSlice(ByteSlicePool.java:118) at org.apache.lucene.index.ByteSlicePool.allocSlice(ByteSlicePool.java:98) at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:226) at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:266) at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:86) at org.apache.lucene.index.FreqProxTermsWriterPerField.addTerm(FreqProxTermsWriterPerField.java:197) at org.apache.lucene.index.TermsHashPerField.positionStreamSlice(TermsHashPerField.java:214) at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:202) at org.apache.lucene.index.IndexingChain$PerField.invertTokenStream(IndexingChain.java:1287) at org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1183) at org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:731) at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:609) at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:263) at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425) at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1558) at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1516) at org.opensearch.index.engine.InternalEngine.addStaleDocs(InternalEngine.java:1291) at org.opensearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1210) at org.opensearch.index.engine.InternalEngine.index(InternalEngine.java:1011) at org.opensearch.index.shard.IndexShard.index(IndexShard.java:1226) ``` I think the check for `IndexWriterHardLimit` in `FlushControl` comes after we do `DocumentsWriter.updateDocuments` where adding many documents could potentially exceed the limit and hit this exception. 1. Do we need a buffer for writer limits to account for next set of documents ? 2. Do we need to limit the number of docs that can be passed to this method ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
