mikemccand commented on issue #1361: LUCENE-8118: Throw exception if DWPT grows beyond it's maximum ram limit URL: https://github.com/apache/lucene-solr/pull/1361#issuecomment-602027021 > > 1. fixing exception handling in DW.updateDocuments to handle this exception gracefully (non-aborting). > > the problem in this method was that we need to flush the DWPT to disk even if we hit an non-aborting exception. Otherwise the next doc would hit the same exception or we would violate the assumption that we never receive a flush pending DWPT. Thanks for the explanation; make sense. > One thing that I keep thinking about is what happens if we index documents close to the limit and then the next batch of documents would exceed that limit. That would mean that we reject docs event though we could index them into a different DWPT. Should we at least try to index it again in a fresh DWPT unless the DWPT that rejected it was empty? I mean that would prevent the issue and should be straight forward to implement. I can work on this in a followup. +1, that's an awesome idea! In general when picking among the N free DWPTs when a new document arrives shouldn't we pick the DWPT that's smallest (least `.ramBytesUsed()`)?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
