jimczi commented on issue #16644: URL: https://github.com/apache/lucene/issues/16644#issuecomment-5566975750
Yeah I dug into it a bit. The handle count is roughly updated fields times the overlay budget. So it does grow with the number of fields, each one keeps its own stack of generations open. It stays bounded per field though, folding collapses the deltas once a field hits the budget. So it won't grow unbounded. The per field bit is not new. It was already like that before overlays, each updated field always got its own generation. Overlays just add the stacking on top, base plus up to N deltas instead of one rewritten gen. We could probably batch the fields updated in the same flush into one generation and drop the times fields factor. The reader already dedups producers by generation, so it would mostly work. I don't think it's needed for the handle limit itself though. It would mainly help the case with hundreds of updatable fields. I put up #16645 to randomize the budget in the tests so they stop tripping the limit, and we get coverage of the lower budgets too. Separately, maybe we lower the default from 16 to 8? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
