salvatorecampagna opened a new issue, #16701: URL: https://github.com/apache/lucene/issues/16701
### Description ## Problem I noticed this while investigating an index under sustained updates where deleted documents accumulated without being reclaimed. `LogMergePolicy.findMerges` can leave heavily deleted segments unmerged when their level has too few segments to form a merge. New segments may reach a level too slowly to keep up with accumulated deletes, even under continued indexing. The policy requires `mergeFactor` segments to form an initial candidate, but may merge fewer of them to stay within the size limit. `calibrateSizeByDeletes` scales estimated segment sizes by `(1 - delRatio)`. This can change level grouping and make previously oversized candidates eligible. Deletes therefore can trigger a merge indirectly through size-based selection, but there is no separate trigger based on the amount of reclaimable deletes. ## Behavior Selection tests using stock Lucene defaults. Each cell indicates whether a merge was selected. **LogByteSizeMergePolicy** (`mergeFactor=10`, `maxMergeMB=2048`, `minMergeMB=16`): | Segments | 0% deleted | 20% | 30% | 50% | 99% | |---|---|---|---|---|---| | 2, 6, or 9 | No | No | No | No | No | | 10 or 64 | Yes | Yes | Yes | Yes | Yes | **TieredMergePolicy** (`segmentsPerTier=8`, `maxMergedSegmentMB=5120`, `deletesPctAllowed=20`): | Segments | 0% deleted | 20% | 30% | 50% | 99% | |---|---|---|---|---|---| | 2, 6, or 9 | No | No | Yes | Yes | Yes | | 10 or 64 | Yes | Yes | Yes | Yes | Yes | Reading across each row, `LogByteSizeMergePolicy` gives the same answer at every tested delete fraction in this setup. `TieredMergePolicy` changes its answer with the delete fraction alone. **With nine segments at 99% deletes, `LogByteSizeMergePolicy` selects nothing; `TieredMergePolicy` selects a merge.** `TieredMergePolicy` uses its index-wide deletion budget, selecting merges at the first tested percentage above `deletesPctAllowed=20`. I'll push a companion PR with a couple of tests that reproduce this behavior and link it here. ## Reclamation latency As a simplified illustration, with 50 MB flush segments and full `mergeFactor = f` merges, segment size at level `k` is `50 MB × f^k`. Producing `f` such segments requires `f^(k+1)` flushes: | Level | Segment size | Full level size | Flushes required | At f = 10 | |---|---|---|---|---| | 0 | 50 MB | 500 MB | f | 10 | | 1 | 500 MB | 5 GB | f² | 100 | | 2 | 5 GB | 50 GB | f³ | 1,000 | This assumes no size reduction from deletes or compression changes and a sufficiently large maximum merge size. The default 2048 MB limit already truncates merges of the 500 MB segments at level 1. At a constant flush rate, higher levels take geometrically longer to fill and can hold tens of gigabytes while still below the segment-count threshold. Deletes left unreclaimed during that wait inflate storage relative to live data, increasing production storage requirements and costs; the exact recoverable bytes depend on the data and codec. ## Discussion [#905](https://github.com/apache/lucene/pull/905) already established a delete-reclamation exception to `TieredMergePolicy`'s anti-thrashing rule. That policy also allows singleton merges that reclaim deletes. Should the same principle apply to `LogMergePolicy`? Two possible approaches are: - **Smaller adjacent merges:** allow fewer than `mergeFactor` segments when a level holds substantial reclaimable data. - **Singleton rewrites:** allow individual delete-heavy segments to be rewritten. Both preserve adjacency and trade additional write amplification for shorter reclamation latency. Related: [#13226](https://github.com/apache/lucene/issues/13226) reports similar symptoms under `TieredMergePolicy`, though the cause may differ. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
