jimczi opened a new pull request, #16571: URL: https://github.com/apache/lucene/pull/16571
A merge takes N segments and writes one. This lets it take N and write M, by giving each output a contiguous document range of every input: ```java public boolean isPartitioned() public int[][] getDocRangePartitions(List<CodecReader> readers) ``` One entry per input: M + 1 non-decreasing boundaries, where output `o` takes `[b[o], b[o+1])`. Called after the merge readers are open, so boundaries can be placed on values read from the readers. `null` is the default and keeps today's behaviour. M may be larger or smaller than N, and N may be 1. Each input is wrapped in a reader presenting everything outside an output's range as deleted, so every format merges through its existing code and none of them learns about partitioning. An ordinary merge is the one-output case of the same path. Bytes written are unchanged: each document is written once, into the output that owns it. An index sort is not required; without one, documents are shared out by position. ### Example `BalancedSegmentsMergePolicy` in `misc`. `forceMerge(n)` says how many segments to leave but not how the documents are shared between them: a hundred equal segments merged into four leaves 97,000 / 1,000 / 1,000 / 1,000. The policy plans the whole forced merge at once, packing segments into groups and giving each group the outputs its size deserves, so the largest merge is about N/M. `forceMerge(16)` on a one-segment index splits it into sixteen. ### Cost Doc values, norms and vectors seek to an output's range (second commit), so the outputs together read those columns once. Stored fields and term vectors random-access per live document, so an output touches only the chunks its range falls in. Terms dictionary and BKD cannot seek by document and are read once per output. Merge inputs are verified once per merge rather than once per output. ### Gaps - Postings and points are read once per output. A push path feeding M writers would fix it and needs a public SPI change; left out. - Boundaries may not cut inside a document block when a parent field is set. Checked on the offsets. - A merge policy cannot record what an output holds, so a key-partitioning policy cannot recognise its own output on the next merge. `OneMerge.getOutputDiagnostics(int)` covers this; follow-up. ### Testing Each output is duelled against a segment built from exactly its documents in the same order, across all field types, sorted and unsorted indexes, blocks aligned and misaligned to boundaries, concurrent deletes and doc-values updates mid-merge, soft deletes, rollback, an IO failure while writing one output, malformed partitions, and wrapper preservation. Codecs are randomized. `CHANGES.txt` to follow once this has a number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
