sgup432 opened a new pull request, #16358: URL: https://github.com/apache/lucene/pull/16358
### Description As of today, if someone needs to add a new metadata or stats in the skip index, it requires a version bump every time, changing the skip index jump table value to the updated value which takes into account of the size from new changes, and then accordingly during read, add version checks to read those extra values. This is because the skip index uses a fixed schema where every byte position is predetermined, so the reader navigates purely by precomputed offsets and any new field invalidates them. For example, [PR #15737](https://github.com/apache/lucene/pull/15737) adds pre-aggregated sum and value count per interval to the skip index, which required a backward compatibility commit, recalculating the fixed byte offsets, and gating reads behind version checks. This is pretty cumbersome and error prone if we need to add new metadata every time. With this PR, we replace the fixed-schema layout with length-prefixed, type-tagged entries. Each stat is identified by a type byte, and the reader uses stored lengths to skip over anything it doesn't recognize. This means future stats can be appended by simply writing a new type tag in the writer and adding a case in the reader's switch, with no version bump, no jump table recalculation, and older readers that don't know about the new stat will just skip past it harmlessly. For example, adding a new stats(eg: sum like mentioned above) after this PR: ``` Adding a new stat (e.g. sum) after this PR: Writer (Consumer): // Just append after existing stats in the entry buffer entryBuffer.writeByte(Lucene90DocValuesFormat.SKIP_STAT_SUM); entryBuffer.writeLong(accumulator.sumValue); Reader (Producer): switch (statType) { case SKIP_STAT_RANGE -> { /* existing */ } case SKIP_STAT_SUM -> { sumValue[level] = input.readLong(); } default -> input.seek(entryEnd); } Format constant: static final byte SKIP_STAT_SUM = 0x02; ``` <!-- If this is your first contribution to Lucene, please make sure you have reviewed the contribution guide. https://github.com/apache/lucene/blob/main/CONTRIBUTING.md --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
