xiangfu0 commented on code in PR #19897:
URL: https://github.com/apache/pinot/pull/19897#discussion_r4366126058
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/text/LuceneTextIndexCreator.java:
##########
@@ -331,6 +313,17 @@ public void add(String[] documents, int length) {
}
}
+ private void addDocIdFields(Document document) {
+ int docId = _nextDocId++;
+ // Keep the stored field for older readers. Immutable indexes add doc
values to avoid decompressing every stored
+ // document on segment load and retain the actual Pinot id when Lucene
flushes or merges reorder documents.
+ document.add(new StoredField(PINOT_DOC_ID_STORED_FIELD_NAME, docId));
+ // A TEXT column with this name must keep the stored-field fallback to
avoid conflicting Lucene field schemas.
+ if (_commitOnClose &&
!_textColumn.equals(PINOT_DOC_ID_DOC_VALUES_FIELD_NAME)) {
Review Comment:
Updated in
[811f525472](https://github.com/apache/pinot/commit/811f525472c7a7c4b23ee056553c95883c899f81):
fresh immutable indexes now replace stored DocID with numeric doc values.
Creation verifies final IDs and persists Skip or packages the legacy mapping
for older readers, so combined indexes avoid the stored+doc-values duplication.
The updated description includes full V3 segment sizes/load times and pinned
1.5.0-rc0 reader checks. It also records the tradeoff: gzip archives grew in
three layouts, while uncombined first load dropped from about 246–255 ms to 14
ms on this dataset. [Reproduction and raw
evidence](https://gist.github.com/xiangfu0/6b70e028a73c03a661a436dce9a432cf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
