xiangfu0 commented on code in PR #19897:
URL: https://github.com/apache/pinot/pull/19897#discussion_r4289315552
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/text/LuceneTextIndexCreator.java:
##########
@@ -264,21 +268,19 @@ private void buildMappingFile(File segmentIndexDir,
String column, Directory dir
try (PinotDataBuffer buffer = PinotDataBuffer.mapFile(docIdMappingFile, /*
readOnly */ false, 0, length,
ByteOrder.LITTLE_ENDIAN, desc)) {
try {
- // If immutableToMutableIdMap is null, then docIds should not change
between the mutable and immutable segments.
- // Therefore, the mapping file can be built without doing an
additional docId conversion
- if (mutableToImmutableDocIdMap == null) {
- for (int i = 0; i < numDocs; i++) {
+ NumericDocValues docIds = MultiDocValues.getNumericValues(indexReader,
PINOT_DOC_ID_DOC_VALUES_FIELD_NAME);
+ for (int i = 0; i < numDocs; i++) {
+ int pinotDocId;
+ if (docIds != null && docIds.advanceExact(i)) {
+ pinotDocId = Math.toIntExact(docIds.longValue());
+ } else {
+ // Older mutable indexes contain only the stored field.
Mixed-version leaves can also lack doc values.
Document document = indexSearcher.storedFields().document(i);
Review Comment:
Cached StoredFields lazily on the first fallback document and reused it for
the remaining legacy or mixed-index documents. The all-doc-values path never
opens it.
Follow-up:
[2c07d0f6c7](https://github.com/apache/pinot/commit/2c07d0f6c70d6830b0e3867a4b4ec8559f45d258).
Validation: 17 focused tests, including real legacy-index corruption, and
affected-module checks passed.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/text/LuceneTextIndexCreator.java:
##########
@@ -264,21 +268,19 @@ private void buildMappingFile(File segmentIndexDir,
String column, Directory dir
try (PinotDataBuffer buffer = PinotDataBuffer.mapFile(docIdMappingFile, /*
readOnly */ false, 0, length,
ByteOrder.LITTLE_ENDIAN, desc)) {
try {
- // If immutableToMutableIdMap is null, then docIds should not change
between the mutable and immutable segments.
- // Therefore, the mapping file can be built without doing an
additional docId conversion
- if (mutableToImmutableDocIdMap == null) {
- for (int i = 0; i < numDocs; i++) {
+ NumericDocValues docIds = MultiDocValues.getNumericValues(indexReader,
PINOT_DOC_ID_DOC_VALUES_FIELD_NAME);
+ for (int i = 0; i < numDocs; i++) {
+ int pinotDocId;
+ if (docIds != null && docIds.advanceExact(i)) {
+ pinotDocId = Math.toIntExact(docIds.longValue());
+ } else {
+ // Older mutable indexes contain only the stored field.
Mixed-version leaves can also lack doc values.
Document document = indexSearcher.storedFields().document(i);
- int pinotDocId =
Integer.parseInt(document.get(LuceneTextIndexCreator.LUCENE_INDEX_DOC_ID_COLUMN_NAME));
- buffer.putInt(i * Integer.BYTES, pinotDocId);
- }
- } else {
- for (int i = 0; i < numDocs; i++) {
- Document document = indexSearcher.storedFields().document(i);
- int mutableDocId =
Integer.parseInt(document.get(LuceneTextIndexCreator.LUCENE_INDEX_DOC_ID_COLUMN_NAME));
- int immutableDocId = mutableToImmutableDocIdMap[mutableDocId];
- buffer.putInt(i * Integer.BYTES, immutableDocId);
+ pinotDocId =
Integer.parseInt(document.get(LUCENE_INDEX_DOC_ID_COLUMN_NAME));
Review Comment:
Added the same missing-ID IOException and document-number message as the
reader path. A regression converts a real legacy Lucene index with a missing
stored DocID and verifies that clear corruption diagnostic.
Follow-up:
[2c07d0f6c7](https://github.com/apache/pinot/commit/2c07d0f6c70d6830b0e3867a4b4ec8559f45d258).
Validation: 17 focused tests, including real legacy-index corruption, and
affected-module checks passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
