xiangfu0 commented on code in PR #19897:
URL: https://github.com/apache/pinot/pull/19897#discussion_r4364557050
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/text/LuceneTextIndexReader.java:
##########
@@ -542,4 +480,53 @@ public void intField(FieldInfo fieldInfo, int value) {
e);
}
}
+
+ /// Reads the actual Pinot ids after Lucene merges and sorting. Doc values
avoid stored-field decompression;
+ /// indexes created by older writers, including individual older leaves,
retain the stored-field fallback.
+ @VisibleForTesting
+ static boolean fillDocIdMapping(IndexSearcher indexSearcher, PinotDataBuffer
buffer, int numDocs)
+ throws IOException {
+ NumericDocValues docIds =
MultiDocValues.getNumericValues(indexSearcher.getIndexReader(),
+ LuceneTextIndexCreator.PINOT_DOC_ID_DOC_VALUES_FIELD_NAME);
+ class DocIdVisitor extends DocumentStoredFieldVisitor {
+ int _pinotDocId;
+
+ @Override
+ public Status needsField(FieldInfo fieldInfo) {
+ return
LuceneTextIndexCreator.LUCENE_INDEX_DOC_ID_COLUMN_NAME.equals(fieldInfo.name) ?
Status.YES : Status.NO;
+ }
+
+ @Override
+ public void intField(FieldInfo fieldInfo, int value) {
+ _pinotDocId = value;
+ }
+
+ @Override
+ public void stringField(FieldInfo fieldInfo, String value) {
+ _pinotDocId = Integer.parseInt(value);
+ }
+ }
+ DocIdVisitor visitor = new DocIdVisitor();
+ StoredFields storedFields = null;
+ boolean allIdsAreEqual = true;
Review Comment:
Applied in
https://github.com/apache/pinot/commit/5c9153975628af902279503127bd7d141f031460:
the single shared mapping loop now uses pinotDocIdValues, luceneDocId and
isIdentityMapping. Conversion parameters and their Javadocs consistently use
mutableToImmutableDocIdMap.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/text/LuceneTextIndexCreator.java:
##########
@@ -58,6 +62,7 @@
public class LuceneTextIndexCreator extends AbstractTextIndexCreator {
private static final Logger LOGGER =
LoggerFactory.getLogger(LuceneTextIndexCreator.class);
public static final String LUCENE_INDEX_DOC_ID_COLUMN_NAME = "DocID";
+ public static final String PINOT_DOC_ID_DOC_VALUES_FIELD_NAME = "PinotDocID";
Review Comment:
Applied the paired PINOT_DOC_ID_STORED_FIELD_NAME /
PINOT_DOC_ID_DOC_VALUES_FIELD_NAME names in
https://github.com/apache/pinot/commit/5c9153975628af902279503127bd7d141f031460.
LUCENE_INDEX_DOC_ID_COLUMN_NAME remains a public alias; the persisted DocID
and PinotDocID strings are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
