Jackie-Jiang commented on a change in pull request #5177: Lucene DocId to
PinotDocId cache
URL: https://github.com/apache/incubator-pinot/pull/5177#discussion_r400583323
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/text/LuceneTextIndexReader.java
##########
@@ -169,5 +173,50 @@ public void close()
throws IOException {
_indexReader.close();
_indexDirectory.close();
+ _docIdTranslator.close();
+ }
+
+ private static class DocIdTranslator implements Closeable {
+ final PinotDataBuffer _buffer;
+
+ DocIdTranslator(File segmentIndexDir, String column, int numDocs,
IndexSearcher indexSearcher)
+ throws Exception {
+ int length = Integer.BYTES * numDocs;
+ File docIdMappingFile = new
File(SegmentDirectoryPaths.findSegmentDirectory(segmentIndexDir),
+ column + LUCENE_TEXT_INDEX_DOCID_MAPPING_FILE_EXTENSION);
+ // The mapping is local to a segment. It is created on the server during
segment load.
+ // Unless we are running Pinot on Solaris/SPARC, the underlying
architecture is
+ // LITTLE_ENDIAN (Linux/x86). So use that as byte order.
+ String desc = "Text index docId mapping buffer: " + column;
+ if (docIdMappingFile.exists()) {
+ // we will be here for segment reload and server restart
+ // for refresh, we will not be here since segment is deleted/replaced
+ // TODO: see if we can prefetch the pages
+ _buffer =
+ PinotDataBuffer.mapFile(docIdMappingFile, /* readOnly */ true, 0,
length, ByteOrder.LITTLE_ENDIAN, desc);
+ } else {
+ _buffer =
+ PinotDataBuffer.mapFile(docIdMappingFile, /* readOnly */ false, 0,
length, ByteOrder.LITTLE_ENDIAN, desc);
+ for (int i = 0; i < numDocs; i++) {
+ try {
+ Document document = indexSearcher.doc(i);
+ int pinotDocId =
Integer.parseInt(document.get(LuceneTextIndexCreator.LUCENE_INDEX_DOC_ID_COLUMN_NAME));
+ _buffer.putInt(i * Integer.BYTES, pinotDocId);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to build doc id mapping during
segment load, " + e);
+ }
+ }
+ }
+ }
+
+ int getPinotDocId(int offset) {
Review comment:
Pass in `luceneDocId` and wrap the `_buffer.getInt(luceneDocId *
Integer.BYTES)` logic inside
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
