Jackie-Jiang commented on a change in pull request #5177: Lucene DocId to
PinotDocId cache
URL: https://github.com/apache/incubator-pinot/pull/5177#discussion_r400568405
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/text/LuceneTextIndexReader.java
##########
@@ -169,5 +178,57 @@ public void close()
throws IOException {
_indexReader.close();
_indexDirectory.close();
+ _docIdReaderWriter.close();
+ }
+
+ private class DocIdReaderWriter implements Closeable {
+ private PinotDataBuffer _buffer;
+ private final boolean _mappingExists;
+
+ DocIdReaderWriter(File segmentIndexDir, String column, int numDocs)
+ throws Exception {
+ int length = Integer.BYTES * numDocs;
+ File docIdMappingFile = new
File(SegmentDirectoryPaths.findSegmentDirectory(segmentIndexDir),
+ column + LUCENE_TEXT_INDEX_DOCID_MAPPING_FILE_EXTENSION);
+ // The mapping is local to a segment. It is created on the server during
segment load.
+ // Unless we are running Pinot on Solaris/SPARC, the underlying
architecture is
+ // LITTLE_ENDIAN (Linux/x86). So use that as byte order.
+ if (docIdMappingFile.exists()) {
+ // we will be here for segment reload and server restart
+ // for refresh, we will not be here since segment is deleted/replaced
+ // TODO: see if we can prefetch the pages
+ _mappingExists = true;
+ _buffer = PinotDataBuffer.mapFile(docIdMappingFile, /* readOnly */
true, 0, length, ByteOrder.LITTLE_ENDIAN,
+ _column + getClass().getSimpleName());
+ } else {
+ _mappingExists = false;
+ _buffer = PinotDataBuffer.mapFile(docIdMappingFile, /* readOnly */
false, 0, length, ByteOrder.LITTLE_ENDIAN,
+ _column + getClass().getSimpleName());
+ }
+ }
+
+ public void buildDocIdMapping(int numDocs) {
+ if (!_mappingExists) {
+ for (int i = 0; i < numDocs; i++) {
+ try {
+ Document document = _indexSearcher.doc(i);
+ int pinotDocId =
Integer.valueOf(document.get(LuceneTextIndexCreator.LUCENE_INDEX_DOC_ID_COLUMN_NAME));
+ _buffer.putInt(i * Integer.BYTES, pinotDocId);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to build doc id mapping during
segment load: " + e);
+ }
+ }
+ }
+ }
+
+ int getInt(int offset) {
Review comment:
`int getPinotDocId(int luceneDocId)` for better abstraction
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
