xiangfu0 commented on code in PR #20215:
URL: https://github.com/apache/pinot/pull/20215#discussion_r4531273638


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java:
##########
@@ -255,9 +259,12 @@ private MutableRoaringBitmap getPinotDocIds(IndexSearcher 
indexSearcher, Mutable
     IntIterator luceneDocIDIterator = luceneDocIds.getIntIterator();
     MutableRoaringBitmap actualDocIDs = new MutableRoaringBitmap();
     try {
+      // One stored-fields reader for all hits, so consecutive documents in a 
compressed block are not decompressed
+      // again for each hit, and only the doc id field is loaded

Review Comment:
   MINOR: this comment doesn't describe what Lucene 9.12 does:
   - The old `IndexSearcher#doc` → `SegmentReader#document` path already reuses 
a per-thread `StoredFieldsReader` (`SegmentCoreReaders.fieldsReaderLocal`).
   - A non-merge `Lucene90CompressingStoredFieldsReader` decompresses the 
document again on every call (`BlockState#document`), whichever reader is used.
   
   Suggest stating the actual reason, e.g. `// Use the non-deprecated 
StoredFields API (IndexSearcher#doc is removed in Lucene 10) and load only the 
doc id field`. The same applies to `MultiColumnRealtimeLuceneTextIndex` and the 
PR description.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java:
##########
@@ -255,9 +259,12 @@ private MutableRoaringBitmap getPinotDocIds(IndexSearcher 
indexSearcher, Mutable
     IntIterator luceneDocIDIterator = luceneDocIds.getIntIterator();
     MutableRoaringBitmap actualDocIDs = new MutableRoaringBitmap();
     try {
+      // One stored-fields reader for all hits, so consecutive documents in a 
compressed block are not decompressed
+      // again for each hit, and only the doc id field is loaded
+      StoredFields storedFields = indexSearcher.storedFields();

Review Comment:
   FYI, perf is mixed. `storedFields()` clones one reader per leaf on every 
query, while the old path reused per-thread clones. I compared the two loops on 
the same reader (Lucene 9.12, JDK 25, p50):
   - 100k hits on 1 leaf: 16.5 → 15.7 ms
   - 200 hits over 20 leaves: 0.32 → 0.47 ms
   - 200 hits over 200 leaves: 0.24 → 0.93 ms
   
   End-to-end `getDocIds` showed no difference within about ±20% noise. Not 
blocking, since this is the cost of the Lucene 10 API, but I wouldn't call it 
an optimization.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to