raghavyadav01 opened a new pull request, #20215:
URL: https://github.com/apache/pinot/pull/20215

   ## Summary
   
   On consuming segments, `RealtimeLuceneTextIndex` and 
`MultiColumnRealtimeLuceneTextIndex` map every Lucene hit of a `TEXT_MATCH` to 
its Pinot doc id by reading the doc id stored field. They did this with 
`IndexSearcher.doc(luceneDocId)` once per hit. That method:
   
   - is deprecated since Lucene 9.7 and removed in Lucene 10;
   - gets a new stored-fields reader on every call, so hits in the same 
compressed block decompress that block again;
   - visits every stored field of the document.
   
   Both indexes now get one `StoredFields` reader per query and load only the 
doc id field. Results are unchanged. The offline 
`MultiColumnLuceneTextIndexReader` already reads stored fields this way.
   
   ## Testing
   
   - New `testHitsMapToPinotDocIdsAcrossManyDocuments` and 
`testMultiColumnHitsMapToPinotDocIdsAcrossManyDocuments`: 5,000 documents 
spanning many stored-field blocks; every hit must map to exactly its Pinot doc 
id. The multi-column realtime index had no unit test before. These tests guard 
the behavior and pass both before and after the change.
   - `LuceneMutableTextIndexTest` (12) and 
`RealtimeLuceneTextIndexResourceTrackingTest` (6) pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to