sgup432 opened a new issue, #16298: URL: https://github.com/apache/lucene/issues/16298
### Description Currently, for sparse fields (where not all documents have a value), we write two structures into the same .dvd file: the IndexedDISI presence bitmap followed by the packed values. IndexedDISI is a compressed list of docIds which helps in answering question like 'Does document X have a value for this field?' We already separated out skip index into a different file - https://github.com/apache/lucene/pull/15976. I wonder if we should do the same for indexedDISI structure? As for example, when we are reading skip index, and we hit a block which is `YES_IF_PRESENT`(all values are in range but some docs are missing), we need to seek into .dvd file to find which docs are present([link](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/DocValuesRangeIterator.java#L354)). I think this shouldn't be required ie interleaving this metadata with actual values. We can either separate out this into a new file or stuff it inside skip index. I prefer the former as it is much clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
