neoremind commented on issue #6916: URL: https://github.com/apache/lucene/issues/6916#issuecomment-5265524536
I think the structural design @dweiss described ten years ago, with a hash table of `hash(byteSequences) -> term ID` + a compact byte storage is essentially what `BytesRefHash` has already done today. The puesdo code looks much alike of `add()` operation in `BytesRefHash`. The one piece that differs from the ten years' ago proposal is the length encoding, we can remove this if we can do something like `offsets[i+1] - offsets[i]` in `bytesStart[]` to deduce the length. But the reality is that `ByteBlockPool` is shared with postings streams, every term appends a 5-byte posting and the posting may point to other far away offset forming a chain, I'd like to optimize in this direction to separate this, I think this not only saves memory footprint, but also increases cache hit/locality rate during indexing, as we read and load terms to compare frequently while only appending postings like doc id, freq, prox sequentially. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
