neoremind commented on PR #16534: URL: https://github.com/apache/lucene/pull/16534#issuecomment-5357412664
@msokolov thanks, good callout! I searched the codebase, this was introduced back in 2010/07 in this [commit](https://github.com/apache/lucene/commit/42b9f0caa32ff71782a949264d033a346ede7b26#diff-22ab6c31dc6317827019d7c86faa08fc826efe921188da71743bc3b884d058ee). There seems no specific reason. But my guess is that originally the two pools were separate but merged later. I can prove that as it's fairly easy to separate them by modifying several lines of code and lucene works well offline on my local machine. By merging them, in theory, from my view, one clear win is that when we add terms, the term bytes and the postings slice are adjacent, so they reside in one cache line during the initial write, also we can avoid another indirection of pool reference as well. However, the downside of merging them is that, as I illustrated in https://github.com/apache/lucene/issues/11608#issuecomment-5265355538, for frequently seen terms, the BytesRefHash lookup has to compare term bytes, with a shared pool, the cache line it pulls also carries postings slices leading to potential cache thrashing; also during the sorting phase before flush, the radix sort walks over term bytes many times, interleaved design might be bad for spatial locality as sorting on ly cares about terms not posting. I plan to try to separate the terms' bytes and terms' posting pool, and vet with luceneutil wikipedia benchmark and some micro JMH benchmarks spanning the UUID case (many unique terms) and the wikipedia-ish frequent-term case to see if this can be a generic one or not. Will share results once I finish. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
