I don't have a direct answer for you, but your message causes me to reflect on how Lucene does *not* give users choice of format on a per-type basis (e.g. BinaryDocValues vs NumericDocValues vs etc.), which is annoying. Ideally the previous simple format would be available for you to choose, but it is not. Lucene lets you mix & match PostingsFormats, stored fields formats, term vectors formats, points format. But when it comes to DocValues, it's an all-encompassing format for five different structures. So you take it or leave it; all or nothing. My colleague filed https://issues.apache.org/jira/browse/LUCENE-9236 on this matter; feel free to comment there with your opinion if you have one.
~ David On Mon, May 18, 2020 at 7:52 PM Viral Gandhi <viral.dev...@gmail.com> wrote: > Hi, > I tried upgrading to lucene 8.5.1 from 8.4 and ran our internal > benchmarking. We noticed that with this upgrade our QPS dropped more than > 40% and also affected latencies. After doing some profiling and reverting > LUCENE-9211 <https://issues.apache.org/jira/browse/LUCENE-9211> commit > related to BinaryDocValues compression, we recovered ~30% of the loss. Did > anyone encounter similar situation? > > We rely on BinaryDocValues very heavily. Should this newly introduced > compression be optional to opt-in? > > Also, any other pointers for on recovering remaining 10% loss. When I run > benchmark on 8.4 index with 8.5.1 code, performance is very similar to 8.4. > > Thanks, > Viral Gandhi >
