Hi Ben, I am working on something very close to what Michael Sokolov has done. I see OOMs on the Writer when it tries to index 130M 8 bit / 4 bit quantized vectors on a single big box with a 40 GB heap, with HNSW disabled. I've tried indexing all the vectors as plain vectors converted to floats converted to BinaryDocValues and that worked fine. I tried smaller heap sizes starting with 20 GB but they all failed. 40 GB heap is already quite a bit and hence the deep dive.. The Writer process is not doing any other RAM heavy things so I am assuming that the memory is dominated by vectors. The vectors originally had 768 dimensions.
My process was initially failing when it reached an index size of about ~40 GB. OOM stack failures were close to when merges were happening. I tried reducing the number of concurrent merges that were allowed, the number of segments that can be merged at once, and that helped, but only a little. I was still seeing OOMs. Then, I adopted a NoMergePolicy and was able to build a ~240 GB quantized index, but that too, OOMs out before indexing all the docs. The ramBufferSizeMB is 4096, so roughly speaking it has 2.5 GB ish segments, and multiple smaller ones per flush. I am assuming the quantization on flush is causing the failures. Which operation during flush is taking up so much memory? I don't know .. I don't think the quantization factor (bits) affects the memory much. Do we do the quartile calculation, eventual quantization in a streaming fashion? Are there any other things that jump out to you as memory bottlenecks/methods that you think would be memory hungry? I have the heap dump and am also analyzing it myself. > - Lucene99FlatVectorsWriter gets the float[] vector and makes a copy of it (does this no matter what) This maybe could've caused the problems.. but if the ramBufferSizeMB is small and merges are disabled, it's hard to imagine how 40 GB could've been consumed. Best, Gautam Worah. On Wed, Jun 12, 2024 at 9:42 AM Benjamin Trent <ben.w.tr...@gmail.com> wrote: > Michael, > > Empirically, I am not surprised there is an increase in heap usage. We > do have extra overhead with the scalar quantization on flush. There > may also be some additional heap usage on merge. > > I just don't think it is via: Lucene99FlatVectorsWriter > > On Wed, Jun 12, 2024 at 11:55 AM Michael Sokolov <msoko...@gmail.com> > wrote: > > > > Empirically I thought I saw the need to increase JVM heap with this, > > but let me do some more testing to narrow down what is going on. It's > > possible the same heap requirements exist for the non-quantized case > > and I am just seeing some random vagary of the merge process happening > > to tip over a limit. It's also possible I messed something up in > > https://github.com/apache/lucene/pull/13469 which I am trying to use > > in order to index quantized vectors without building an HNSW graph. > > > > On Wed, Jun 12, 2024 at 10:24 AM Benjamin Trent <ben.w.tr...@gmail.com> > wrote: > > > > > > Heya Michael, > > > > > > > the first one I traced was referenced by vector writers involved in > a merge (Lucene99FlatVectorsWriter.FieldsWriter.vectors). Is this expected? > > > > > > Yes, that is holding the raw floats before flush. You should see > > > nearly the exact same overhead there as you would indexing raw > > > vectors. I would be surprised if there is a significant memory usage > > > difference due to Lucene99FlatVectorsWriter when using quantized vs. > > > not. > > > > > > The flow is this: > > > > > > - Lucene99FlatVectorsWriter gets the float[] vector and makes a copy > > > of it (does this no matter what) and passes on to the next part of the > > > chain > > > - If quantizing, the next part of the chain is > > > Lucene99ScalarQuantizedVectorsWriter.FieldsWriter, which only keeps a > > > REFERENCE to the array, it doesn't copy it. The float vector array is > > > then passed to the HNSW indexer (if its being used), which also does > > > NOT copy, but keeps a reference. > > > - If not quantizing but indexing, Lucene99FlatVectorsWriter will pass > > > it directly to the hnsw indexer, which does not copy it, but does add > > > it to the HNSW graph > > > > > > > I wonder if there is an opportunity to move some of this off-heap? > > > > > > I think we could do some things off-heap in the ScalarQuantizer. Maybe > > > even during "flush", but we would have to adjust the interfaces some > > > so that the scalarquantizer can know where the vectors are being > > > stored after the initial flush. Right now there is no way to know the > > > file nor file handle. > > > > > > > I can imagine that when we requantize we need to scan all the > vectors to determine the new quantization settings? > > > > > > We shouldn't be scanning every vector. We do take a sampling, though > > > that sampling can be large. There is here an opportunity for off-heap > > > action if possible. Though I don't know how we could do that before > > > flush. I could see the off-heap idea helping on merge. > > > > > > > Maybe we could do two passes - merge the float vectors while > recalculating, and then re-scan to do the actual quantization? > > > > > > I am not sure what you mean here by "merge the float vectors". If you > > > mean simply reading the individual float vector files and combining > > > them into a single file, we already do that separately from > > > quantizing. > > > > > > Thank you for digging into this. Glad others are experimenting! > > > > > > Ben > > > > > > On Wed, Jun 12, 2024 at 8:57 AM Michael Sokolov <msoko...@gmail.com> > wrote: > > > > > > > > Hi folks. I've been experimenting with our new scalar quantization > > > > support - yay, thanks for adding it! I'm finding that when I index a > > > > large number of large vectors, enabling quantization (vs simply > > > > indexing the full-width floats) requires more heap - I keep getting > > > > OOMs and have to increase heap size. I took a heap dump, and not > > > > surprisingly I found some big arrays of floats and bytes, and the > > > > first one I traced was referenced by vector writers involved in a > > > > merge (Lucene99FlatVectorsWriter.FieldsWriter.vectors). Is this > > > > expected? I wonder if there is an opportunity to move some of this > > > > off-heap? I can imagine that when we requantize we need to scan all > > > > the vectors to determine the new quantization settings? Maybe we > > > > could do two passes - merge the float vectors while recalculating, > and > > > > then re-scan to do the actual quantization? > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
