unsubscribe On Tue, Mar 19, 2024 at 2:59 PM Shubham Chaudhary <shubhmas...@gmail.com> wrote:
> Hi Michael, > > Lucene already had int8 vector support since 9.5 (#1054 > <https://github.com/apache/lucene/pull/1054>) but it was left to the user > to get those quantized vectors and index using KnnByteVectorField > < > https://lucene.apache.org/core/9_5_0/core/org/apache/lucene/document/KnnByteVectorField.html > >, > but with Lucene 9.9 out now there is a native support for int8 scalar > quantization (#12582 <https://github.com/apache/lucene/pull/12582>) using > Lucene99ScalarQuantizedVectorsFormat > < > https://lucene.apache.org/core/9_9_1/core/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsFormat.html > > > that > expects a confidence interval from 90-100. Here is a nice blog(s) that > talks about how it works in Lucene. > > - > > https://www.elastic.co/search-labs/blog/articles/scalar-quantization-in-lucene > - https://www.elastic.co/search-labs/blog/articles/scalar-quantization-101 > > Some other references : > - > > https://lucene.apache.org/core/9_9_1/core/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsFormat.html > - > > https://lucene.apache.org/core/9_9_1/core/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsReader.html > - > > https://lucene.apache.org/core/9_9_1/core/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsWriter.html > > > > On Wed, Mar 20, 2024 at 1:54 AM Michael Wechner <michael.wech...@wyona.com > > > wrote: > > > Hi > > > > Cohere recently announced there "compressed" embeddings > > > > https://twitter.com/Nils_Reimers/status/1769809006762037368 > > > > > https://www.linkedin.com/posts/bhavsarpratik_rag-genai-search-activity-7175850704928989187-Ki1N/?utm_source=share&utm_medium=member_desktop > > > > Does Lucene Vector Search support this already, or is somebody working > > on this? > > > > Thanks > > > > Michael > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > >
