Fair enough - are you planning to offer such a service;) sounds exciting! -Mike
On Tue, Feb 15, 2022 at 6:00 PM Michael Wechner <michael.wech...@wyona.com> wrote: > true :-) when you are the one controlling the input of vectors, then a > method to disable the maximum limit would be sufficient. > > But I could imagine when you offer Lucene as a service where people can > for example configure their own "sentence embedding models" and you would > like to offer a different maximum limit than the default of 1024, then I > think a method to reset the maximum limit would make sense. Examples could > be a service of OpenAI or vector search databases like for example Weaviate > or Pinecone. > > Thanks > > Michael > > > > > Am 15.02.22 um 23:34 schrieb Michael Sokolov: > > I don't think it makes sense to have a static variable maximum that you > can change by calling a method. What purpose would it serve? > > On Tue, Feb 15, 2022, 2:39 PM Michael Wechner <michael.wech...@wyona.com> > wrote: > >> Hi Alessandro >> >> No, I have not created a Jira ticket, but I would be happy to create one, >> just let me know or please feel free to create one. >> >> I understand the concerns about the limits in general and I think it >> makes sense to have a default max dimensions limit, but I could imagine it >> needs to be increased eventually and being able to increase it >> programmatically and at your own risk will help people using Lucene. >> >> Thanks >> >> Michael >> >> Am 15.02.22 um 19:22 schrieb Alessandro Benedetti: >> >> Hi Michael, >> let's create a Jira ticket to use a higher value(if you haven't already). >> I would be happy to consider the patch/or do it myself but after 10/03. >> Once the pull request is ready (including the Javadoc documentation that >> clearly states that if you go above X it's at your own risk), we'll involve >> also Michael Sokolov and the other committers familiar with this area of >> the code. >> >> Cheers >> >> -------------------------- >> Alessandro Benedetti >> Apache Lucene/Solr PMC member and Committer >> Director, R&D Software Engineer, Search Consultant >> >> www.sease.io >> >> >> On Sat, 12 Feb 2022 at 22:53, Michael Wechner <michael.wech...@wyona.com> >> wrote: >> >>> Hi >>> >>> I just tried to test the OpenAI model "text-similarity-davinci-001" with >>> 12288 dimensions and receive the following error >>> >>> java.lang.IllegalArgumentException: vector numDimensions must be <= >>> VectorValues.MAX_DIMENSIONS (=1024); got 12288 >>> at >>> org.apache.lucene.document.FieldType.setVectorDimensionsAndSimilarityFunction(FieldType.java:381) >>> ~[lucene-core-9.0.0.jar:9.0.0 0b18b3b965cedaf5eb129aa41243a44c83ca826d - >>> jpountz - 2021-12-01 14:23:49] >>> at >>> org.apache.lucene.document.KnnVectorField.createFieldType(KnnVectorField.java:69) >>> ~[lucene-core-9.0.0.jar:9.0.0 0b18b3b965cedaf5eb129aa41243a44c83ca826d - >>> jpountz - 2021-12-01 14:23:49] >>> >>> IIUC I can not increase programmatically the max vector size which is >>> set inside lucene/core/src/java/org/apache/lucene/index/VectorValues.java >>> >>> public static int MAX_DIMENSIONS = 1024; >>> >>> right? >>> >>> I guess I could rebuild Lucene with a greater size or what are the >>> possbilities to increase the max vector size? >>> >>> Thanks >>> >>> Michael >>> >>> >>> >> >
