Hi Michael, let's create a Jira ticket to use a higher value(if you haven't already). I would be happy to consider the patch/or do it myself but after 10/03. Once the pull request is ready (including the Javadoc documentation that clearly states that if you go above X it's at your own risk), we'll involve also Michael Sokolov and the other committers familiar with this area of the code.
Cheers -------------------------- Alessandro Benedetti Apache Lucene/Solr PMC member and Committer Director, R&D Software Engineer, Search Consultant www.sease.io On Sat, 12 Feb 2022 at 22:53, Michael Wechner <michael.wech...@wyona.com> wrote: > Hi > > I just tried to test the OpenAI model "text-similarity-davinci-001" with > 12288 dimensions and receive the following error > > java.lang.IllegalArgumentException: vector numDimensions must be <= > VectorValues.MAX_DIMENSIONS (=1024); got 12288 > at > org.apache.lucene.document.FieldType.setVectorDimensionsAndSimilarityFunction(FieldType.java:381) > ~[lucene-core-9.0.0.jar:9.0.0 0b18b3b965cedaf5eb129aa41243a44c83ca826d - > jpountz - 2021-12-01 14:23:49] > at > org.apache.lucene.document.KnnVectorField.createFieldType(KnnVectorField.java:69) > ~[lucene-core-9.0.0.jar:9.0.0 0b18b3b965cedaf5eb129aa41243a44c83ca826d - > jpountz - 2021-12-01 14:23:49] > > IIUC I can not increase programmatically the max vector size which is set > inside lucene/core/src/java/org/apache/lucene/index/VectorValues.java > > public static int MAX_DIMENSIONS = 1024; > > right? > > I guess I could rebuild Lucene with a greater size or what are the > possbilities to increase the max vector size? > > Thanks > > Michael > > >
