Hi
I just tried to test the OpenAI model "text-similarity-davinci-001" with
12288 dimensions and receive the following error
java.lang.IllegalArgumentException: vector numDimensions must be <=
VectorValues.MAX_DIMENSIONS (=1024); got 12288
at
org.apache.lucene.document.FieldType.setVectorDimensionsAndSimilarityFunction(FieldType.java:381)
~[lucene-core-9.0.0.jar:9.0.0 0b18b3b965cedaf5eb129aa41243a44c83ca826d -
jpountz - 2021-12-01 14:23:49]
at
org.apache.lucene.document.KnnVectorField.createFieldType(KnnVectorField.java:69)
~[lucene-core-9.0.0.jar:9.0.0 0b18b3b965cedaf5eb129aa41243a44c83ca826d -
jpountz - 2021-12-01 14:23:49]
IIUC I can not increase programmatically the max vector size which is
set inside lucene/core/src/java/org/apache/lucene/index/VectorValues.java
public static int MAX_DIMENSIONS = 1024;
right?
I guess I could rebuild Lucene with a greater size or what are the
possbilities to increase the max vector size?
Thanks
Michael
