I would suggest building Lucene from source and adding your own similarity function to VectorSimilarity. That is the proper extension point for similarity functions. If you find there is some substantial benefit, it wouldn't be a big lift to add something like that. However I'm dubious about the likely benefit; just because scipy supports lots of functions doesn't mean you will get substantially better results with L3 metric vs L2 metric or so. I think you'd probably find this community receptive to a metric that *doesn't lose* accuracy and provides a more efficient computation -- maybe L1 would do that?
On Sat, Jan 14, 2023 at 6:04 PM Michael Wechner <michael.wech...@wyona.com> wrote: > > Hi Adrien > > Thanks for your feedback! Whereas I am not sure I fully understand what > you mean > > At the moment I am using something like: > > float[] vector = ...; > FieldType vectorFieldType = KnnVectorField.createFieldType(vector.length, > VectorSimilarityFunction.COSINE); > KnnVectorField vectorField =new KnnVectorField("vector_field", vector, > vectorFieldType); > doc.add(vectorField); > > Could you give me some sample code what you mean with "custom KNN > vectors format"? > > Thanks > > Michael > > Am 14.01.23 um 22:14 schrieb Adrien Grand: > > Hi Michael, > > > > You could create a custom KNN vectors format that ignores the vector > > similarity configured on the field and uses its own. > > > > Le sam. 14 janv. 2023, 21:33, Michael Wechner<michael.wech...@wyona.com> a > > écrit : > > > >> Hi > >> > >> IIUC Lucene currently supports > >> > >> VectorSimilarityFunction.COSINE > >> VectorSimilarityFunction.DOT_PRODUCT > >> VectorSimilarityFunction.EUCLIDEAN > >> > >> whereas some embedding models have been trained with other metrics. > >> Also see > >> > >> https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cdist.html > >> > >> How can I best implement another metric? > >> > >> Thanks > >> > >> Michael > >> > >> > >> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail:java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail:java-user-h...@lucene.apache.org > >> > >> --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org