[
https://issues.apache.org/jira/browse/LUCENE-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565468#comment-17565468
]
Michael Sokolov commented on LUCENE-10471:
------------------------------------------
We should not be imposing an arbitrary limit that prevents people with CNNs
(image-processing models) from using this feature. It makes sense to me to
increase the limit to the point where we would see actual bugs/failures, or
where the large numbers might prevent us from making some future optimization,
rather than trying to determine where the performance stops being acceptable -
that's a question for users to decide for themselves. Of course we don't know
where that place is that we might want to optimize in the future (Rob and I
discussed an idea using all-integer math that would suffer from overflow, but
still we should not just allow MAX_INT dimensions I think? To me a limit like
16K makes sense – well beyond any stated use case, but not effectively infinite?
> Increase the number of dims for KNN vectors to 2048
> ---------------------------------------------------
>
> Key: LUCENE-10471
> URL: https://issues.apache.org/jira/browse/LUCENE-10471
> Project: Lucene - Core
> Issue Type: Wish
> Reporter: Mayya Sharipova
> Priority: Trivial
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The current maximum allowed number of dimensions is equal to 1024. But we see
> in practice a couple well-known models that produce vectors with > 1024
> dimensions (e.g
> [mobilenet_v2|https://tfhub.dev/google/imagenet/mobilenet_v2_035_224/feature_vector/1]
> uses 1280d vectors, OpenAI / GPT-3 Babbage uses 2048d vectors). Increasing
> max dims to `2048` will satisfy these use cases.
> I am wondering if anybody has strong objections against this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
