I'm also in favor of raising this limit. We do see some
datasets with higher than 1024 dims. I also think we
need to keep a limit. For example we currently need to
keep all the vectors in RAM while indexing and we want
to be able to support reasonable numbers of vectors in
an index segment. Also we don't know what innovations
might come down the road. Maybe someday we want to do
product quantization and enforce that (k, m) both fit in
a byte -- we wouldn't be able to do that if a vector's
dimension were to exceed 32K.
On Fri, Mar 31, 2023 at 11:57 AM Alessandro Benedetti
<a.benede...@sease.io> wrote:
I am also curious what would be the worst-case
scenario if we remove the constant at all (so
automatically the limit becomes the Java
Integer.MAX_VALUE).
i.e.
right now if you exceed the limit you get:
if (dimension > ByteVectorValues.MAX_DIMENSIONS) {
throw new IllegalArgumentException(
"cannot index vectors with dimension greater
than " + ByteVectorValues.MAX_DIMENSIONS);
}
in relation to:
These limits allow us to
better tune our data structures, prevent
overflows, help ensure we
have good test coverage, etc.
I agree 100% especially for typing stuff properly
and avoiding resource waste here and there, but I am
not entirely sure this is the case for the current
implementation i.e. do we have optimizations in
place that assume the max dimension to be 1024?
If I missed that (and I likely have), I of course
suggest the contribution should not just blindly
remove the limit, but do it appropriately.
I am not in favor of just doubling it as suggested
by some people, I would ideally prefer a solution
that remains there to a decent extent, rather than
having to modifying it anytime someone requires a
higher limit.
Cheers
--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
/Apache Lucene/Solr Committer/
/Apache Solr PMC Member/
e-mail: a.benede...@sease.io/
/
*Sease* - Information Retrieval Applied
Consulting | Training | Open Source
Website: Sease.io <http://sease.io/>
LinkedIn <https://linkedin.com/company/sease-ltd> |
Twitter <https://twitter.com/seaseltd> | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> |
Github <https://github.com/seaseltd>
On Fri, 31 Mar 2023 at 16:12, Michael Wechner
<michael.wech...@wyona.com> wrote:
OpenAI reduced their size to 1536 dimensions
https://openai.com/blog/new-and-improved-embedding-model
so 2048 would work :-)
but other services do provide also higher
dimensions with sometimes
slightly better accuracy
Thanks
Michael
Am 31.03.23 um 14:45 schrieb Adrien Grand:
> I'm supportive of bumping the limit on the
maximum dimension for
> vectors to something that is above what the
majority of users need,
> but I'd like to keep a limit. We have limits
for other things like the
> max number of docs per index, the max term
length, the max number of
> dimensions of points, etc. and there are a few
things that we don't
> have limits on that I wish we had limits on.
These limits allow us to
> better tune our data structures, prevent
overflows, help ensure we
> have good test coverage, etc.
>
> That said, these other limits we have in place
are quite high. E.g.
> the 32kB term limit, nobody would ever type a
32kB term in a text box.
> Likewise for the max of 8 dimensions for
points: a segment cannot
> possibly have 2 splits per dimension on
average if it doesn't have
> 512*2^(8*2)=34M docs, a sizable dataset
already, so more dimensions
> than 8 would likely defeat the point of
indexing. In contrast, our
> limit on the number of dimensions of vectors
seems to be under what
> some users would like, and while I understand
the performance argument
> against bumping the limit, it doesn't feel to
me like something that
> would be so bad that we need to prevent users
from using numbers of
> dimensions in the low thousands, e.g. top-k
KNN searches would still
> look at a very small subset of the full dataset.
>
> So overall, my vote would be to bump the limit
to 2048 as suggested by
> Mayya on the issue that you linked.
>
> On Fri, Mar 31, 2023 at 2:38 PM Michael Wechner
> <michael.wech...@wyona.com> wrote:
>> Thanks Alessandro for summarizing the
discussion below!
>>
>> I understand that there is no clear reasoning
re what is the best embedding size, whereas I
think heuristic approaches like described by the
following link can be helpful
>>
>>
https://datascience.stackexchange.com/questions/51404/word2vec-how-to-choose-the-embedding-size-parameter
>>
>> Having said this, we see various embedding
services providing higher dimensions than 1024,
like for example OpenAI, Cohere and Aleph Alpha.
>>
>> And it would be great if we could run
benchmarks without having to recompile Lucene
ourselves.
>>
>> Therefore I would to suggest to either
increase the limit or even better to remove the
limit and add a disclaimer, that people should
be aware of possible crashes etc.
>>
>> Thanks
>>
>> Michael
>>
>>
>>
>>
>> Am 31.03.23 um 11:43 schrieb Alessandro
Benedetti:
>>
>>
>> I've been monitoring various discussions on
Pull Requests about changing the max number of
dimensions allowed for Lucene HNSW vectors:
>>
>> https://github.com/apache/lucene/pull/12191
>>
>> https://github.com/apache/lucene/issues/11507
>>
>>
>> I would like to set up a discussion and
potentially a vote about this.
>>
>> I have seen some strong opposition from a few
people but a majority of favor in this direction.
>>
>>
>> Motivation
>>
>> We were discussing in the Solr slack channel
with Ishan Chattopadhyaya, Marcus Eagan, and
David Smiley about some neural search
integrations in Solr:
https://github.com/openai/chatgpt-retrieval-plugin
>>
>>
>> Proposal
>>
>> No hard limit at all.
>>
>> As for many other Lucene areas, users will be
allowed to push the system to the limit of their
resources and get terrible performances or
crashes if they want.
>>
>>
>> What we are NOT discussing
>>
>> - Quality and scalability of the HNSW algorithm
>>
>> - dimensionality reduction
>>
>> - strategies to fit in an arbitrary
self-imposed limit
>>
>>
>> Benefits
>>
>> - users can use the models they want to
generate vectors
>>
>> - removal of an arbitrary limit that blocks
some integrations
>>
>>
>> Cons
>>
>> - if you go for vectors with high
dimensions, there's no guarantee you get
acceptable performance for your use case
>>
>>
>>
>> I want to keep it simple, right now in many
Lucene areas, you can push the system to not
acceptable performance/ crashes.
>>
>> For example, we don't limit the number of
docs per index to an arbitrary maximum of N, you
push how many docs you like and if they are too
much for your system, you get terrible
performance/crashes/whatever.
>>
>>
>> Limits caused by primitive java types will
stay there behind the scene, and that's
acceptable, but I would prefer to not have
arbitrary hard-coded ones that may limit the
software usability and integration which is
extremely important for a library.
>>
>>
>> I strongly encourage people to add benefits
and cons, that I missed (I am sure I missed some
of them, but wanted to keep it simple)
>>
>>
>> Cheers
>>
>> --------------------------
>> Alessandro Benedetti
>> Director @ Sease Ltd.
>> Apache Lucene/Solr Committer
>> Apache Solr PMC Member
>>
>> e-mail: a.benede...@sease.io
>>
>>
>> Sease - Information Retrieval Applied
>> Consulting | Training | Open Source
>>
>> Website: Sease.io
>> LinkedIn | Twitter | Youtube | Github
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail:
dev-unsubscr...@lucene.apache.org
For additional commands, e-mail:
dev-h...@lucene.apache.org
