[
https://issues.apache.org/jira/browse/SOLR-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799923#comment-17799923
]
David Smiley commented on SOLR-17045:
-------------------------------------
Is this worth bringing to 9.4.1? AFAICT, a user can at least easily fix via
explicitly specifying the SchemaCodecFactory in their solrconfig.xml. :shrug:
> DenseVectorField w/ vectorDimension > 1024 no longer works by default
> ---------------------------------------------------------------------
>
> Key: SOLR-17045
> URL: https://issues.apache.org/jira/browse/SOLR-17045
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Affects Versions: 9.3
> Reporter: Chris M. Hostetter
> Assignee: Chris M. Hostetter
> Priority: Major
> Fix For: main (10.0), 9.5
>
>
> Starting with Solr 9.4, configuring a {{DenseVectorField}} w/
> {{vectorDimension}} > 1024 no longer works by default. There is no error on
> startup, but when indexing you'll get errors like...
> {noformat}
> 2> => org.apache.solr.common.SolrException: Exception writing
> document id 2 to the index; possible analysis error: Field [vector]vector's
> dimensions must be <= [1024]; got 1600
> 2> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:329)
> 2> org.apache.solr.common.SolrException: Exception writing document id 2 to
> the index; possible analysis error: Field [vector]vector's dimensions must be
> <= [1024]; got 1600
> 2> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:329)
> ~[solr-core-9.4.0.jar:9.4.0 71e101bb37497f730078d9afe1991b60d10bfe96 -
> stillalex - 2023-10-10 19:10:39]
> 2> at
> org.apache.solr.update.processor.RunUpdateProcessorFactory$RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76)
> ~[solr-core-9.4.0.jar:9.4.0 71e101bb37497f730078d9afe1991b60d10bfe96 -
> stillalex - 2023-10-10 19:10:39]
> ...
> 2> Caused by: java.lang.IllegalArgumentException: Field [vector]vector's
> dimensions must be <= [1024]; got 1600
> 2> at
> org.apache.lucene.index.IndexingChain.validateMaxVectorDimension(IndexingChain.java:843)
> ~[lucene-core-9.8.0.jar:9.8.0 d914b3722bd5b8ef31ccf7e8ddc638a87fd648db -
> 2023-09-21 21:57:47]
> ...
> {noformat}
> This is because Lucene 9.8 moved the dimension size limitation to the codec –
> and while Solr 9.4's {{SchemaCodecFactory}} was updated to implement a
> per-field {{SolrDelegatingKnnVectorsFormat}} that respected the
> {{vectorDimension}} configured for each {{DenseVectorField}} *the
> {{SchemaCodecFactory}} _is not implicitly used by default in Solr_* – nor is
> it explicitly configured in the {{_default}} configset.
> {panel:title=Known Work Around}
> Existing {{DenseVectorField}} who encounter this error when upgrading to Solr
> >= 9.4, or new user attempting to use {{DenseVectorField}} with
> {{vectorDimension}} > 1024, need to explicitly configure
> {{SchemaCodecFactory}} in the solrconfig.xml for each collection.
> {panel}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
