[
https://issues.apache.org/jira/browse/SOLR-17892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18024203#comment-18024203
]
Chris M. Hostetter commented on SOLR-17892:
-------------------------------------------
FYI: The addition of this module to main now means that main can not be merged
into the SOLR-17917 branch, and upgrading main to to Lucene 10.3 is blocked.
This is because even if the code in {{solr/modules/cuvs}} is updated to use
{{Lucene103Codec}} (to allow compilation) tests fail immediately because the
SPI loader can't initialize
{{com.nvidia.cuvs.lucene.Lucene101AcceleratedHNSWCodec}} ([here i
believe?|https://github.com/rapidsai/cuvs-lucene/blob/branch-25.10/src/main/java/com/nvidia/cuvs/lucene/Lucene101AcceleratedHNSWCodec.java])
.... IIUC this is because {{Lucene101AcceleratedHNSWCodec}} has been compiled
to depend on {{{}org.apache.lucene.codecs.lucene101.Lucene101Codec{}}}, but in
10.3 this code has moved to
{{org.apache.lucene.backward_codecs.lucene101.Lucene101Codec}}
I won't pretend to understand how Lucene's codec management (in terms of moving
codecs to the {{backward_codecs}} ) is expected to play nicely with third-party
codecs that want to extend {{FilterCodec}} around a "core" codec, but
unless/until this {{solr/modules/cuvs}} can play nicely with being able to
upgrade Lucene, we should consider reverting it.
> Add support for cuVS-Lucene as a pluggable codec in Solr
> --------------------------------------------------------
>
> Key: SOLR-17892
> URL: https://issues.apache.org/jira/browse/SOLR-17892
> Project: Solr
> Issue Type: New Feature
> Components: vector-search
> Reporter: Vivek Narang
> Assignee: Ishan Chattopadhyaya
> Priority: Major
> Labels: pull-request-available
> Fix For: main (10.0)
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This issue proposes adding *cuVS-Lucene* as a pluggable codec in Solr to
> enable GPU-accelerated vector indexing.
> *Background*
> * [cuVS-Lucene|https://github.com/rapidsai/cuvs-lucene] is a new NVIDIA
> project that integrates GPU acceleration into Lucene for vector search.
> * It supports building HNSW graphs on GPUs via the state-of-the-art *CAGRA*
> algorithm.
> * The first official release of cuVS-Lucene is planned for {*}early
> October{*}.
> * At present, artifacts are not yet published on Maven Central. For early
> adoption, development, and testing, SearchScale is publishing temporary
> artifacts to its Maven repository. Once released, official artifacts will be
> available on Maven Central, and the PR will be updated to use the released
> artifact accordingly.
>
> *Usage in Solr*
> This change introduces the ability to configure Solr to use the
> {{*Lucene101AcceleratedHNSWVectorsFormat*}} provided by the cuVS-Lucene.
> Users can opt to use the GPU-accelerated indexing by selecting the
> {{CuVSCodec}} codec, while retaining compatibility with existing CPU-based
> codecs. Documentation will include steps on how to enable and configure this
> format within Solr. Prerequisites includes a
>
> *Testing Strategy*
> * NVIDIA is dedicating GPU resources for testing cuVS-Lucene directly.
> * In Solr’s test framework:
> ** Tests for cuVS-Lucene will be skipped on non-GPU machines.
> ** On GPU-enabled machines, the tests will run fully, validating integration.
> * We are also exploring the possibility of contributing GPU resources to
> Apache Solr’s CI infrastructure for continuous GPU test coverage (to be
> discussed further with the community).
>
> *Benchmarks*
> We will publish benchmark results shortly to demonstrate the performance
> improvements of GPU-accelerated HNSW graph construction compared to CPU-only
> implementations.
> *Motivation for Solr 10*
> Including cuVS-Lucene support in *Solr 10* would highlight Solr’s adoption of
> first-class GPU acceleration, delivering significant performance improvements
> for vector search and positioning Solr as a leader in large-scale AI/ML
> search workloads.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
