JingsongLi commented on PR #11249: URL: https://github.com/apache/paimon/pull/11249#issuecomment-7075930024
> Hi, @JingsongLi Thank you for this contribution! Centroid-based sharding is a valuable improvement for reducing query fan-out across index shards, especially under high QPS. > > I wonder would it make sense to make the implementation more backend-independent? For example, we could have a shared centroid build and query layer responsible for training the router, assigning vectors to centroids, and selecting shards at query time. Each shard’s writer and reader could then be created through a backend-specific factory, supporting Native IVF/DiskANN as well as Lumina. > > This would let other vector index backends reuse the routing layer while retaining their own indexing and search implementations. What do you think about this separation? Hi @lxy-9602 , Since this layer is essentially the same as IVF, it should be considered part of the index implementation. I believe the Lumina module could either implement its own IVF partitioning or, alternatively, rely directly on the `paimon-vindex` module. This layer is intended to provide flexibility to the indexing layer, with the current interfaces allowing the indexing layer to handle its own implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
