lxy-9602 commented on PR #11249: URL: https://github.com/apache/paimon/pull/11249#issuecomment-7074545677
Hi, @JingsongLi Thank you for this contribution! Centroid-based sharding is a valuable improvement for reducing query fan-out across index shards, especially under high QPS. I wonder would it make sense to make the implementation more backend-independent? For example, we could have a shared centroid build and query layer responsible for training the router, assigning vectors to centroids, and selecting shards at query time. Each shard’s writer and reader could then be created through a backend-specific factory, supporting Native IVF/DiskANN as well as Lumina. This would let other vector index backends reuse the routing layer while retaining their own indexing and search implementations. What do you think about this separation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
