JingsongLi commented on PR #11249:
URL: https://github.com/apache/paimon/pull/11249#issuecomment-7075930024

   > Hi, @JingsongLi Thank you for this contribution! Centroid-based sharding 
is a valuable improvement for reducing query fan-out across index shards, 
especially under high QPS.
   > 
   > I wonder would it make sense to make the implementation more 
backend-independent? For example, we could have a shared centroid build and 
query layer responsible for training the router, assigning vectors to 
centroids, and selecting shards at query time. Each shard’s writer and reader 
could then be created through a backend-specific factory, supporting Native 
IVF/DiskANN as well as Lumina.
   > 
   > This would let other vector index backends reuse the routing layer while 
retaining their own indexing and search implementations. What do you think 
about this separation?
   
   Hi @lxy-9602 , Since this layer is essentially the same as IVF, it should be 
considered part of the index implementation. I believe the Lumina module could 
either implement its own IVF partitioning or, alternatively, rely directly on 
the `paimon-vindex` module. This layer is intended to provide flexibility to 
the indexing layer, with the current interfaces allowing the indexing layer to 
handle its own implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to