Hi, Arnav Thanks for taking a look and for raising this. The intent is to keep centroid shards compatible with the existing IVF-PQ reader, while storing shared training state in the partition-level global index file to avoid duplication. So the PQ codebook would not be duplicated in every centroid shard，it would be stored once in the global routing index file.
Best, zhoufa Arnav Balyan <[email protected]> 于2026年8月31日周一 00:15写道： > Hi Zhoufa, > +1 thanks for the proposal, this would significantly help reduce index file > scanning for vector queries. > > I just had a small question, on section 2.4 > > Centroid data shards reuse the existing IVF-PQ IndexFile binary format: > each shard can still be opened by the standard VectorIndexReader > Does this mean the whole PQ codebook is serialized into every centroid > shard in addition to the routing file (potentially duplicating across > files) or is this reused from the routing file ? > > Thanks for the proposal! > > Warm Regards, > Arnav > > On Sun, Aug 30, 2026 at 7:40 AM Jingsong Li <[email protected]> > wrote: > > > This proposal applies to IVF in general, and supports IVF_PQ first. > > > > Looks good to me. > > > > +1 for this. > > > > Best, > > Jingsong > > > > On Fri, Aug 28, 2026 at 2:29 PM 周伐 <[email protected]> wrote: > > > > > > Thanks for raising this. > > > > > > My understanding is that this proposal applies to IVF in general. It > > > focuses on index building and query-time pruning for IVF-sharded > layouts, > > > so it should also be applicable to other IVF-based index types. > > > > > > We plan to support IVF_PQ first, and then extend it to other IVF > indexes > > > later. I'll also update the wording in PIP-47 accordingly. > > > > > > Best, > > > zhoufa > > > > > > Jingsong Li <[email protected]> 于2026年8月28日周五 13:29写道： > > > > > > > +1 for this. > > > > > > > > Is this only for PQ? Do you think it's possible to apply it to > IVF-RQ? > > > > > > > > Best, > > > > Jingsong > > > > > > > > On Fri, Aug 28, 2026 at 12:25 PM 周伐 <[email protected]> > wrote: > > > > > > > > > > Hi devs, > > > > > > > > > > I'd like to initiate a discussion on PIP-47: Centroid-Based > Sharding > > > > Design > > > > > for IVF_PQ [1]. > > > > > > > > > > Paimon already supports IVF_PQ for vector search, but query-time > > pruning > > > > > for centroid-based layouts is still limited. As a result, queries > may > > > > read > > > > > shards that are not actually relevant. This proposal introduces a > > > > > routing-based pruning mechanism to narrow the search to relevant > > shards > > > > and > > > > > improve query efficiency. > > > > > > > > > > Looking forward to hearing from you, thanks > > > > > > > > > > [1] > > > > > > > > > > > > https://cwiki.apache.org/confluence/spaces/PAIMON/pages/444334660/PIP-47+Centroid-Based+Sharding+Design+for+IVF_PQ > > > > > > > > > > Best, > > > > > zhoufa > > > > > > >
