Hi Zhoufa, +1 thanks for the proposal, this would significantly help reduce index file scanning for vector queries.
I just had a small question, on section 2.4 > Centroid data shards reuse the existing IVF-PQ IndexFile binary format: each shard can still be opened by the standard VectorIndexReader Does this mean the whole PQ codebook is serialized into every centroid shard in addition to the routing file (potentially duplicating across files) or is this reused from the routing file ? Thanks for the proposal! Warm Regards, Arnav On Sun, Aug 30, 2026 at 7:40 AM Jingsong Li <[email protected]> wrote: > This proposal applies to IVF in general, and supports IVF_PQ first. > > Looks good to me. > > +1 for this. > > Best, > Jingsong > > On Fri, Aug 28, 2026 at 2:29 PM 周伐 <[email protected]> wrote: > > > > Thanks for raising this. > > > > My understanding is that this proposal applies to IVF in general. It > > focuses on index building and query-time pruning for IVF-sharded layouts, > > so it should also be applicable to other IVF-based index types. > > > > We plan to support IVF_PQ first, and then extend it to other IVF indexes > > later. I'll also update the wording in PIP-47 accordingly. > > > > Best, > > zhoufa > > > > Jingsong Li <[email protected]> 于2026年8月28日周五 13:29写道： > > > > > +1 for this. > > > > > > Is this only for PQ? Do you think it's possible to apply it to IVF-RQ? > > > > > > Best, > > > Jingsong > > > > > > On Fri, Aug 28, 2026 at 12:25 PM 周伐 <[email protected]> wrote: > > > > > > > > Hi devs, > > > > > > > > I'd like to initiate a discussion on PIP-47: Centroid-Based Sharding > > > Design > > > > for IVF_PQ [1]. > > > > > > > > Paimon already supports IVF_PQ for vector search, but query-time > pruning > > > > for centroid-based layouts is still limited. As a result, queries may > > > read > > > > shards that are not actually relevant. This proposal introduces a > > > > routing-based pruning mechanism to narrow the search to relevant > shards > > > and > > > > improve query efficiency. > > > > > > > > Looking forward to hearing from you, thanks > > > > > > > > [1] > > > > > > > > https://cwiki.apache.org/confluence/spaces/PAIMON/pages/444334660/PIP-47+Centroid-Based+Sharding+Design+for+IVF_PQ > > > > > > > > Best, > > > > zhoufa > > > >
