+1 for this PIP, this is the core capability of Paimon for vector computation.
Best, FangYong On Mon, Aug 31, 2026 at 2:08 PM Arnav Balyan <[email protected]> wrote: > `Hi Zhoufa, > Thanks for the clarification, that makes sense! > > Warm Regards, > Arnav > > On Mon, Aug 31, 2026 at 11:35 AM 周伐 <[email protected]> wrote: > > > Hi, Arnav > > > > Thanks for taking a look and for raising this. > > The intent is to keep centroid shards compatible with the existing IVF-PQ > > reader, while storing shared training state in the partition-level global > > index file to avoid duplication. > > So the PQ codebook would not be duplicated in every centroid shard，it > would > > be stored once in the global routing index file. > > > > Best, > > zhoufa > > > > Arnav Balyan <[email protected]> 于2026年8月31日周一 00:15写道： > > > > > Hi Zhoufa, > > > +1 thanks for the proposal, this would significantly help reduce index > > file > > > scanning for vector queries. > > > > > > I just had a small question, on section 2.4 > > > > Centroid data shards reuse the existing IVF-PQ IndexFile binary > format: > > > each shard can still be opened by the standard VectorIndexReader > > > Does this mean the whole PQ codebook is serialized into every centroid > > > shard in addition to the routing file (potentially duplicating across > > > files) or is this reused from the routing file ? > > > > > > Thanks for the proposal! > > > > > > Warm Regards, > > > Arnav > > > > > > On Sun, Aug 30, 2026 at 7:40 AM Jingsong Li <[email protected]> > > > wrote: > > > > > > > This proposal applies to IVF in general, and supports IVF_PQ first. > > > > > > > > Looks good to me. > > > > > > > > +1 for this. > > > > > > > > Best, > > > > Jingsong > > > > > > > > On Fri, Aug 28, 2026 at 2:29 PM 周伐 <[email protected]> wrote: > > > > > > > > > > Thanks for raising this. > > > > > > > > > > My understanding is that this proposal applies to IVF in general. > It > > > > > focuses on index building and query-time pruning for IVF-sharded > > > layouts, > > > > > so it should also be applicable to other IVF-based index types. > > > > > > > > > > We plan to support IVF_PQ first, and then extend it to other IVF > > > indexes > > > > > later. I'll also update the wording in PIP-47 accordingly. > > > > > > > > > > Best, > > > > > zhoufa > > > > > > > > > > Jingsong Li <[email protected]> 于2026年8月28日周五 13:29写道： > > > > > > > > > > > +1 for this. > > > > > > > > > > > > Is this only for PQ? Do you think it's possible to apply it to > > > IVF-RQ? > > > > > > > > > > > > Best, > > > > > > Jingsong > > > > > > > > > > > > On Fri, Aug 28, 2026 at 12:25 PM 周伐 <[email protected]> > > > wrote: > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > I'd like to initiate a discussion on PIP-47: Centroid-Based > > > Sharding > > > > > > Design > > > > > > > for IVF_PQ [1]. > > > > > > > > > > > > > > Paimon already supports IVF_PQ for vector search, but > query-time > > > > pruning > > > > > > > for centroid-based layouts is still limited. As a result, > queries > > > may > > > > > > read > > > > > > > shards that are not actually relevant. This proposal > introduces a > > > > > > > routing-based pruning mechanism to narrow the search to > relevant > > > > shards > > > > > > and > > > > > > > improve query efficiency. > > > > > > > > > > > > > > Looking forward to hearing from you, thanks > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/spaces/PAIMON/pages/444334660/PIP-47+Centroid-Based+Sharding+Design+for+IVF_PQ > > > > > > > > > > > > > > Best, > > > > > > > zhoufa > > > > > > > > > > > > > > > >
