Re: PIP-30: Optimize bitmap index format

[Jingsong Li]

Thanks zhonghang,

Overall design looks good to me!

I have some questions:

1. Are there any other bitmap designs that can be referenced, such as
the bitmap structure of starrocks' internal tables?
2. We should store the size of the roaring bitmap to make
deserialization faster. See
https://github.com/apache/incubator-paimon/pull/4765
3. Maybe we can have better names for dictionaries, maybe we can have
three parts: Index header + Index blocks + Bitmap blocks

Best,
Jingsong

On Fri, Jan 17, 2025 at 5:16 PM Jingsong Li <jingsongl...@gmail.com> wrote:
>
> Hi zhonghang,
>
> Please grant public access to the document.
>
> Best,
> Jingsong
>
> On Fri, Jan 17, 2025 at 3:46 PM zhonghang <1649067...@qq.com.invalid> wrote:
> >
> > Hi, devs:
> > &nbsp; &nbsp; We found that the first version of the bitmap index had&nbsp;
> > some performance issues in high cardinality scenarios.&nbsp;
> > We have now made some optimizations and hope to discuss&nbsp;
> > them with you.&nbsp;The detailed design document is as follows [1].
> >
> >
> > [1]:
> > https://docs.google.com/document/d/11dJlGlSX3dwYKKrPN0DQ2XQTsx6d9wI6DTBIiiBwomM/edit?usp=sharing
> >
> >
> > Thanks
> > ZhonghangLiu.
> >
> >
> > zhonghang
> > 1649067...@qq.com
> >
> >
> >
> > &nbsp;