Re: PIP-30: Optimize bitmap index format

[Jingsong Li]

Hi zhonghang,

> The bitmap size can be calculated as follows: bitmap size = next offset - 
> offset

Can you use this size to deserialize bitmap from bytes in your pull request [1]?

[1] https://github.com/apache/paimon/pull/4956

Best,
Jingsong

On Thu, Jan 23, 2025 at 5:40 PM zhonghang <1649067...@qq.com.invalid> wrote:
>
> Hi, JiaLiang.
>
>
> This is the configuration in the code:&nbsp;
> public static final int ROW_COUNT = 1000000;
> public static final int APPROX_CARDINALITY = 100000;
>
>
> Best,
> Zhonghang.
>
>
> zhonghang
> 1649067...@qq.com
>
>
>
> &nbsp;
>
>
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:                                                                          
>                                               "jialiang tan"                  
>                                                                   
> <tanjialiang1...@gmail.com&gt;;
> 发送时间:&nbsp;2025年1月23日(星期四) 下午5:28
> 收件人:&nbsp;"zhonghang"<1649067...@qq.com&gt;;
>
> 主题:&nbsp;Re: PIP-30: Optimize bitmap index format
>
>
>
> Very thanks to Zhonghang privoding the benchmark!
>
> Seems to introduce the secondary index is necessary.
>
>
> By the way, how many cardinary in this benchmark?
>
>
> Best,
> JiaLiang.
>
>
> zhonghang <1649067...@qq.com&gt; 于2025年1月23日周四 16:46写道：
>
> Hi&nbsp;JiaLiang,
>
>
> I added a benchmark for bitmap index queries without changing the current 
> index serialization format.&nbsp;
> There are only two variables in these test cases, namely bitmap index version 
> v1/v2,&nbsp;
> and whether BufferedInputStream is used.&nbsp;
>
>
> Here are the test results:
>
>
> bitmap-index-query-benchmark:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Best/Avg Time(ms)&nbsp; 
> &nbsp; Row Rate(K/s)&nbsp; &nbsp; &nbsp; Per Row(ns)&nbsp; &nbsp;Relative
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
> OPERATORTEST_bitmap-index-query-benchmark_format-v1&nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; 843 /&nbsp; 852&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
> &nbsp; 0.1&nbsp; &nbsp; &nbsp; &nbsp; 8428130.9&nbsp; &nbsp; &nbsp; &nbsp;1.0X
> OPERATORTEST_bitmap-index-query-benchmark_format-v1-buffered-input&nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 17 /&nbsp; 
> &nbsp;17&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 6.0&nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp;166435.5&nbsp; &nbsp; &nbsp; 50.6X
> OPERATORTEST_bitmap-index-query-benchmark_format-v2&nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; 8 /&nbsp; &nbsp; 8&nbsp; &nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp;13.2&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 75760.8&nbsp; 
> &nbsp; &nbsp;111.2X
> OPERATORTEST_bitmap-index-query-benchmark_format-v2-buffered-input&nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0 
> /&nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 267.5&nbsp; &nbsp; 
> &nbsp; &nbsp; &nbsp; &nbsp;3737.8&nbsp; &nbsp; 2254.9X
>
>
> Best,
> Zhonghang.
>
>
>
> zhonghang
> 1649067...@qq.com
>
>
>
> &nbsp;