Hi Yuxiang, I think so; Kylin uses RoaringBitmap, which is a compressed bitmap. For more information, you can check its home page[1].
[1] https://github.com/RoaringBitmap/RoaringBitmap 2017-08-31 13:51 GMT+08:00 Yuxiang Mai <[email protected]>: > Shaofeng: > Thanks for your reply. Is the bitmap size related to the Cardinality of > the count distnct column? > more Cardinality results bigger bitmap size? > > Thanks > > BR//MYX > > On Thu, Aug 31, 2017 at 11:18 AM, ShaoFeng Shi <[email protected]> > wrote: > >> The global dictionary is stored in kylin metadata table. So you can >> backup the metadata to local disk, then you can find and check the >> dictionary size there. >> >> The bitmap, as a measure, is stored in Cube (in HBase). The HBase table's >> size can roughly reflect its size, but that is the total size of all >> dimensions/measures. If you put the count distinct measure in a separate >> column family, then you can size that CF's HFile (this is just my guess, >> not verified). >> >> 2017-08-31 10:19 GMT+08:00 Yuxiang Mai <[email protected]>: >> >>> Hi, all >>> >>> We are now testing Kylin count distinct function. It works good. But we >>> have some questions about the global dictionary & bitmap: >>> >>> We need to do count distinct across cube segment on a bigint column, so >>> we use Global dictionary Builder. We found that the cube segment storage >>> size approximately double after adding count_distinct in measures. It's >>> stated that there are a global bitmap & dictionary to do the distinct. >>> I wonder how to check the size of the global bitmap & dictionary? >>> Because in Kylin web UI, we only can check the size of the cube segment. >>> And where is the global bitmap & dictionary storage location? A cube >>> segment is stored as a table in HBASE, but I can't find any Hbase tables >>> related to the global bitmap & dictionary. >>> >>> Thanks in advance for your help. >>> >>> >>> >>> -- >>> Yuxiang Mai >>> >>> >> >> >> -- >> Best regards, >> >> Shaofeng Shi 史少锋 >> >> > > > -- > Yuxiang Mai > Sun Yat-Sen Unitversity > State Key Lab of Optoelectronic > Materials and Technologies > -- Best regards, Shaofeng Shi 史少锋
