[
https://issues.apache.org/jira/browse/KYLIN-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825776#comment-15825776
]
Shaofeng SHI commented on KYLIN-2387:
-------------------------------------
If it is not easy to keep compitable with old version, we can shade the old
version when building binary package.
> A new BitmapCounter with better performance
> -------------------------------------------
>
> Key: KYLIN-2387
> URL: https://issues.apache.org/jira/browse/KYLIN-2387
> Project: Kylin
> Issue Type: Improvement
> Components: Metadata, Query Engine, Storage - HBase
> Affects Versions: v2.0.0
> Reporter: Dayue Gao
> Assignee: Dayue Gao
>
> We found the old BitmapCounter does not perform very well on very large
> bitmap. The inefficiency comes from
> * Poor serialize implementation: instead of serialize bitmap directly to
> ByteBuffer, it uses ByteArrayOutputStream as a temporal storage, which causes
> superfluous memory allocations
> * Poor peekLength implementation: the whole bitmap is deserialized in order
> to retrieve its serialized size
> * Extra deserialize cost: even if only cardinality info is needed to answer
> query, the whole bitmap is deserialize into MutableRoaringBitmap
> A new BitmapCounter is designed to solve these problems
> * It comes in tow flavors, mutable and immutable, which is based on
> Mutable/Immutable RoaringBitmap correspondingly
> * ImmutableBitmapCounter has lower deserialize cost, as it just maps to a
> copied buffer. So we always deserialize to ImmutableBitmapCounter at first,
> and convert it to MutableBitmapCounter only when necessary
> * peekLength is implemented using
> ImmutableRoaringBitmap.serializedSizeInBytes, which is very fast since only
> the header of roaring format is examined
> * It can directly serializes to ByteBuffer, no intermediate buffer is
> allocated
> * The wire format is the same as before
> ([RoaringFormatSpec|https://github.com/RoaringBitmap/RoaringFormatSpec/]).
> Therefore no cube rebuild is needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
