[ 
https://issues.apache.org/jira/browse/KYLIN-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dayue Gao updated KYLIN-2387:
-----------------------------
    Description: 
We found the old BitmapCounter does not perform very well on very large bitmap. 
The inefficiency comes from
* Poor serialize implementation: instead of serialize bitmap directly to 
ByteBuffer, it uses ByteArrayOutputStream as a temporal storage, which causes 
superfluous memory allocations
* Poor peekLength implementation: the whole bitmap is deserialized in order to 
retrieve its serialized size
* Extra deserialize cost: even if only cardinality info is needed to answer 
query, the whole bitmap is deserialize into MutableRoaringBitmap

A new BitmapCounter is designed to solve these problems
* It comes in tow flavors, mutable and immutable, which is based on 
Mutable/Immutable RoaringBitmap correspondingly
* ImmutableBitmapCounter has lower deserialize cost, as it just maps to a 
copied buffer. So we always deserialize to ImmutableBitmapCounter at first, and 
convert it to MutableBitmapCounter only when necessary
* peekLength is implemented using ImmutableRoaringBitmap.serializedSizeInBytes, 
which is very fast since only the header of roaring format is examined
* It can directly serializes to ByteBuffer, no intermediate buffer is allocated
* The wire format is the same as before 
([RoaringFormatSpec|https://github.com/RoaringBitmap/RoaringFormatSpec/]). 
Therefore no cube rebuild is needed

  was:
We found the old BitmapCounter does not perform very well on very large bitmap. 
The inefficiency comes from
* Poor serialize implementation: instead of serialize bitmap directly to 
ByteBuffer, it uses ByteArrayOutputStream as a temporal storage, which causes 
superfluous memory allocations
* Poor peekLength implementation: the whole bitmap is deserialized in order to 
retrieve its serialized size
* Extra deserialize cost: even if only cardinality info is needed to answer 
query, the whole bitmap is deserialize into MutableRoaringBitmap

A new BitmapCounter is designed to solve these problems
* It comes in tow flavors, mutable and immutable, which is based on 
Mutable/Immutable RoaringBitmap correspondingly
* ImmutableBitmapCounter has lower deserialize cost, as it just maps to a 
copied buffer. So we always deserialize to ImmutableBitmapCounter at first, and 
convert it to MutableBitmapCounter only when necessary
* peekLength is implemented using ImmutableRoaringBitmap.serializedSizeInBytes, 
which is very fast since only the header of roaring format is examined
* It can directly serializes to ByteBuffer, no intermediate buffer is allocated
* The wire format is the same as before, see 
[RoaringFormatSpec|https://github.com/RoaringBitmap/RoaringFormatSpec/]. 
Therefore no cube rebuild is needed


> A new BitmapCounter with better performance
> -------------------------------------------
>
>                 Key: KYLIN-2387
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2387
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Metadata, Query Engine, Storage - HBase
>    Affects Versions: v2.0.0
>            Reporter: Dayue Gao
>            Assignee: Dayue Gao
>
> We found the old BitmapCounter does not perform very well on very large 
> bitmap. The inefficiency comes from
> * Poor serialize implementation: instead of serialize bitmap directly to 
> ByteBuffer, it uses ByteArrayOutputStream as a temporal storage, which causes 
> superfluous memory allocations
> * Poor peekLength implementation: the whole bitmap is deserialized in order 
> to retrieve its serialized size
> * Extra deserialize cost: even if only cardinality info is needed to answer 
> query, the whole bitmap is deserialize into MutableRoaringBitmap
> A new BitmapCounter is designed to solve these problems
> * It comes in tow flavors, mutable and immutable, which is based on 
> Mutable/Immutable RoaringBitmap correspondingly
> * ImmutableBitmapCounter has lower deserialize cost, as it just maps to a 
> copied buffer. So we always deserialize to ImmutableBitmapCounter at first, 
> and convert it to MutableBitmapCounter only when necessary
> * peekLength is implemented using 
> ImmutableRoaringBitmap.serializedSizeInBytes, which is very fast since only 
> the header of roaring format is examined
> * It can directly serializes to ByteBuffer, no intermediate buffer is 
> allocated
> * The wire format is the same as before 
> ([RoaringFormatSpec|https://github.com/RoaringBitmap/RoaringFormatSpec/]). 
> Therefore no cube rebuild is needed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to