[ https://issues.apache.org/jira/browse/KYLIN-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074933#comment-15074933 ]
liyang commented on KYLIN-1186: ------------------------------- Some quick comments on this patch. Getting early feedback like this is a great practice. :-) - No need to change FunctionRule. Validation logic should go inside MeasureType.validate(). FunctionRule will call it. - Test case should cover both HLLC and bitmap. Maybe create a new measure? Don't drop HLLC. - Need unit tests on BitmapCounter, BitmapAggregator, BitmapSerializer. Troubleshooting integration test is very difficult. Hope unit tests can discover most issues. - BitmapSerializer.maxLength() must return the maximum possible length of a serialized counter. If the max is unclear, we can generate large random numbers and test out an experience max. > Support precise Count Distinct using bitmap > ------------------------------------------- > > Key: KYLIN-1186 > URL: https://issues.apache.org/jira/browse/KYLIN-1186 > Project: Kylin > Issue Type: Improvement > Components: Job Engine > Affects Versions: v1.1 > Reporter: Yerui Sun > Assignee: Yerui Sun > Fix For: v2.0, v1.3 > > Attachments: KYLIN-1186-1.x-staging.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > For now, kylin only support non-precise count distinct by hyperloglog. > In our production scenario, there're strongly requirements for precise count > distinct, mainly for the column of type int or bigint, such as user-id, > product-id, etc. > Implementing of precise count distinct for all types is difficult and not > efficiency. However, only supporting int or bigint make this much easier. The > values can be projected into a bitmap, which is easy to be compressed and > stored, and easy to count. > I've created a POC based on RoaringBitmap, proving that worked. There's some > more work to be done: > * RoaringBitmap only support int, there need a solution to support bigint; > * Add a new measure and codec, like HyperLogLogPlusCounter, make it easy to > use; > * Add new measure on web ui, and check that whether the column type is int > or bigint; -- This message was sent by Atlassian JIRA (v6.3.4#6332)