[
https://issues.apache.org/jira/browse/KYLIN-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529142#comment-16529142
]
Lemont edited comment on KYLIN-3416 at 7/1/18 4:15 PM:
-------------------------------------------------------
[~Shaofengshi]
I'm sorry i am the novice of kylin,I don't know how calcite explain the sql.I
will provide the message for you and try to fix it at a later time.
To make it easy ,I will use the KYLIN_SALES table.
The simple sql:
{quote}select count(distinct KYLIN_SALES.PRICE) from KYLIN_SALES group by
KYLIN_SALES.LEAF_CATEG_ID/100
{quote}
I think the case of this error is exactAggregation is true, cuboid id is 1
The exactAggregation case the bitmap only have the size of self but don't have
the bitmap value
The model definition is:
{quote}{ "uuid": "d4a843df-f135-4038-9d54-f3241ddc491c", "last_modified":
1530459283091, "version": "2.4.0.20500", "name": "kylin_sales_model_testnull",
"owner": "ADMIN", "is_draft": false, "description": "", "fact_table":
"DEFAULT.KYLIN_SALES", "lookups": [], "dimensions": [ \\{ "table":
"KYLIN_SALES", "columns": [ "TRANS_ID", "SELLER_ID", "BUYER_ID", "PART_DT",
"LEAF_CATEG_ID", "LSTG_FORMAT_NAME", "LSTG_SITE_ID", "OPS_USER_ID",
"OPS_REGION" ] }
],
"metrics": [
"KYLIN_SALES.PRICE",
"KYLIN_SALES.ITEM_COUNT"
],
"filter_condition": "",
"partition_desc": {
"partition_date_column": null,
"partition_time_column": null,
"partition_date_start": 1325376000000,
"partition_date_format": "yyyy-MM-dd",
"partition_time_format": "HH:mm:ss",
"partition_type": "APPEND",
"partition_condition_builder":
"org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
},
"capacity": "MEDIUM"
}
{quote}
The cube definition :
{quote}{ "uuid": "b88422cf-2cbf-4cd3-921b-73a927fd09aa", "last_modified":
1530459405127, "version": "2.4.0.20500", "name": "test_bitmap_null_cube",
"is_draft": false, "model_name": "kylin_sales_model_testnull", "description":
"", "null_string": null, "dimensions": [ \\{ "name": "LEAF_CATEG_ID", "table":
"KYLIN_SALES", "column": "LEAF_CATEG_ID", "derived": null }
],
"measures": [
{
"name": "_COUNT_",
"function": {
"expression": "COUNT",
"parameter":
{ "type": "constant", "value": "1" }
,
"returntype": "bigint"
}
},
{
"name": "PRICE",
"function": {
"expression": "COUNT_DISTINCT",
"parameter":
{ "type": "column", "value": "KYLIN_SALES.PRICE" }
,
"returntype": "bitmap"
}
}
],
"dictionaries": [
{
"column": "KYLIN_SALES.PRICE",
"builder": "org.apache.kylin.dict.GlobalDictionaryBuilder"
}
],
"rowkey": {
"rowkey_columns": [
{ "column": "KYLIN_SALES.LEAF_CATEG_ID", "encoding": "dict",
"encoding_version": 1, "isShardBy": false }
]
},
"hbase_mapping": {
"column_family": [
{
"name": "F1",
"columns": [
{ "qualifier": "M", "measure_refs": [ "_COUNT_" ] }
]
},
{
"name": "F2",
"columns": [
{ "qualifier": "M", "measure_refs": [ "PRICE" ] }
]
}
]
},
"aggregation_groups": [
{
"includes": [
"KYLIN_SALES.LEAF_CATEG_ID"
],
"select_rule":
{ "hierarchy_dims": [], "mandatory_dims": [], "joint_dims": [] }
}
],
"signature": "TVE29H3H4ElAm/4pEljkOQ==",
"notify_list": [],
"status_need_notify": [
"ERROR",
"DISCARDED",
"SUCCEED"
],
"partition_date_start": 0,
"partition_date_end": 3153600000000,
"auto_merge_time_ranges": [
604800000,
2419200000
],
"volatile_range": 0,
"retention_range": 0,
"engine_type": 2,
"storage_type": 2,
"override_kylin_properties": {},
"cuboid_black_list": [],
"parent_forward": 3,
"mandatory_dimension_set_list": [],
"snapshot_table_desc_list": []
}
{quote}
was (Author: lemontsr):
[~Shaofengshi]
I'm sorry i am the novice of kylin,I don't know how calcite explain the sql.I
will provide the message for you and try to fix it at a later time.
To make it easy ,I will use the KYLIN_SALES table.
The simple sql:
select count(distinct KYLIN_SALES.PRICE) from KYLIN_SALES group by
KYLIN_SALES.LEAF_CATEG_ID/100
I think the case of this error is exactAggregation is true, cuboid id is 1
the exactAggregation case the bitmap only have the size of self but don't have
the bitmap value
The model definition is:
{
"uuid": "d4a843df-f135-4038-9d54-f3241ddc491c",
"last_modified": 1530459283091,
"version": "2.4.0.20500",
"name": "kylin_sales_model_testnull",
"owner": "ADMIN",
"is_draft": false,
"description": "",
"fact_table": "DEFAULT.KYLIN_SALES",
"lookups": [],
"dimensions": [
\{
"table": "KYLIN_SALES",
"columns": [
"TRANS_ID",
"SELLER_ID",
"BUYER_ID",
"PART_DT",
"LEAF_CATEG_ID",
"LSTG_FORMAT_NAME",
"LSTG_SITE_ID",
"OPS_USER_ID",
"OPS_REGION"
]
}
],
"metrics": [
"KYLIN_SALES.PRICE",
"KYLIN_SALES.ITEM_COUNT"
],
"filter_condition": "",
"partition_desc": \{
"partition_date_column": null,
"partition_time_column": null,
"partition_date_start": 1325376000000,
"partition_date_format": "yyyy-MM-dd",
"partition_time_format": "HH:mm:ss",
"partition_type": "APPEND",
"partition_condition_builder":
"org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
},
"capacity": "MEDIUM"
}
The cube definition :
{
"uuid": "b88422cf-2cbf-4cd3-921b-73a927fd09aa",
"last_modified": 1530459405127,
"version": "2.4.0.20500",
"name": "test_bitmap_null_cube",
"is_draft": false,
"model_name": "kylin_sales_model_testnull",
"description": "",
"null_string": null,
"dimensions": [
\{
"name": "LEAF_CATEG_ID",
"table": "KYLIN_SALES",
"column": "LEAF_CATEG_ID",
"derived": null
}
],
"measures": [
\{
"name": "_COUNT_",
"function": {
"expression": "COUNT",
"parameter": {
"type": "constant",
"value": "1"
},
"returntype": "bigint"
}
},
\{
"name": "PRICE",
"function": {
"expression": "COUNT_DISTINCT",
"parameter": {
"type": "column",
"value": "KYLIN_SALES.PRICE"
},
"returntype": "bitmap"
}
}
],
"dictionaries": [
\{
"column": "KYLIN_SALES.PRICE",
"builder": "org.apache.kylin.dict.GlobalDictionaryBuilder"
}
],
"rowkey": \{
"rowkey_columns": [
{
"column": "KYLIN_SALES.LEAF_CATEG_ID",
"encoding": "dict",
"encoding_version": 1,
"isShardBy": false
}
]
},
"hbase_mapping": \{
"column_family": [
{
"name": "F1",
"columns": [
{
"qualifier": "M",
"measure_refs": [
"_COUNT_"
]
}
]
},
\{
"name": "F2",
"columns": [
{
"qualifier": "M",
"measure_refs": [
"PRICE"
]
}
]
}
]
},
"aggregation_groups": [
\{
"includes": [
"KYLIN_SALES.LEAF_CATEG_ID"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [],
"joint_dims": []
}
}
],
"signature": "TVE29H3H4ElAm/4pEljkOQ==",
"notify_list": [],
"status_need_notify": [
"ERROR",
"DISCARDED",
"SUCCEED"
],
"partition_date_start": 0,
"partition_date_end": 3153600000000,
"auto_merge_time_ranges": [
604800000,
2419200000
],
"volatile_range": 0,
"retention_range": 0,
"engine_type": 2,
"storage_type": 2,
"override_kylin_properties": {},
"cuboid_black_list": [],
"parent_forward": 3,
"mandatory_dimension_set_list": [],
"snapshot_table_desc_list": []
}
> Kylin bitmap null pointer exception
> -----------------------------------
>
> Key: KYLIN-3416
> URL: https://issues.apache.org/jira/browse/KYLIN-3416
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: v2.3.1
> Reporter: Lemont
> Priority: Blocker
>
> Hi,team:
> Ithink there is a conflict between dimensional aggregation and count
> distinct.
> For example:
> select
> (1524931200 - biz_time)/(30*86400),
> count(DISTINCT id) id
> from test
> where pt ='20180621'
> group by (1524931200 - biz_time)/(30*86400)
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.measure.bitmap.RoaringBitmapCounter.getMutableBitmap(RoaringBitmapCounter.java:58)
> at
> org.apache.kylin.measure.bitmap.RoaringBitmapCounter.orWith(RoaringBitmapCounter.java:72)
> at
> org.apache.kylin.measure.bitmap.BitmapAggregator.aggregate(BitmapAggregator.java:43)
> at
> org.apache.kylin.measure.bitmap.BitmapDistinctCountAggFunc.add(BitmapDistinctCountAggFunc.java:31)
>
> The problem cased by GTCubeStorageQueryBase.isNeedStorageAggregation
> There are three dimension int the cube,the sql use cuboid 5 and the
> cuboid.getColumns is
> pt,bize_time.
> The groupD is bize_time and singleValueD is pt.
> So isNeedStorageAggregation return false.
> But in fact this sql need storage agregation because the group is (1524931200
> - biz_time)/(30*86400) not only biz_time
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
