[jira] [Created] (KYLIN-2850) Throw clear error message when there're more than 63 columns in rowkey
Roger Shi created KYLIN-2850: Summary: Throw clear error message when there're more than 63 columns in rowkey Key: KYLIN-2850 URL: https://issues.apache.org/jira/browse/KYLIN-2850 Project: Kylin Issue Type: Improvement Reporter: Roger Shi Priority: Minor Now the error message is not that clear when there're more than 63 columns in rowkey. Provide clearer error message in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged
[
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
翟玉勇 updated KYLIN-2849:
---
Description:
cube duplicate segments。
cannot be deleted and data cannot be refreshed and merged
{code}
try
curl -X DELETE
"http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300;
-H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type:
application/json;charset=UTF-8"
Cannot delete segment '2017082200_2017082300' as it is neither the
first nor the last
segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
Cannot delete segment '2017082200_2017082300' as it is neither the
first nor the last segment
{code}
暂时解决办法：
{code}
public CubeInstance deleteSegment(CubeInstance cube, String segmentName) throws
IOException {
if (!segmentName.equals(cube.getSegments().get(0).getName()) &&
!segmentName.equals(cube.getSegments().get(cube.getSegments().size() -
1).getName())) {
//throw new IllegalArgumentException("Cannot delete segment '" +
segmentName + "' as it is neither the first nor the last segment.");
}
CubeSegment toDelete = null;
for (CubeSegment seg : cube.getSegments()) {
if (seg.getName().equals(segmentName)) {
toDelete = seg;
}
}
if (toDelete == null) {
throw new IllegalArgumentException("Cannot find segment '" +
segmentName + "'");
}
if (toDelete.getStatus() != SegmentStatusEnum.READY) {
//throw new IllegalArgumentException("Cannot delete segment '" +
segmentName + "' as its status is not READY. Discard the on-going job for it.");
}
CubeUpdate update = new CubeUpdate(cube);
update.setToRemoveSegs(new CubeSegment[] { toDelete });
return CubeManager.getInstance(getConfig()).updateCube(update);
}
{code}
was:
cube duplicate segments。
cannot be deleted and data cannot be refreshed and merged
{code}
try
curl -X DELETE
"http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300;
-H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type:
application/json;charset=UTF-8"
Cannot delete segment '2017082200_2017082300' as it is neither the
first nor the last
segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
Cannot delete segment '2017082200_2017082300' as it is neither the
first nor the last segment
{code}
> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300;
> -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type:
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the
> first nor the last
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
> Cannot delete segment '2017082200_2017082300' as it is neither the
> first nor the last segment
> {code}
> 暂时解决办法：
> {code}
> public CubeInstance deleteSegment(CubeInstance cube, String segmentName)
> throws IOException {
> if (!segmentName.equals(cube.getSegments().get(0).getName()) &&
> !segmentName.equals(cube.getSegments().get(cube.getSegments().size() -
> 1).getName())) {
> //throw new IllegalArgumentException("Cannot delete segment '" +
> segmentName + "' as it is neither the first nor the last segment.");
> }
> CubeSegment toDelete = null;
> for (CubeSegment seg : cube.getSegments()) {
> if (seg.getName().equals(segmentName)) {
> toDelete = seg;
> }
> }
> if (toDelete == null) {
> throw new IllegalArgumentException("Cannot find segment '" +
> segmentName + "'");
> }
> if (toDelete.getStatus() != SegmentStatusEnum.READY) {
> //throw new IllegalArgumentException("Cannot delete segment '" +
> segmentName + "' as its status is not READY. Discard the on-going job for
> it.");
> }
> CubeUpdate update = new CubeUpdate(cube);
> update.setToRemoveSegs(new CubeSegment[] { toDelete });
> return
[jira] [Updated] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged
[
https://issues.apache.org/jira/browse/KYLIN-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
翟玉勇 updated KYLIN-2849:
---
Description:
cube duplicate segments。
cannot be deleted and data cannot be refreshed and merged
{code}
try
curl -X DELETE
"http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300;
-H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type:
application/json;charset=UTF-8"
Cannot delete segment '2017082200_2017082300' as it is neither the
first nor the last
segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
Cannot delete segment '2017082200_2017082300' as it is neither the
first nor the last segment
{code}
was:
cube duplicate segments。
cannot be deleted and data cannot be refreshed and merged
try
curl -X DELETE
"http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300;
-H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type:
application/json;charset=UTF-8"
{code}
Cannot delete segment '2017082200_2017082300' as it is neither the
first nor the last
segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
Cannot delete segment '2017082200_2017082300' as it is neither the
first nor the last segment
{code}
> duplicate segment，cannot be deleted and data cannot be refreshed and merged
> ---
>
> Key: KYLIN-2849
> URL: https://issues.apache.org/jira/browse/KYLIN-2849
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, REST Service
>Affects Versions: v2.0.0
> Environment: hadoop：hadoop-2.6.0-cdh5.8.2
> hive ：2.1.0
> hbase：0.98
>Reporter: 翟玉勇
>Assignee: Dong Li
> Attachments: kylin-1.png, kylin-2.png
>
>
> cube duplicate segments。
> cannot be deleted and data cannot be refreshed and merged
> {code}
> try
> curl -X DELETE
> "http://127.0.0.1:7070/kylin/api/cubes/Remain_Cube_2/segs/2017082200_2017082300;
> -H "Authorization: Basic QURNSU46S1lMSU4=" -H "Content-Type:
> application/json;charset=UTF-8"
> Cannot delete segment '2017082200_2017082300' as it is neither the
> first nor the last
> segment.","stacktrace":"org.apache.kylin.rest.exception.InternalErrorException:
> Cannot delete segment '2017082200_2017082300' as it is neither the
> first nor the last segment
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KYLIN-2849) duplicate segment，cannot be deleted and data cannot be refreshed and merged
翟玉勇 created KYLIN-2849: -- Summary: duplicate segment，cannot be deleted and data cannot be refreshed and merged Key: KYLIN-2849 URL: https://issues.apache.org/jira/browse/KYLIN-2849 Project: Kylin Issue Type: Bug Components: Job Engine, Metadata, REST Service Affects Versions: v2.0.0 Environment: hadoop：hadoop-2.6.0-cdh5.8.2 hive ：2.1.0 hbase：0.98 Reporter: 翟玉勇 Assignee: Dong Li Attachments: kylin-1.png, kylin-2.png cube duplicate segments。 cannot be deleted and data cannot be refreshed and merged -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2737) Change the dictionary building process to MR
[ https://issues.apache.org/jira/browse/KYLIN-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2737: -- Affects Version/s: (was: v2.0.0) v2.2.0 > Change the dictionary building process to MR > > > Key: KYLIN-2737 > URL: https://issues.apache.org/jira/browse/KYLIN-2737 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.2.0 >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2721) Introduce a new metrics framework based on dropwizard metrics
[ https://issues.apache.org/jira/browse/KYLIN-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2721: -- Affects Version/s: (was: v2.0.0) v2.2.0 > Introduce a new metrics framework based on dropwizard metrics > - > > Key: KYLIN-2721 > URL: https://issues.apache.org/jira/browse/KYLIN-2721 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.2.0 >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: Metrics Framework.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2800) All dictionaries should be built based on the flat hive table
[ https://issues.apache.org/jira/browse/KYLIN-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154666#comment-16154666 ] liyang commented on KYLIN-2800: --- Agree. Am working this patch. > All dictionaries should be built based on the flat hive table > - > > Key: KYLIN-2800 > URL: https://issues.apache.org/jira/browse/KYLIN-2800 > Project: Kylin > Issue Type: Bug >Reporter: zhengdong >Assignee: zhengdong > Attachments: > 0001-KYLIN-2800-All-dictionaries-should-be-built-based-on.patch > > > After KYLIN-2457, we still got wrong query result sometimes after a merging > job finished. > Finally, we realize the root cause is that we always use lookup table as > source data to build dictionaries for FK columns. > However, incremental lookup table doesn't mean sequential and incremental PK. > If a new record inserted into the lookup table while its PK column does not > have the max value, ID numbers in the new dictionary could be changed for > those PK value larger than the newest one. What's more, using lookup table as > source data for FK column's dictionary may has performance advantage for > merging job, but also may encounter too big dictionary problem for large > lookup tables. And we must add some validation rules to ensure the PK value > sequential and incremental. > On the another hand, we could just unify using the flat hive table as data > source for all dictionaries. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2847) avoid doing useless work by checking query deadline
[ https://issues.apache.org/jira/browse/KYLIN-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dayue Gao updated KYLIN-2847: - Attachment: KYLIN-2847.patch patch uploaded, please review > avoid doing useless work by checking query deadline > --- > > Key: KYLIN-2847 > URL: https://issues.apache.org/jira/browse/KYLIN-2847 > Project: Kylin > Issue Type: Improvement > Components: Query Engine, Storage - HBase >Affects Versions: v2.1.0 >Reporter: Dayue Gao >Assignee: Dayue Gao > Attachments: KYLIN-2847.patch > > > Under high load, requests can spend a long time waiting in RPC queue, > probably longer than query timeout. However, current coprocessor timeout > mechanism doesn't take RPC queue time into account. As a result, handling > these requests not only waste server resources while doing nothing useful, > but can also cause cascading failure when server crashes and client retries. > To recover from server overload ASAP, we should check query deadline at each > stages of query processing, and avoid spending resources on query that will > exceed their deadline. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2848) Query of non-aggregated columns returns bogus results
Vladimir Prus created KYLIN-2848:
Summary: Query of non-aggregated columns returns bogus results
Key: KYLIN-2848
URL: https://issues.apache.org/jira/browse/KYLIN-2848
Project: Kylin
Issue Type: Bug
Reporter: Vladimir Prus
I have a cube called fact_secondary_event, with a dimension called has_text.
That dimension is not used as measure at all. I execute a query such as:
{code}
select dim_date_id, count(*), sum(has_text)
from mart.star_secondary_event
where has_text = 1
group by dim_date_id
{code}
Expected result: an error saying that there's no aggregation for `sum(has_text)`
Actual result: rows such as
{code}
2017-08-31,18598,1
{code}
In other words, for aggregation that cannot be computed, I get back '1'. I
don't think that behaviour of returning random values is helpful, and it's not
even easy to debug after the data travels through a few levels of backend and
UI code. Could an error be omitted in such cases?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KYLIN-2847) avoid doing useless work by checking query deadline
Dayue Gao created KYLIN-2847: Summary: avoid doing useless work by checking query deadline Key: KYLIN-2847 URL: https://issues.apache.org/jira/browse/KYLIN-2847 Project: Kylin Issue Type: Improvement Components: Query Engine, Storage - HBase Affects Versions: v2.1.0 Reporter: Dayue Gao Assignee: Dayue Gao Under high load, requests can spend a long time waiting in RPC queue, probably longer than query timeout. However, current coprocessor timeout mechanism doesn't take RPC queue time into account. As a result, handling these requests not only waste server resources while doing nothing useful, but can also cause cascading failure when server crashes and client retries. To recover from server overload ASAP, we should check query deadline at each stages of query processing, and avoid spending resources on query that will exceed their deadline. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2846) Add a config of hbase namespace for cube storage
[ https://issues.apache.org/jira/browse/KYLIN-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-2846: --- Attachment: KYLIN-2846-001.patch Patch for KYLIN-2846 > Add a config of hbase namespace for cube storage > > > Key: KYLIN-2846 > URL: https://issues.apache.org/jira/browse/KYLIN-2846 > Project: Kylin > Issue Type: New Feature > Components: Storage - HBase >Affects Versions: v2.1.0 >Reporter: Liu Shaohui >Assignee: liyang >Priority: Minor > Fix For: Future > > Attachments: KYLIN-2846-001.patch > > > In multi-tenancy HBase cluster, namespace is important for quota management > and permission control. So we add a global configuration of hbase namespace > for cube storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2846) Add a config of hbase namespace for cube storage
Liu Shaohui created KYLIN-2846: -- Summary: Add a config of hbase namespace for cube storage Key: KYLIN-2846 URL: https://issues.apache.org/jira/browse/KYLIN-2846 Project: Kylin Issue Type: New Feature Components: Storage - HBase Affects Versions: v2.1.0 Reporter: Liu Shaohui Assignee: liyang Priority: Minor Fix For: Future In multi-tenancy HBase cluster, namespace is important for quota management and permission control. So we add a global configuration of hbase namespace for cube storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2845) Merging cube get error IllegalStateException
[
https://issues.apache.org/jira/browse/KYLIN-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangQin updated KYLIN-2845:
Description:
When trying to merge cubes, I encountered following errors :
Error: java.lang.IllegalStateException at
com.google.common.base.Preconditions.checkState(Preconditions.java:129) at
org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123) at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
I digged into the source code and found the following code:
{code:java}
public void encode(ByteArray bodyBytes, ByteArray outputBuf) {
Preconditions.checkState(bodyBytes.length() == bodyLength);
Preconditions.checkState(bodyBytes.length() + getHeaderLength() ==
outputBuf.length(), //
"bodybytes length: " + bodyBytes.length() + " outputBuf length:
" + outputBuf.length() + " header length: " + getHeaderLength());
System.arraycopy(bodyBytes.array(), bodyBytes.offset(),
outputBuf.array(), getHeaderLength(), bodyLength);
//fill shard and cuboid
fillHeader(outputBuf.array());
}
{code}
And also I found an old similar bug
[https://issues.apache.org/jira/browse/KYLIN-1004].
The cube can build correctly, but the merge just does not work.
Here is the detail for this error:
2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Reloaded
cube cube_fact_dui_assemble_log being CUBE[name=cube_fact_dui_assemble_log]
having 7 segments
2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Loaded 1
cubes, fail on 0 cubes
2017-09-05 09:35:57,805 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_CALENDAR/P_DAY/7dbd2f63-ffc6-4eb6-972a-b712954bcff0.dict
2017-09-05 09:35:57,821 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/31e3578a-94fe-4d99-b00f-c523f59b8eb2.dict
2017-09-05 09:35:57,833 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/9e365c29-52d8-4397-acb9-63cb9214a05e.dict
2017-09-05 09:35:57,834 INFO [main] org.apache.kylin.engine.mr.KylinMapper:
Accepting Mapper Key with ordinal: 1
2017-09-05 09:35:57,837 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
2017-09-05 09:35:57,866 INFO [main] org.apache.hadoop.mapred.MapTask: Starting
flush of map output
2017-09-05 09:35:57,958 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new compressor [.deflate]
2017-09-05 09:35:57,964 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
was:
* When trying to merge cubes,
[jira] [Updated] (KYLIN-2845) Merging cube get error IllegalStateException
[
https://issues.apache.org/jira/browse/KYLIN-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangQin updated KYLIN-2845:
Description:
When trying to merge cubes, I encountered following errors :
Error: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
I digged into the source code and found the following code:
{code:java}
public void encode(ByteArray bodyBytes, ByteArray outputBuf) {
Preconditions.checkState(bodyBytes.length() == bodyLength);
Preconditions.checkState(bodyBytes.length() + getHeaderLength() ==
outputBuf.length(), //
"bodybytes length: " + bodyBytes.length() + " outputBuf length:
" + outputBuf.length() + " header length: " + getHeaderLength());
System.arraycopy(bodyBytes.array(), bodyBytes.offset(),
outputBuf.array(), getHeaderLength(), bodyLength);
//fill shard and cuboid
fillHeader(outputBuf.array());
}
{code}
And also I found an old similar bug
[https://issues.apache.org/jira/browse/KYLIN-1004].
The cube can build correctly, but the merge just does not work.
Here is the detail for this error:
2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Reloaded
cube cube_fact_dui_assemble_log being CUBE[name=cube_fact_dui_assemble_log]
having 7 segments
2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Loaded 1
cubes, fail on 0 cubes
2017-09-05 09:35:57,805 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_CALENDAR/P_DAY/7dbd2f63-ffc6-4eb6-972a-b712954bcff0.dict
2017-09-05 09:35:57,821 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/31e3578a-94fe-4d99-b00f-c523f59b8eb2.dict
2017-09-05 09:35:57,833 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/9e365c29-52d8-4397-acb9-63cb9214a05e.dict
2017-09-05 09:35:57,834 INFO [main] org.apache.kylin.engine.mr.KylinMapper:
Accepting Mapper Key with ordinal: 1
2017-09-05 09:35:57,837 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
2017-09-05 09:35:57,866 INFO [main] org.apache.hadoop.mapred.MapTask: Starting
flush of map output
2017-09-05 09:35:57,958 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new compressor [.deflate]
2017-09-05 09:35:57,964 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
was:
When trying to merge cubes, I
[jira] [Updated] (KYLIN-2845) Merging cube get error IllegalStateException
[
https://issues.apache.org/jira/browse/KYLIN-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangQin updated KYLIN-2845:
Description:
* When trying to merge cubes, I encountered following errors :
Error: java.lang.IllegalStateException at
com.google.common.base.Preconditions.checkState(Preconditions.java:129) at
org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123) at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
I digged into the source code and found the following code:
{code:java}
public void encode(ByteArray bodyBytes, ByteArray outputBuf) {
Preconditions.checkState(bodyBytes.length() == bodyLength);
Preconditions.checkState(bodyBytes.length() + getHeaderLength() ==
outputBuf.length(), //
"bodybytes length: " + bodyBytes.length() + " outputBuf length:
" + outputBuf.length() + " header length: " + getHeaderLength());
System.arraycopy(bodyBytes.array(), bodyBytes.offset(),
outputBuf.array(), getHeaderLength(), bodyLength);
//fill shard and cuboid
fillHeader(outputBuf.array());
}
{code}
And also I found an old similar bug
[https://issues.apache.org/jira/browse/KYLIN-1004].
The cube can build correctly, but the merge just does not work.
Here is the detail for this error:
2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Reloaded
cube cube_fact_dui_assemble_log being CUBE[name=cube_fact_dui_assemble_log]
having 7 segments
2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Loaded 1
cubes, fail on 0 cubes
2017-09-05 09:35:57,805 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_CALENDAR/P_DAY/7dbd2f63-ffc6-4eb6-972a-b712954bcff0.dict
2017-09-05 09:35:57,821 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/31e3578a-94fe-4d99-b00f-c523f59b8eb2.dict
2017-09-05 09:35:57,833 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/9e365c29-52d8-4397-acb9-63cb9214a05e.dict
2017-09-05 09:35:57,834 INFO [main] org.apache.kylin.engine.mr.KylinMapper:
Accepting Mapper Key with ordinal: 1
2017-09-05 09:35:57,837 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
2017-09-05 09:35:57,866 INFO [main] org.apache.hadoop.mapred.MapTask: Starting
flush of map output
2017-09-05 09:35:57,958 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new compressor [.deflate]
2017-09-05 09:35:57,964 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
was:
When trying to merge cubes,
[jira] [Updated] (KYLIN-2844) override max-visit-scanrange and max-fuzzykey-scan at cube level
[ https://issues.apache.org/jira/browse/KYLIN-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dayue Gao updated KYLIN-2844: - Attachment: KYLIN-2844.patch patch uploaded, please review > override max-visit-scanrange and max-fuzzykey-scan at cube level > > > Key: KYLIN-2844 > URL: https://issues.apache.org/jira/browse/KYLIN-2844 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Dayue Gao >Assignee: Dayue Gao > Attachments: KYLIN-2844.patch > > > User should be able to override "kylin.storage.hbase.max-fuzzykey-scan" and > "kylin.storage.hbase.max-visit-scanrange" at cube/project level. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
