[
https://issues.apache.org/jira/browse/KYLIN-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangQin updated KYLIN-2845:
----------------------------
Comment: was deleted
(was: The phenomenon takes place when the length of dimensions varies from one
segment to another.
With the tool org.apache.kylin.engine.mr.common.CubeStatsReader, I found that
the 'Length of dimension' of one column has changed from 1 to 2.
FYI, I did not change the length of this column in hive table , nor did I make
any difference in Kylin Cube design ('Advanced Setting'->'Rowkeys')
And the source code I pasted, 'Preconditions.checkState', seems to block this
change and throws an exception.)
> Merging cube get error IllegalStateException
> --------------------------------------------
>
> Key: KYLIN-2845
> URL: https://issues.apache.org/jira/browse/KYLIN-2845
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v2.0.0
> Reporter: JiangQin
> Assignee: Dong Li
>
> When trying to merge cubes, I encountered following errors :
> Error: java.lang.IllegalStateException
> at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
> at
> org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
> I digged into the source code and found the following code:
> {code:java}
> public void encode(ByteArray bodyBytes, ByteArray outputBuf) {
> Preconditions.checkState(bodyBytes.length() == bodyLength);
> Preconditions.checkState(bodyBytes.length() + getHeaderLength() ==
> outputBuf.length(), //
> "bodybytes length: " + bodyBytes.length() + " outputBuf
> length: " + outputBuf.length() + " header length: " + getHeaderLength());
> System.arraycopy(bodyBytes.array(), bodyBytes.offset(),
> outputBuf.array(), getHeaderLength(), bodyLength);
> //fill shard and cuboid
> fillHeader(outputBuf.array());
> }
> {code}
> And also I found an old similar bug
> [https://issues.apache.org/jira/browse/KYLIN-1004].
> The cube can build correctly, but the merge just does not work.
> Here is the detail for this error:
> 2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager:
> Reloaded cube cube_fact_dui_assemble_log being
> CUBE[name=cube_fact_dui_assemble_log] having 7 segments
> 2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Loaded
> 1 cubes, fail on 0 cubes
> 2017-09-05 09:35:57,805 INFO [main] org.apache.kylin.dict.DictionaryManager:
> DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
> /dict/DIM_DUI.DIM_CALENDAR/P_DAY/7dbd2f63-ffc6-4eb6-972a-b712954bcff0.dict
> 2017-09-05 09:35:57,821 INFO [main] org.apache.kylin.dict.DictionaryManager:
> DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
> /dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/31e3578a-94fe-4d99-b00f-c523f59b8eb2.dict
> 2017-09-05 09:35:57,833 INFO [main] org.apache.kylin.dict.DictionaryManager:
> DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
> /dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/9e365c29-52d8-4397-acb9-63cb9214a05e.dict
> 2017-09-05 09:35:57,834 INFO [main] org.apache.kylin.engine.mr.KylinMapper:
> Accepting Mapper Key with ordinal: 1
> 2017-09-05 09:35:57,837 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
> at
> org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
> at
> org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> 2017-09-05 09:35:57,866 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> 2017-09-05 09:35:57,958 INFO [main] org.apache.hadoop.io.compress.CodecPool:
> Got brand-new compressor [.deflate]
> 2017-09-05 09:35:57,964 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
> at
> org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
> at
> org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
