When trying to merge cubes, I encountered following errors :
Error: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
I digged into the source code and found the following code:
public void encode(ByteArray bodyBytes, ByteArray outputBuf) {
Preconditions.checkState(bodyBytes.length() == bodyLength);
Preconditions.checkState(bodyBytes.length() + getHeaderLength() ==
outputBuf.length(), //
"bodybytes length: " + bodyBytes.length() + " outputBuf length: " +
outputBuf.length() + " header length: " + getHeaderLength());
System.arraycopy(bodyBytes.array(), bodyBytes.offset(),
outputBuf.array(), getHeaderLength(), bodyLength);
//fill shard and cuboid
fillHeader(outputBuf.array());
}
And I use org.apache.kylin.engine.mr.common.CubeStatsReader command find out
that the Length of dimensions in these two segemnts that I tried to merge are
different . FYI, I havent revised the setting in 'Rowkeys', all of them are as
following:
And also I found an old similar bug
https://issues.apache.org/jira/browse/KYLIN-1004.
The cube can build correctly, but the merge just does not work. Here is the
detail for this error:
2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Reloaded
cube cube_fact_dui_assemble_log being CUBE[name=cube_fact_dui_assemble_log]
having 7 segments
2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Loaded 1
cubes, fail on 0 cubes
2017-09-05 09:35:57,805 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_CALENDAR/P_DAY/7dbd2f63-ffc6-4eb6-972a-b712954bcff0.dict
2017-09-05 09:35:57,821 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/31e3578a-94fe-4d99-b00f-c523f59b8eb2.dict
2017-09-05 09:35:57,833 INFO [main] org.apache.kylin.dict.DictionaryManager:
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at
/dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/9e365c29-52d8-4397-acb9-63cb9214a05e.dict
2017-09-05 09:35:57,834 INFO [main] org.apache.kylin.engine.mr.KylinMapper:
Accepting Mapper Key with ordinal: 1
2017-09-05 09:35:57,837 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
2017-09-05 09:35:57,866 INFO [main] org.apache.hadoop.mapred.MapTask: Starting
flush of map output
2017-09-05 09:35:57,958 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new compressor [.deflate]
2017-09-05 09:35:57,964 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
