Hello 不清
>From your errors: "Failed to build cube in mapper " & A- "java.lang.OutOfMemoryError: Java heap space at java" & B- "java.lang.OutOfMemoryError: GC overhead limit" For error A:Check override this parameters from kylin: * kylin.job.mr.config.override.mapred.map.child.java.opts=-Xmx8g * * kylin.job.mr.config.override.mapreduce.map.memory.mb=8500* *For error B: (this is more complicated)* * Check you are using Java 8 or higer* * Try with this *-XX:+UseG1GC Explanation: https://wiki.apache.org/solr/ShawnHeisey yes, use Integer dictionary is the best option 2017-02-13 3:53 GMT+01:00 不清 <452652...@qq.com>: > kylin社区,您好！ > > 是手机号作为维度，这个维度的去重值在500w~1500w。 > 我是使用的integer 编码 然后length设置为8. 测试的数据量大约在1亿条。是我设置的有问题么？ > > 对于超大维度，kylin需要进行什么设置么？ > > 我使用的kylin版本是1.6. > > 谢谢 > > 报错步骤是 build cube > map任务耗时特别长，最后还报错了,如下 > Error: java.io.IOException: Failed to build cube in mapper 36 at > org.apache.kylin.engine.mr.steps.InMemCuboidMapper. > cleanup(InMemCuboidMapper.java:145) at > org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:415) at > org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1642) at org.apache.hadoop.mapred. > YarnChild.main(YarnChild.java:163) Caused by: > java.util.concurrent.ExecutionException: > java.lang.RuntimeException: java.io.IOException: java.io.IOException: > java.lang.RuntimeException: java.io.IOException: > java.lang.OutOfMemoryError: Java heap space at java.util.concurrent. > FutureTask.report(FutureTask.java:122) at java.util.concurrent. > FutureTask.get(FutureTask.java:188) at org.apache.kylin.engine.mr. > steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:143) ... 8 more > Caused by: java.lang.RuntimeException: java.io.IOException: > java.io.IOException: java.lang.RuntimeException: java.io.IOException: > java.lang.OutOfMemoryError: Java heap space at org.apache.kylin.cube. > inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:84) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: > java.io.IOException: java.lang.RuntimeException: java.io.IOException: > java.lang.OutOfMemoryError: Java heap space at org.apache.kylin.cube. > inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:128) > at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder. > build(DoggedCubeBuilder.java:75) at org.apache.kylin.cube.inmemcubing. > AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82) ... 5 > more Caused by: java.io.IOException: java.lang.RuntimeException: > java.io.IOException: java.lang.OutOfMemoryError: Java heap space at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:196) > at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$ > BuildOnce.checkException(DoggedCubeBuilder.java:169) at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116) > ... 7 more Caused by: java.lang.RuntimeException: java.io.IOException: > java.lang.OutOfMemoryError: Java heap space at org.apache.kylin.cube. > inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:289) > Caused by: java.io.IOException: java.lang.OutOfMemoryError: Java heap space > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.throwExceptionIfAny(InMemCubeBuilder.java:226) > at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > build(InMemCubeBuilder.java:186) at org.apache.kylin.cube. > inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:137) at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:284) > Caused by: java.lang.OutOfMemoryError: Java heap space at > java.math.BigInteger.<init>(BigInteger.java:973) at > java.math.BigInteger.valueOf(BigInteger.java:957) at > java.math.BigDecimal.inflate(BigDecimal.java:3519) at > java.math.BigDecimal.unscaledValue(BigDecimal.java:2205) at > org.apache.kylin.metadata.datatype.BigDecimalSerializer.serialize(BigDecimalSerializer.java:56) > at > org.apache.kylin.metadata.datatype.BigDecimalSerializer.serialize(BigDecimalSerializer.java:33) > at org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:76) at > org.apache.kylin.measure.BufferedMeasureCodec.encode(BufferedMeasureCodec.java:93) > at org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$ > ReturningRecord.load(GTAggregateScanner.java:412) at > org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$2.next(GTAggregateScanner.java:355) > at > org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$2.next(GTAggregateScanner.java:342) > at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > scanAndAggregateGridTable(InMemCubeBuilder.java:436) at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.aggregateCuboid(InMemCubeBuilder.java:399) > at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > buildCuboid(InMemCubeBuilder.java:391) at org.apache.kylin.cube. > inmemcubing.InMemCubeBuilder.access$300(InMemCubeBuilder.java:65) at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$CuboidTaskThread.run(InMemCubeBuilder.java:270) > Container killed by the ApplicationMaster. Container killed on request. > Exit code is 143 Container exited with a non-zero exit code 143 > > > > 还有一次报错是: > Error: java.io.IOException: Failed to build cube in mapper 11 at > org.apache.kylin.engine.mr.steps.InMemCuboidMapper. > cleanup(InMemCuboidMapper.java:145) at > org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:415) at > org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1642) at org.apache.hadoop.mapred. > YarnChild.main(YarnChild.java:163) Caused by: > java.util.concurrent.ExecutionException: > java.lang.RuntimeException: java.io.IOException: java.io.IOException: > java.lang.RuntimeException: java.io.IOException: > java.lang.OutOfMemoryError: GC overhead limit exceeded at > java.util.concurrent.FutureTask.report(FutureTask.java:122) at > java.util.concurrent.FutureTask.get(FutureTask.java:188) at > org.apache.kylin.engine.mr.steps.InMemCuboidMapper. > cleanup(InMemCuboidMapper.java:143) ... 8 more Caused by: > java.lang.RuntimeException: java.io.IOException: java.io.IOException: > java.lang.RuntimeException: java.io.IOException: > java.lang.OutOfMemoryError: GC overhead limit exceeded at > org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1. > run(AbstractInMemCubeBuilder.java:84) at java.util.concurrent. > Executors$RunnableAdapter.call(Executors.java:471) at > java.util.concurrent.FutureTask.run(FutureTask.java:262) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: > java.io.IOException: java.lang.RuntimeException: java.io.IOException: > java.lang.OutOfMemoryError: GC overhead limit exceeded at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:128) > at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder. > build(DoggedCubeBuilder.java:75) at org.apache.kylin.cube.inmemcubing. > AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82) ... 5 > more Caused by: java.io.IOException: java.lang.RuntimeException: > java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:196) > at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$ > BuildOnce.checkException(DoggedCubeBuilder.java:169) at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116) > ... 7 more Caused by: java.lang.RuntimeException: java.io.IOException: > java.lang.OutOfMemoryError: GC overhead limit exceeded at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:289) > Caused by: java.io.IOException: java.lang.OutOfMemoryError: GC overhead > limit exceeded at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > throwExceptionIfAny(InMemCubeBuilder.java:226) at org.apache.kylin.cube. > inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:186) at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > build(InMemCubeBuilder.java:137) at org.apache.kylin.cube. > inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:284) > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded at > java.math.BigInteger.valueOf(BigInteger.java:957) at > java.math.BigDecimal.inflate(BigDecimal.java:3519) at > java.math.BigDecimal.unscaledValue(BigDecimal.java:2205) at > org.apache.kylin.metadata.datatype.BigDecimalSerializer.serialize(BigDecimalSerializer.java:56) > at > org.apache.kylin.metadata.datatype.BigDecimalSerializer.serialize(BigDecimalSerializer.java:33) > at org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:76) at > org.apache.kylin.measure.BufferedMeasureCodec.encode(BufferedMeasureCodec.java:93) > at org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$ > ReturningRecord.load(GTAggregateScanner.java:412) at > org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$2.next(GTAggregateScanner.java:355) > at > org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$2.next(GTAggregateScanner.java:342) > at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > scanAndAggregateGridTable(InMemCubeBuilder.java:436) at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.aggregateCuboid(InMemCubeBuilder.java:399) > at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > buildCuboid(InMemCubeBuilder.java:391) at org.apache.kylin.cube. > inmemcubing.InMemCubeBuilder.access$300(InMemCubeBuilder.java:65) at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$CuboidTaskThread.run( > InMemCubeBuilder.java:270) > >
