for B: its a java option, (. . . java.opts) Check if your JVM isn't very old, there are a lot of optimizacions for GC in last versions of Java 8
TIP 1: Check if you can reduce dimensionality of cube or use AGG to make lighter the build process You canTake some ideas from this <https://github.com/albertoRamon/Kylin/tree/master/KylinPerformance> TIP 2: solve first problem A, because if you enlarge Heap, the B will be worst 2017-02-13 10:16 GMT+01:00 不清 <452652...@qq.com>: > thanks for reply! > > For error A, I can set these parameters in kylin. > > But for error B,should I fix this problem for whole hadoop cluster? Can > you speak the parameter fix in detail? > > This really helped us a lot! > > > ------------------ 原始邮件 ------------------ > *发件人:* "Alberto Ramón";<a.ramonporto...@gmail.com>; > *发送时间:* 2017年2月13日(星期一) 下午3:58 > *收件人:* "user"<user@kylin.apache.org>; > *主题:* Re: 求助有一个超大维度 > > Hello 不清 > > > From your errors: "Failed to build cube in mapper " & > A- "java.lang.OutOfMemoryError: Java heap space at java" & > B- "java.lang.OutOfMemoryError: GC overhead limit" > > For error A:Check override this parameters from kylin: > > > * kylin.job.mr.config.override.mapred.map.child.java.opts=-Xmx8g * > > * kylin.job.mr.config.override.mapreduce.map.memory.mb=8500* > > > > *For error B: (this is more complicated)* > > * Check you are using Java 8 or higer* > > * Try with this *-XX:+UseG1GC > > Explanation: https://wiki.apache.org/solr/ShawnHeisey > > > yes, use Integer dictionary is the best option > > > > 2017-02-13 3:53 GMT+01:00 不清 <452652...@qq.com>: > >> kylin社区,您好! >> >> 是手机号作为维度,这个维度的去重值在500w~1500w。 >> 我是使用的integer 编码 然后length设置为8. 测试的数据量大约在1亿条。是我设置的有问题么? >> >> 对于超大维度,kylin需要进行什么设置么? >> >> 我使用的kylin版本是1.6. >> >> 谢谢 >> >> 报错步骤是 build cube >> map任务耗时特别长,最后还报错了,如下 >> Error: java.io.IOException: Failed to build cube in mapper 36 at >> org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:145) >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at >> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at >> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at >> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at >> java.security.AccessController.doPrivileged(Native Method) at >> javax.security.auth.Subject.doAs(Subject.java:415) at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) >> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused >> by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: >> java.io.IOException: java.io.IOException: java.lang.RuntimeException: >> java.io.IOException: java.lang.OutOfMemoryError: Java heap space at >> java.util.concurrent.FutureTask.report(FutureTask.java:122) at >> java.util.concurrent.FutureTask.get(FutureTask.java:188) at >> org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:143) >> ... 8 more Caused by: java.lang.RuntimeException: java.io.IOException: >> java.io.IOException: java.lang.RuntimeException: java.io.IOException: >> java.lang.OutOfMemoryError: Java heap space at >> org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run( >> AbstractInMemCubeBuilder.java:84) at java.util.concurrent.Executors >> $RunnableAdapter.call(Executors.java:471) at >> java.util.concurrent.FutureTask.run(FutureTask.java:262) at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: >> java.io.IOException: java.lang.RuntimeException: java.io.IOException: >> java.lang.OutOfMemoryError: Java heap space at >> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnc >> e.build(DoggedCubeBuilder.java:128) at org.apache.kylin.cube.inmemcub >> ing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75) at >> org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run( >> AbstractInMemCubeBuilder.java:82) ... 5 more Caused by: >> java.io.IOException: java.lang.RuntimeException: java.io.IOException: >> java.lang.OutOfMemoryError: Java heap space at >> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnc >> e.abort(DoggedCubeBuilder.java:196) at org.apache.kylin.cube.inmemcub >> ing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:169) >> at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnc >> e.build(DoggedCubeBuilder.java:116) ... 7 more Caused by: >> java.lang.RuntimeException: java.io.IOException: >> java.lang.OutOfMemoryError: Java heap space at >> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThr >> ead.run(DoggedCubeBuilder.java:289) Caused by: java.io.IOException: >> java.lang.OutOfMemoryError: Java heap space at >> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.throwExce >> ptionIfAny(InMemCubeBuilder.java:226) at org.apache.kylin.cube.inmemcub >> ing.InMemCubeBuilder.build(InMemCubeBuilder.java:186) at >> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:137) >> at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThr >> ead.run(DoggedCubeBuilder.java:284) Caused by: >> java.lang.OutOfMemoryError: Java heap space at >> java.math.BigInteger.<init>(BigInteger.java:973) at >> java.math.BigInteger.valueOf(BigInteger.java:957) at >> java.math.BigDecimal.inflate(BigDecimal.java:3519) at >> java.math.BigDecimal.unscaledValue(BigDecimal.java:2205) at >> org.apache.kylin.metadata.datatype.BigDecimalSerializer.seri >> alize(BigDecimalSerializer.java:56) at org.apache.kylin.metadata.data >> type.BigDecimalSerializer.serialize(BigDecimalSerializer.java:33) at >> org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:76) at >> org.apache.kylin.measure.BufferedMeasureCodec.encode(BufferedMeasureCodec.java:93) >> at org.apache.kylin.gridtable.GTAggregateScanner$AggregationCac >> he$ReturningRecord.load(GTAggregateScanner.java:412) at >> org.apache.kylin.gridtable.GTAggregateScanner$AggregationCac >> he$2.next(GTAggregateScanner.java:355) at org.apache.kylin.gridtable.GTA >> ggregateScanner$AggregationCache$2.next(GTAggregateScanner.java:342) at >> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.scanAndAg >> gregateGridTable(InMemCubeBuilder.java:436) at >> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.aggregate >> Cuboid(InMemCubeBuilder.java:399) at org.apache.kylin.cube.inmemcub >> ing.InMemCubeBuilder.buildCuboid(InMemCubeBuilder.java:391) at >> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.access$300(InMemCubeBuilder.java:65) >> at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$CuboidTas >> kThread.run(InMemCubeBuilder.java:270) Container killed by the >> ApplicationMaster. Container killed on request. Exit code is 143 Container >> exited with a non-zero exit code 143 >> >> >> >> 还有一次报错是: >> Error: java.io.IOException: Failed to build cube in mapper 11 at >> org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:145) >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at >> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at >> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at >> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at >> java.security.AccessController.doPrivileged(Native Method) at >> javax.security.auth.Subject.doAs(Subject.java:415) at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) >> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused >> by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: >> java.io.IOException: java.io.IOException: java.lang.RuntimeException: >> java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded >> at java.util.concurrent.FutureTask.report(FutureTask.java:122) at >> java.util.concurrent.FutureTask.get(FutureTask.java:188) at >> org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:143) >> ... 8 more Caused by: java.lang.RuntimeException: java.io.IOException: >> java.io.IOException: java.lang.RuntimeException: java.io.IOException: >> java.lang.OutOfMemoryError: GC overhead limit exceeded at >> org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run( >> AbstractInMemCubeBuilder.java:84) at java.util.concurrent.Executors >> $RunnableAdapter.call(Executors.java:471) at >> java.util.concurrent.FutureTask.run(FutureTask.java:262) at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: >> java.io.IOException: java.lang.RuntimeException: java.io.IOException: >> java.lang.OutOfMemoryError: GC overhead limit exceeded at >> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnc >> e.build(DoggedCubeBuilder.java:128) at org.apache.kylin.cube.inmemcub >> ing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75) at >> org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run( >> AbstractInMemCubeBuilder.java:82) ... 5 more Caused by: >> java.io.IOException: java.lang.RuntimeException: java.io.IOException: >> java.lang.OutOfMemoryError: GC overhead limit exceeded at >> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnc >> e.abort(DoggedCubeBuilder.java:196) at org.apache.kylin.cube.inmemcub >> ing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:169) >> at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnc >> e.build(DoggedCubeBuilder.java:116) ... 7 more Caused by: >> java.lang.RuntimeException: java.io.IOException: >> java.lang.OutOfMemoryError: GC overhead limit exceeded at >> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThr >> ead.run(DoggedCubeBuilder.java:289) Caused by: java.io.IOException: >> java.lang.OutOfMemoryError: GC overhead limit exceeded at >> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.throwExce >> ptionIfAny(InMemCubeBuilder.java:226) at org.apache.kylin.cube.inmemcub >> ing.InMemCubeBuilder.build(InMemCubeBuilder.java:186) at >> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:137) >> at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThr >> ead.run(DoggedCubeBuilder.java:284) Caused by: >> java.lang.OutOfMemoryError: GC overhead limit exceeded at >> java.math.BigInteger.valueOf(BigInteger.java:957) at >> java.math.BigDecimal.inflate(BigDecimal.java:3519) at >> java.math.BigDecimal.unscaledValue(BigDecimal.java:2205) at >> org.apache.kylin.metadata.datatype.BigDecimalSerializer.seri >> alize(BigDecimalSerializer.java:56) at org.apache.kylin.metadata.data >> type.BigDecimalSerializer.serialize(BigDecimalSerializer.java:33) at >> org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:76) at >> org.apache.kylin.measure.BufferedMeasureCodec.encode(BufferedMeasureCodec.java:93) >> at org.apache.kylin.gridtable.GTAggregateScanner$AggregationCac >> he$ReturningRecord.load(GTAggregateScanner.java:412) at >> org.apache.kylin.gridtable.GTAggregateScanner$AggregationCac >> he$2.next(GTAggregateScanner.java:355) at org.apache.kylin.gridtable.GTA >> ggregateScanner$AggregationCache$2.next(GTAggregateScanner.java:342) at >> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.scanAndAg >> gregateGridTable(InMemCubeBuilder.java:436) at >> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.aggregate >> Cuboid(InMemCubeBuilder.java:399) at org.apache.kylin.cube.inmemcub >> ing.InMemCubeBuilder.buildCuboid(InMemCubeBuilder.java:391) at >> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.access$300(InMemCubeBuilder.java:65) >> at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$CuboidTas >> kThread.run(InMemCubeBuilder.java:270) >> >> >