thanks for reply??
For error A, I can set these parameters in kylin. But for error B,should I fix this problem for whole hadoop cluster? Can you speak the parameter fix in detail? This really helped us a lot! ------------------ ???????? ------------------ ??????: "Alberto Ram??n";<[email protected]>; ????????: 2017??2??13??(??????) ????3:58 ??????: "user"<[email protected]>; ????: Re: ?????????????????? Hello ???? From your errors: "Failed to build cube in mapper " & A- "java.lang.OutOfMemoryError: Java heap space at java" & B- "java.lang.OutOfMemoryError: GC overhead limit" For error A:Check override this parameters from kylin: kylin.job.mr.config.override.mapred.map.child.java.opts=-Xmx8g kylin.job.mr.config.override.mapreduce.map.memory.mb=8500 For error B: (this is more complicated) Check you are using Java 8 or higer Try with this -XX:+UseG1GC Explanation: https://wiki.apache.org/solr/ShawnHeisey yes, use Integer dictionary is the best option 2017-02-13 3:53 GMT+01:00 ???? <[email protected]>: kylin????,?????? ????????????????????????????????????500w~1500w?? ??????????integer ???? ????length??????8. ??????????????????1?????????????????????????? ??????????????kylin???????????????????? ????????kylin??????1.6. ???? ?????????? build cube map????????????????????????????,???? Error: java.io.IOException: Failed to build cube in mapper 36 at org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:145) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: java.io.IOException: java.lang.RuntimeException: java.io.IOException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:143) ... 8 more Caused by: java.lang.RuntimeException: java.io.IOException: java.io.IOException: java.lang.RuntimeException: java.io.IOException: java.lang.OutOfMemoryError: Java heap space at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:84) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: java.io.IOException: java.lang.RuntimeException: java.io.IOException: java.lang.OutOfMemoryError: Java heap space at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:128) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75) at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82) ... 5 more Caused by: java.io.IOException: java.lang.RuntimeException: java.io.IOException: java.lang.OutOfMemoryError: Java heap space at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:196) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:169) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116) ... 7 more Caused by: java.lang.RuntimeException: java.io.IOException: java.lang.OutOfMemoryError: Java heap space at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:289) Caused by: java.io.IOException: java.lang.OutOfMemoryError: Java heap space at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.throwExceptionIfAny(InMemCubeBuilder.java:226) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:186) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:137) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:284) Caused by: java.lang.OutOfMemoryError: Java heap space at java.math.BigInteger.<init>(BigInteger.java:973) at java.math.BigInteger.valueOf(BigInteger.java:957) at java.math.BigDecimal.inflate(BigDecimal.java:3519) at java.math.BigDecimal.unscaledValue(BigDecimal.java:2205) at org.apache.kylin.metadata.datatype.BigDecimalSerializer.serialize(BigDecimalSerializer.java:56) at org.apache.kylin.metadata.datatype.BigDecimalSerializer.serialize(BigDecimalSerializer.java:33) at org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:76) at org.apache.kylin.measure.BufferedMeasureCodec.encode(BufferedMeasureCodec.java:93) at org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$ReturningRecord.load(GTAggregateScanner.java:412) at org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$2.next(GTAggregateScanner.java:355) at org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$2.next(GTAggregateScanner.java:342) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.scanAndAggregateGridTable(InMemCubeBuilder.java:436) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.aggregateCuboid(InMemCubeBuilder.java:399) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.buildCuboid(InMemCubeBuilder.java:391) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.access$300(InMemCubeBuilder.java:65) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$CuboidTaskThread.run(InMemCubeBuilder.java:270) Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 ??????????????: Error: java.io.IOException: Failed to build cube in mapper 11 at org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:145) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: java.io.IOException: java.lang.RuntimeException: java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:143) ... 8 more Caused by: java.lang.RuntimeException: java.io.IOException: java.io.IOException: java.lang.RuntimeException: java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:84) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: java.io.IOException: java.lang.RuntimeException: java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:128) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75) at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82) ... 5 more Caused by: java.io.IOException: java.lang.RuntimeException: java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:196) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:169) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116) ... 7 more Caused by: java.lang.RuntimeException: java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:289) Caused by: java.io.IOException: java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.throwExceptionIfAny(InMemCubeBuilder.java:226) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:186) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:137) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:284) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded at java.math.BigInteger.valueOf(BigInteger.java:957) at java.math.BigDecimal.inflate(BigDecimal.java:3519) at java.math.BigDecimal.unscaledValue(BigDecimal.java:2205) at org.apache.kylin.metadata.datatype.BigDecimalSerializer.serialize(BigDecimalSerializer.java:56) at org.apache.kylin.metadata.datatype.BigDecimalSerializer.serialize(BigDecimalSerializer.java:33) at org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:76) at org.apache.kylin.measure.BufferedMeasureCodec.encode(BufferedMeasureCodec.java:93) at org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$ReturningRecord.load(GTAggregateScanner.java:412) at org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$2.next(GTAggregateScanner.java:355) at org.apache.kylin.gridtable.GTAggregateScanner$AggregationCache$2.next(GTAggregateScanner.java:342) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.scanAndAggregateGridTable(InMemCubeBuilder.java:436) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.aggregateCuboid(InMemCubeBuilder.java:399) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.buildCuboid(InMemCubeBuilder.java:391) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.access$300(InMemCubeBuilder.java:65) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$CuboidTaskThread.run(InMemCubeBuilder.java:270)
