Could you provide more information like Kylin/Hadoop version, your deployment topology, how do you encounter this problem (what changes made before getting this problem), etc? And what findings you have now?
2017-09-16 13:20 GMT+08:00 Fei Yi <[email protected]>: > > > 2017-09-16 12:26:02,431 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641] > mapreduce.LoadIncrementalHFiles:439 : Split occured while grouping > HFiles, retry attempt 10 with 18 files remaining to group or split > 2017-09-16 12:26:02,449 ERROR [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641] > mapreduce.LoadIncrementalHFiles:482 : ------------------------------ > ------------------- > Bulk load aborted with some files not yet loaded: > ------------------------------------------------- > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/ > 3582d9fa7e2b4f528a3f3faa0f7fd7bd > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/ > 738feed13b9a43f9a34ab2d2e7cf6b03 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/ > be0b4e45309a4ca49ddc2aa7402f408f > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/ > 7dd4e4de28d9490ca249b69b216fae89 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/ > f5802d050b4649b2895b8c780825fd70 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/ > ccf41c1f2d78436db3d772bc5facc415 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/ > eb2dc01f7ee84ebb9d913ef3627dbea2 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/ > e0a0d08920e54614b1515a1a698dfe81 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/ > 285323e81d5c49a8acb45bb5e30e5d98 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/ > e91dda576b1c4d29a9e0c0f39a0aca97 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/ > 10fea9b19da344e4a28d7d3148715593 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/ > 1aaf33082e93469eb420c61af4b41561 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/ > ef09b5baeefe4e0aac40b20d61ecc720 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/ > 2f3a9d028e61474e8b191278cdb448f9 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/ > c0b0944aee544f86801711fb36b49a16 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F1/ > c7cd732f1b6f436abf439c13d2c7e111 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/ > 2c310e63ed074d378b82c807bdcc0d65 > hdfs://nameservice1/kylin/kylin_metadata/kylin-fffd693d- > dfcd-4235-82e0-5ede25687805/C_RET_V2/hfile/F2/ > e70e10ed69534ff0a1081db4775fcf0f > > 2017-09-16 12:26:02,449 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641] > client.ConnectionManager$HConnectionImplementation:2259 : Closing master > protocol: MasterService > 2017-09-16 12:26:02,449 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641] > client.ConnectionManager$HConnectionImplementation:1830 : Closing > zookeeper sessionid=0x25e31997e98c034 > 2017-09-16 12:26:02,461 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641] > zookeeper.ZooKeeper:684 : Session: 0x25e31997e98c034 closed > 2017-09-16 12:26:02,461 INFO [Job > fffd693d-dfcd-4235-82e0-5ede25687805-4641-EventThread] > zookeeper.ClientCnxn:512 : EventThread shut down > 2017-09-16 12:26:02,472 ERROR [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641] > common.HadoopShellExecutable:64 : error execute HadoopShellExecutable{id= > fffd693d-dfcd-4235-82e0-5ede25687805-13, name=Load HFile to HBase Table, > state=RUNNING} > java.io.IOException: Retry attempted 10 times without completing, bailing > out > at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles. > doBulkLoad(LoadIncrementalHFiles.java:446) > at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles. > doBulkLoad(LoadIncrementalHFiles.java:339) > at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run( > LoadIncrementalHFiles.java:1063) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.kylin.storage.hbase.steps.BulkLoadJob.run( > BulkLoadJob.java:77) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.kylin.engine.mr.common.HadoopShellExecutable. > doWork(HadoopShellExecutable.java:62) > at org.apache.kylin.job.execution.AbstractExecutable. > execute(AbstractExecutable.java:125) > at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork( > DefaultChainedExecutable.java:65) > at org.apache.kylin.job.execution.AbstractExecutable. > execute(AbstractExecutable.java:125) > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$ > JobRunner.run(DefaultScheduler.java:141) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2017-09-16 12:26:02,479 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641] > execution.ExecutableManager:425 : job > id:fffd693d-dfcd-4235-82e0-5ede25687805-13 > from RUNNING to ERROR > 2017-09-16 12:26:02,499 INFO [Job fffd693d-dfcd-4235-82e0-5ede25687805-4641] > execution.ExecutableManager:425 : job id:fffd693d-dfcd-4235-82e0-5ede25687805 > from RUNNING to ERROR > > -- Best regards, Shaofeng Shi 史少锋
