Hi All, Currently I have around 3 cubes to build, which some times fails at Second Last Step of Cube creation i.e Load Hfile to Hbase Table with the exception
java.io.IOException: BulkLoad encountered an unrecoverable problem at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.bulkLoadPhase(LoadIncrementalHFiles.java:381) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:310) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:896) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.kylin.job.hadoop.hbase.BulkLoadJob.run(BulkLoadJob.java:83) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:132) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Thu Jul 09 19:38:48 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:38:49 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:38:49 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:38:50 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:38:52 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:38:56 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:39:06 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:39:16 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:39:26 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:39:36 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:39:57 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:40:17 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:40:37 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:40:57 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:41:17 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:41:37 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:41:57 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:42:17 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:42:37 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:42:57 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:43:17 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:43:37 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:43:57 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:44:18 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:44:38 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:44:58 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:45:18 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:45:38 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:45:58 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:46:18 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:46:38 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:46:58 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:47:18 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:47:38 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 Thu Jul 09 19:47:59 PDT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, java.net.UnknownHostException: unknown host: prod-hadoop-data02 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:129) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:679) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:360) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:358) at java.util.concurrent.FutureTask.run(FutureTask.java:262) ... 3 more Caused by: java.net.UnknownHostException: unknown host: prod-hadoop-data02 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.<init>(RpcClient.java:385) at org.apache.hadoop.hbase.ipc.RpcClient.createConnection(RpcClient.java:351) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1530) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.bulkLoadHFile(ClientProtos.java:30340) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUtil.java:1605) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUtil.java:1585) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:634) at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:623) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114) ... 8 more No doubt, the exception clearly states that its not able to reach to the mentioned host, but unfortunately the host-ip entry was missing in the hosts file which I made sure to include in the hosts file. Moreover yesterday after including the host entry I re-ran the job(still it wasn't working with resume) and it worked as expected but today again its falling at the same point. Any helpful advice would be appreciated. Thanks,
