Moreover one more interesting thing which came to my mind was that since I have 3 cubes in total and this is generally the issue I am facing while building only one cube.
So basically 2/3 cubes build properly, but only one cube fails. Let me mention it, the failing cubes are not same. Thanks, On Fri, Jul 10, 2015 at 11:55 AM, Vineet Mishra <[email protected]> wrote: > Hi All, > > Currently I have around 3 cubes to build, which some times fails at Second > Last Step of Cube creation i.e Load Hfile to Hbase Table with the exception > > java.io.IOException: BulkLoad encountered an unrecoverable problem > at > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.bulkLoadPhase(LoadIncrementalHFiles.java:381) > at > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:310) > at > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:896) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.kylin.job.hadoop.hbase.BulkLoadJob.run(BulkLoadJob.java:83) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:132) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: > Failed after attempts=35, exceptions: > Thu Jul 09 19:38:48 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:38:49 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:38:49 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:38:50 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:38:52 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:38:56 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:39:06 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:39:16 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:39:26 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:39:36 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:39:57 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:40:17 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:40:37 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:40:57 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:41:17 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:41:37 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:41:57 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:42:17 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:42:37 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:42:57 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:43:17 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:43:37 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:43:57 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:44:18 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:44:38 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:44:58 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:45:18 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:45:38 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:45:58 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:46:18 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:46:38 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:46:58 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:47:18 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:47:38 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > Thu Jul 09 19:47:59 PDT 2015, > org.apache.hadoop.hbase.client.RpcRetryingCaller@485c74a6, > java.net.UnknownHostException: unknown host: prod-hadoop-data02 > > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:129) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90) > at > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:679) > at > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:360) > at > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:358) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > Caused by: java.net.UnknownHostException: unknown host: prod-hadoop-data02 > at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.<init>(RpcClient.java:385) > at > org.apache.hadoop.hbase.ipc.RpcClient.createConnection(RpcClient.java:351) > at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1530) > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) > at > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) > at > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.bulkLoadHFile(ClientProtos.java:30340) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUtil.java:1605) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUtil.java:1585) > at > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:634) > at > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:623) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114) > ... 8 more > > > No doubt, the exception clearly states that its not able to reach to the > mentioned host, but unfortunately the host-ip entry was missing in the > hosts file which I made sure to include in the hosts file. > > Moreover yesterday after including the host entry I re-ran the job(still > it wasn't working with resume) and it worked as expected but today again > its falling at the same point. > > Any helpful advice would be appreciated. > > Thanks, >
