Re: HBase HTable constructor hangs
Thanks for your confirmation. On Thu, Apr 30, 2015 at 10:17 AM, Tridib Samanta wrote: > You are right. After I moved from HBase 0.98.1 to 1.0.0 this problem got > solved. Thanks all! > > -- > Date: Wed, 29 Apr 2015 06:58:59 -0700 > Subject: Re: HBase HTable constructor hangs > From: yuzhih...@gmail.com > To: tridib.sama...@live.com > CC: d...@ocirs.com; user@spark.apache.org > > Can you verify whether the hbase release you're using has the following > fix ? > HBASE-8 non environment variable solution for "IllegalAccessError" > > Cheers > > On Tue, Apr 28, 2015 at 10:47 PM, Tridib Samanta > wrote: > > I turned on the TRACE and I see lot of following exception: > > java.lang.IllegalAccessError: com/google/protobuf/ZeroCopyLiteralByteString > at > org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:897) > at > org.apache.hadoop.hbase.protobuf.RequestConverter.buildGetRowOrBeforeRequest(RequestConverter.java:131) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1402) > at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:701) > at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:699) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120) > at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:705) > at > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011) > at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:192) > > Thanks > Tridib > > -- > From: d...@ocirs.com > Date: Tue, 28 Apr 2015 22:24:39 -0700 > Subject: Re: HBase HTable constructor hangs > To: tridib.sama...@live.com > > In that case, something else is failing and the reason HBase looks like it > hangs is that the hbase timeout or retry count is too high. > > Try setting the following conf and hbase will only hang for a few mins max > and return a helpful error message. > > hbaseConf.set(HConstants.HBASE_CLIENT_RETRIES_NUMBER, "2") > > > > > -- > Dean Chen > > On Tue, Apr 28, 2015 at 10:18 PM, Tridib Samanta > wrote: > > Nope, my hbase is unsecured. > > -- > From: d...@ocirs.com > Date: Tue, 28 Apr 2015 22:09:51 -0700 > Subject: Re: HBase HTable constructor hangs > To: tridib.sama...@live.com > > > Hi Tridib, > > Are you running this on a secure Hadoop/HBase cluster? I ran in to a > similar issue where the HBase client can successfully connect in local mode > and in the yarn-client driver but not on remote executors. The problem is > that Spark doesn't distribute the hbase auth key, see the following Jira > ticket and PR. > > https://issues.apache.org/jira/browse/SPARK-6918 > > > -- > Dean Chen > > On Tue, Apr 28, 2015 at 9:34 PM, Tridib Samanta > wrote: > > I am 100% sure how it's picking up the configuration. I copied the > hbase-site.xml in hdfs/spark cluster (single machine). I also included > hbase-site.xml in spark-job jar files. spark-job jar file also have > yarn-site and mapred-site and core-site.xml in it. > > One interesting thing is, when I run the spark-job jar as standalone and > execute the HBase client from a main method, it works fine. Same client > unable to connect/hangs when the jar is distributed in spark. > > Thanks > Tridib > > -- > Date: Tue, 28 Apr 2015 21:25:41 -0700 > > Subject: Re: HBase HTable constructor hangs > From: yuzhih...@gmail.com > To: tridib.sama...@live.com > CC: user@spark.apache.org > > How did you distribute hbase-site.xml to the nodes ? > > Looks like HConnectionManager couldn't find the hbase:meta server. > > Cheers > > On Tue, Apr 28, 2015 at 9:19 PM, Tridib Samanta > wrote: > > I am using Spark 1.2.0 and HBase 0.98.1-cdh5.1.0. > > Here is the jstack trace. Complete stack trace attached. > > "Executor task launch worker-1" #58 daemon prio=5 os_prio=0
RE: HBase HTable constructor hangs
You are right. After I moved from HBase 0.98.1 to 1.0.0 this problem got solved. Thanks all! Date: Wed, 29 Apr 2015 06:58:59 -0700 Subject: Re: HBase HTable constructor hangs From: yuzhih...@gmail.com To: tridib.sama...@live.com CC: d...@ocirs.com; user@spark.apache.org Can you verify whether the hbase release you're using has the following fix ? HBASE-8 non environment variable solution for "IllegalAccessError" Cheers On Tue, Apr 28, 2015 at 10:47 PM, Tridib Samanta wrote: I turned on the TRACE and I see lot of following exception: java.lang.IllegalAccessError: com/google/protobuf/ZeroCopyLiteralByteString at org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:897) at org.apache.hadoop.hbase.protobuf.RequestConverter.buildGetRowOrBeforeRequest(RequestConverter.java:131) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1402) at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:701) at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:699) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120) at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:705) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326) at org.apache.hadoop.hbase.client.HTable.(HTable.java:192) Thanks Tridib From: d...@ocirs.com Date: Tue, 28 Apr 2015 22:24:39 -0700 Subject: Re: HBase HTable constructor hangs To: tridib.sama...@live.com In that case, something else is failing and the reason HBase looks like it hangs is that the hbase timeout or retry count is too high. Try setting the following conf and hbase will only hang for a few mins max and return a helpful error message. hbaseConf.set(HConstants.HBASE_CLIENT_RETRIES_NUMBER, "2") -- Dean Chen On Tue, Apr 28, 2015 at 10:18 PM, Tridib Samanta wrote: Nope, my hbase is unsecured. From: d...@ocirs.com Date: Tue, 28 Apr 2015 22:09:51 -0700 Subject: Re: HBase HTable constructor hangs To: tridib.sama...@live.com Hi Tridib, Are you running this on a secure Hadoop/HBase cluster? I ran in to a similar issue where the HBase client can successfully connect in local mode and in the yarn-client driver but not on remote executors. The problem is that Spark doesn't distribute the hbase auth key, see the following Jira ticket and PR. https://issues.apache.org/jira/browse/SPARK-6918 -- Dean Chen On Tue, Apr 28, 2015 at 9:34 PM, Tridib Samanta wrote: I am 100% sure how it's picking up the configuration. I copied the hbase-site.xml in hdfs/spark cluster (single machine). I also included hbase-site.xml in spark-job jar files. spark-job jar file also have yarn-site and mapred-site and core-site.xml in it. One interesting thing is, when I run the spark-job jar as standalone and execute the HBase client from a main method, it works fine. Same client unable to connect/hangs when the jar is distributed in spark. Thanks Tridib Date: Tue, 28 Apr 2015 21:25:41 -0700 Subject: Re: HBase HTable constructor hangs From: yuzhih...@gmail.com To: tridib.sama...@live.com CC: user@spark.apache.org How did you distribute hbase-site.xml to the nodes ? Looks like HConnectionManager couldn't find the hbase:meta server. Cheers On Tue, Apr 28, 2015 at 9:19 PM, Tridib Samanta wrote: I am using Spark 1.2.0 and HBase 0.98.1-cdh5.1.0. Here is the jstack trace. Complete stack trace attached. "Executor task launch worker-1" #58 daemon prio=5 os_prio=0 tid=0x7fd3d0445000 nid=0x488 waiting on condition [0x7fd4507d9000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:152) - locked <0xf8cb7258> (a org.apache.hadoop.hbase.client.RpcRetryingCaller) at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:705) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162) - locked <0xf84ac0b0> (a java.lang.
Re: HBase HTable constructor hangs
Can you verify whether the hbase release you're using has the following fix ? HBASE-8 non environment variable solution for "IllegalAccessError" Cheers On Tue, Apr 28, 2015 at 10:47 PM, Tridib Samanta wrote: > I turned on the TRACE and I see lot of following exception: > > java.lang.IllegalAccessError: com/google/protobuf/ZeroCopyLiteralByteString > at > org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:897) > at > org.apache.hadoop.hbase.protobuf.RequestConverter.buildGetRowOrBeforeRequest(RequestConverter.java:131) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1402) > at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:701) > at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:699) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120) > at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:705) > at > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011) > at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:192) > > Thanks > Tridib > > ---------- > From: d...@ocirs.com > Date: Tue, 28 Apr 2015 22:24:39 -0700 > Subject: Re: HBase HTable constructor hangs > To: tridib.sama...@live.com > > In that case, something else is failing and the reason HBase looks like it > hangs is that the hbase timeout or retry count is too high. > > Try setting the following conf and hbase will only hang for a few mins max > and return a helpful error message. > > hbaseConf.set(HConstants.HBASE_CLIENT_RETRIES_NUMBER, "2") > > > > > -- > Dean Chen > > On Tue, Apr 28, 2015 at 10:18 PM, Tridib Samanta > wrote: > > Nope, my hbase is unsecured. > > -- > From: d...@ocirs.com > Date: Tue, 28 Apr 2015 22:09:51 -0700 > Subject: Re: HBase HTable constructor hangs > To: tridib.sama...@live.com > > > Hi Tridib, > > Are you running this on a secure Hadoop/HBase cluster? I ran in to a > similar issue where the HBase client can successfully connect in local mode > and in the yarn-client driver but not on remote executors. The problem is > that Spark doesn't distribute the hbase auth key, see the following Jira > ticket and PR. > > https://issues.apache.org/jira/browse/SPARK-6918 > > > -- > Dean Chen > > On Tue, Apr 28, 2015 at 9:34 PM, Tridib Samanta > wrote: > > I am 100% sure how it's picking up the configuration. I copied the > hbase-site.xml in hdfs/spark cluster (single machine). I also included > hbase-site.xml in spark-job jar files. spark-job jar file also have > yarn-site and mapred-site and core-site.xml in it. > > One interesting thing is, when I run the spark-job jar as standalone and > execute the HBase client from a main method, it works fine. Same client > unable to connect/hangs when the jar is distributed in spark. > > Thanks > Tridib > > -- > Date: Tue, 28 Apr 2015 21:25:41 -0700 > > Subject: Re: HBase HTable constructor hangs > From: yuzhih...@gmail.com > To: tridib.sama...@live.com > CC: user@spark.apache.org > > How did you distribute hbase-site.xml to the nodes ? > > Looks like HConnectionManager couldn't find the hbase:meta server. > > Cheers > > On Tue, Apr 28, 2015 at 9:19 PM, Tridib Samanta > wrote: > > I am using Spark 1.2.0 and HBase 0.98.1-cdh5.1.0. > > Here is the jstack trace. Complete stack trace attached. > > "Executor task launch worker-1" #58 daemon prio=5 os_prio=0 > tid=0x7fd3d0445000 nid=0x488 waiting on condition [0x7fd4507d9000] >java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:152) > - locked <0xf8cb7258> (a > org.apache.hadoop.hbase.client.RpcRetryingCaller) > at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:705) > at > org.apache.hadoop.hbase.cli
RE: HBase HTable constructor hangs
I turned on the TRACE and I see lot of following exception: java.lang.IllegalAccessError: com/google/protobuf/ZeroCopyLiteralByteString at org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:897) at org.apache.hadoop.hbase.protobuf.RequestConverter.buildGetRowOrBeforeRequest(RequestConverter.java:131) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1402) at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:701) at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:699) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120) at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:705) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326) at org.apache.hadoop.hbase.client.HTable.(HTable.java:192) Thanks Tridib From: d...@ocirs.com Date: Tue, 28 Apr 2015 22:24:39 -0700 Subject: Re: HBase HTable constructor hangs To: tridib.sama...@live.com In that case, something else is failing and the reason HBase looks like it hangs is that the hbase timeout or retry count is too high. Try setting the following conf and hbase will only hang for a few mins max and return a helpful error message. hbaseConf.set(HConstants.HBASE_CLIENT_RETRIES_NUMBER, "2") -- Dean Chen On Tue, Apr 28, 2015 at 10:18 PM, Tridib Samanta wrote: Nope, my hbase is unsecured. From: d...@ocirs.com Date: Tue, 28 Apr 2015 22:09:51 -0700 Subject: Re: HBase HTable constructor hangs To: tridib.sama...@live.com Hi Tridib, Are you running this on a secure Hadoop/HBase cluster? I ran in to a similar issue where the HBase client can successfully connect in local mode and in the yarn-client driver but not on remote executors. The problem is that Spark doesn't distribute the hbase auth key, see the following Jira ticket and PR. https://issues.apache.org/jira/browse/SPARK-6918 -- Dean Chen On Tue, Apr 28, 2015 at 9:34 PM, Tridib Samanta wrote: I am 100% sure how it's picking up the configuration. I copied the hbase-site.xml in hdfs/spark cluster (single machine). I also included hbase-site.xml in spark-job jar files. spark-job jar file also have yarn-site and mapred-site and core-site.xml in it. One interesting thing is, when I run the spark-job jar as standalone and execute the HBase client from a main method, it works fine. Same client unable to connect/hangs when the jar is distributed in spark. Thanks Tridib Date: Tue, 28 Apr 2015 21:25:41 -0700 Subject: Re: HBase HTable constructor hangs From: yuzhih...@gmail.com To: tridib.sama...@live.com CC: user@spark.apache.org How did you distribute hbase-site.xml to the nodes ? Looks like HConnectionManager couldn't find the hbase:meta server. Cheers On Tue, Apr 28, 2015 at 9:19 PM, Tridib Samanta wrote: I am using Spark 1.2.0 and HBase 0.98.1-cdh5.1.0. Here is the jstack trace. Complete stack trace attached. "Executor task launch worker-1" #58 daemon prio=5 os_prio=0 tid=0x7fd3d0445000 nid=0x488 waiting on condition [0x7fd4507d9000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:152) - locked <0xf8cb7258> (a org.apache.hadoop.hbase.client.RpcRetryingCaller) at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:705) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162) - locked <0xf84ac0b0> (a java.lang.Object) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326) at org.apache.hadoop.hbase.client.HTable.(HTable.java:192) at org.apache.hadoop.hbase.client.HTable.(HTable.java:150) at com.mypackage.storeTuples(CubeStoreService.java:59) at com.my
RE: HBase HTable constructor hangs
I am 100% sure how it's picking up the configuration. I copied the hbase-site.xml in hdfs/spark cluster (single machine). I also included hbase-site.xml in spark-job jar files. spark-job jar file also have yarn-site and mapred-site and core-site.xml in it. One interesting thing is, when I run the spark-job jar as standalone and execute the HBase client from a main method, it works fine. Same client unable to connect/hangs when the jar is distributed in spark. Thanks Tridib Date: Tue, 28 Apr 2015 21:25:41 -0700 Subject: Re: HBase HTable constructor hangs From: yuzhih...@gmail.com To: tridib.sama...@live.com CC: user@spark.apache.org How did you distribute hbase-site.xml to the nodes ? Looks like HConnectionManager couldn't find the hbase:meta server. Cheers On Tue, Apr 28, 2015 at 9:19 PM, Tridib Samanta wrote: I am using Spark 1.2.0 and HBase 0.98.1-cdh5.1.0. Here is the jstack trace. Complete stack trace attached. "Executor task launch worker-1" #58 daemon prio=5 os_prio=0 tid=0x7fd3d0445000 nid=0x488 waiting on condition [0x7fd4507d9000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:152) - locked <0xf8cb7258> (a org.apache.hadoop.hbase.client.RpcRetryingCaller) at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:705) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162) - locked <0xf84ac0b0> (a java.lang.Object) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326) at org.apache.hadoop.hbase.client.HTable.(HTable.java:192) at org.apache.hadoop.hbase.client.HTable.(HTable.java:150) at com.mypackage.storeTuples(CubeStoreService.java:59) at com.mypackage.StorePartitionToHBaseStoreFunction.call(StorePartitionToHBaseStoreFunction.java:23) at com.mypackage.StorePartitionToHBaseStoreFunction.call(StorePartitionToHBaseStoreFunction.java:13) at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:195) at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:195) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:773) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:773) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1314) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1314) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:56) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "Executor task launch worker-0" #57 daemon prio=5 os_prio=0 tid=0x7fd3d0443800 nid=0x487 waiting for monitor entry [0x7fd4506d8000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1156) - waiting to lock <0xf84ac0b0> (a java.lang.Object) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326) at org.apache.hadoop.hbase.client.HTable.(HTable.java:192) at org.apache.hadoop.hbase.client.HTable.(HTable.java:150) at com.mypackage.storeTuples(CubeStoreService.java:59) at com.mypackage.StorePartitionToHBaseStoreFunction.call(StorePartitionToHBaseStoreFunction.java:23) at com.mypackage.StorePartitionToHBaseStoreFunction.call(StorePartitionToHBaseStoreFunction.java:13) at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:195) at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:195) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:773) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:773) at org.apache.spark.SparkContext$$ano
Re: HBase HTable constructor hangs
56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > -- > Date: Tue, 28 Apr 2015 19:35:26 -0700 > Subject: Re: HBase HTable constructor hangs > From: yuzhih...@gmail.com > To: tridib.sama...@live.com > CC: user@spark.apache.org > > Can you give us more information ? > Such as hbase release, Spark release. > > If you can pastebin jstack of the hanging HTable process, that would help. > > BTW I used > http://search-hadoop.com/?q=spark+HBase+HTable+constructor+hangs and saw > a very old thread with this subject. > > Cheers > > On Tue, Apr 28, 2015 at 7:12 PM, tridib wrote: > > I am exactly having same issue. I am running hbase and spark in docker > container. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/HBase-HTable-constructor-hangs-tp4926p22696.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > >
RE: HBase HTable constructor hangs
I am using Spark 1.2.0 and HBase 0.98.1-cdh5.1.0. Here is the jstack trace. Complete stack trace attached. "Executor task launch worker-1" #58 daemon prio=5 os_prio=0 tid=0x7fd3d0445000 nid=0x488 waiting on condition [0x7fd4507d9000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:152) - locked <0xf8cb7258> (a org.apache.hadoop.hbase.client.RpcRetryingCaller) at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:705) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:144) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1102) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1162) - locked <0xf84ac0b0> (a java.lang.Object) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326) at org.apache.hadoop.hbase.client.HTable.(HTable.java:192) at org.apache.hadoop.hbase.client.HTable.(HTable.java:150) at com.mypackage.storeTuples(CubeStoreService.java:59) at com.mypackage.StorePartitionToHBaseStoreFunction.call(StorePartitionToHBaseStoreFunction.java:23) at com.mypackage.StorePartitionToHBaseStoreFunction.call(StorePartitionToHBaseStoreFunction.java:13) at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:195) at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:195) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:773) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:773) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1314) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1314) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:56) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "Executor task launch worker-0" #57 daemon prio=5 os_prio=0 tid=0x7fd3d0443800 nid=0x487 waiting for monitor entry [0x7fd4506d8000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1156) - waiting to lock <0xf84ac0b0> (a java.lang.Object) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1054) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1011) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:326) at org.apache.hadoop.hbase.client.HTable.(HTable.java:192) at org.apache.hadoop.hbase.client.HTable.(HTable.java:150) at com.mypackage.storeTuples(CubeStoreService.java:59) at com.mypackage.StorePartitionToHBaseStoreFunction.call(StorePartitionToHBaseStoreFunction.java:23) at com.mypackage.StorePartitionToHBaseStoreFunction.call(StorePartitionToHBaseStoreFunction.java:13) at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:195) at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:195) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:773) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:773) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1314) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1314) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:56) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Date: Tue, 28 Apr 2015 19:35:26 -0700 Subject: Re: HBase HTable constructor hangs From: yuzhih...@gmail.com To: tridib.sama...@live.com CC: user@spark.apache.org Can you give us more information ?Such as hbase release, Spark release. If you can pastebin jstack of the hanging HTable process, that would hel
Re: HBase HTable constructor hangs
Can you give us more information ? Such as hbase release, Spark release. If you can pastebin jstack of the hanging HTable process, that would help. BTW I used http://search-hadoop.com/?q=spark+HBase+HTable+constructor+hangs and saw a very old thread with this subject. Cheers On Tue, Apr 28, 2015 at 7:12 PM, tridib wrote: > I am exactly having same issue. I am running hbase and spark in docker > container. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/HBase-HTable-constructor-hangs-tp4926p22696.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: HBase HTable constructor hangs
I am exactly having same issue. I am running hbase and spark in docker container. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/HBase-HTable-constructor-hangs-tp4926p22696.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
