Hmm. Looking at that error, my guess would be that there is an incorrect usage of TraceScope#detach going on somewhere in hbase... perhaps a double detach. But I could be wrong. We added some code recently to catch issues like this.
best, Colin On Wed, Feb 25, 2015 at 12:28 AM, Masatake Iwasaki <[email protected]> wrote: > I tried hbase-1 built against today's htrace-3.2.0-SNAPSHOT (with quick fix > to TestHTraceHooks). > I got the error below in regionserver log. > I will dig this tomorrow.:: > > 2015-02-25 00:18:29,270 ERROR [RS_OPEN_META-centos7:16201-0] > htrace.Tracer: Tried to detach trace span null but it has already been > detached. > 2015-02-25 00:18:29,271 ERROR [RS_OPEN_META-centos7:16201-0] > handler.OpenRegionHandler: Failed open of region=hbase:meta,,1.1588230740, > starting to roll back the global memstore size. > java.lang.RuntimeException: Tried to detach trace span null but it has > already been detached. > at org.apache.htrace.Tracer.clientError(Tracer.java:61) > at org.apache.htrace.TraceScope.detach(TraceScope.java:57) > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1559) > at > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:94) > at > org.apache.hadoop.hbase.regionserver.HRegion.writeRegionOpenMarker(HRegion.java:910) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4911) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4874) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4845) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4801) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4752) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > > On 2/24/15 18:27, Colin P. McCabe wrote: >> >> Thanks for trying this, Mastake. I've got HDFS working on my cluster >> with tracing and LocalFileSpanReceiver. Did you try using HBase + >> HDFS with LocalFileSpanReceiver? Be sure to use a build including >> HTRACE-112 since LFSR was kind of busted prior to that. >> >> I'm going to do a longer writeup about getting HDFS + HBase working >> with other span receivers just as soon as I finish stomping a few more >> bugs. >> >> best, >> Colin >> >> On Tue, Feb 24, 2015 at 12:04 PM, Masatake Iwasaki >> <[email protected]> wrote: >>> >>> Hi, >>> >>> Thanks for trying this. I am sorry for late reply. >>> >>> I tried this today >>> by hbase-1.0.1-SANPSHOT built with >>> {{-Dhadoop-two.version=2.7.0-SNAPSHOT}} >>> in pseudo distributed cluster >>> but failed to get end-to-end trace. >>> >>> I checked that >>> * tracing works for both of hbase and hdfs, >>> * hbase runs with 2.7.0-SNAPSHOT jar of hadoop. >>> >>> When I did do put with tracing on, >>> I saw span named "FSHLog.sync" with annotations such as >>> "syncing writer" and "writer synced". >>> The code for tracing in FSHLog worked at least. >>> >>> I'm still looking into this. >>> If it turned out that tracing spans are not reached to >>> actual HDFS writer thread in HBase, I will file a JIRA. >>> >>> # We need hadoop-2.6.0 or higher in order to trace HDFS. >>> # Building hbase from source with {{-Dhadoop-two.version=2.6.0}} >>> # is straight forward way to do this >>> # because the binary release of hbase-1.0.0 bundles hadoop-2.5.1 jars. >>> >>> Masatake >>> >>> >>> On 2/11/15 08:56, Nick Dimiduk wrote: >>>> >>>> Hi Joshua, >>>> >>>> In theory there's nothing special for you to do. Just issue your query >>>> to >>>> HBase with tracing enabled. The active span will go through HBase, down >>>> into HDFS, and back again. You'll need both systems collecting spans >>>> into >>>> the same place so that you can report on the complete trace tree. >>>> >>>> I've not recently tested the end-to-end, but I believe it's all there. >>>> If >>>> not, it's a bug -- this is an intended use case. Can you give it a try >>>> and let us know how it goes? >>>> >>>> FYI, 0.99.x are preview releases of HBase and not for production use. >>>> Just >>>> so you know :) >>>> >>>> -n >>>> >>>> On Wednesday, February 11, 2015, Chunxu Tang <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Now I’m exploiting HTrace to trace request level data flows in HBase >>>>> and >>>>> HDFS. I have successfully traced HBase and HDFS by using HTrace, >>>>> respectively. >>>>> >>>>> After that, I combine HBase and HDFS together and I want to just send a >>>>> PUT/GET request to HBase, but to trace the whole data flow in both >>>>> HBase >>>>> and HDFS. In my opinion, when I send a request such as Get to HBase, it >>>>> will at last try to read the blocks on HDFS, so I can construct a whole >>>>> data flow tracing through HBase and HDFS. While, the fact is that I can >>>>> only get tracing data of HBase, with no data of HDFS. >>>>> >>>>> Could you give me any suggestions on how to trace the data flow in both >>>>> HBase and HDFS? Does anyone have similar experience? Do I need to >>>>> modify >>>>> the source code? And maybe which part(s) should I touch? If I need to >>>>> modify the code, I will try to create a patch for that. >>>>> >>>>> Thank you. >>>>> >>>>> My Configurations: >>>>> Hadoop version: 2.6.0 >>>>> HBase version: 0.99.2 >>>>> HTrace version: htrace-master >>>>> OS: Ubuntu 12.04 >>>>> >>>>> >>>>> Joshua >>>>> >
