March 2015 Report
Greetings, {podling} developers,
The marvin automated report reminder script didn't fire for whatever
reason this week, so I'm sending out a bulk reminder manually for the 18
podlings who are expected to submit a March 2015 report. Boilerplate
repinder text below.
Best,
Marvin Humphrey
---
Dear podling,
This email was sent by an automated system on behalf of the Apache
Incubator PMC. It is an initial reminder to give you plenty of time to
prepare your quarterly board report.
The board meeting is scheduled for Wed, 18 March 2015, 10:30 am PST.
The report for your podling will form a part of the Incubator PMC
report. The Incubator PMC requires your report to be submitted 2 weeks
before the board meeting, to allow sufficient time for review and
submission (Wed, March 4th).
Please submit your report with sufficient time to allow the incubator
PMC, and subsequently board members to review and digest. Again, the
very latest you should submit your report is 2 weeks prior to the board
meeting.
Thanks,
The Apache Incubator PMC
Submitting your Report
--
Your report should contain the following:
* Your project name
* A brief description of your project, which assumes no knowledge of
the project or necessarily of its field
* A list of the three most important issues to address in the move
towards graduation.
* Any issues that the Incubator PMC or ASF Board might wish/need to be
aware of
* How has the community developed since the last report
* How has the project developed since the last report.
This should be appended to the Incubator Wiki page at:
http://wiki.apache.org/incubator/March2015
Note: This is manually populated. You may need to wait a little before
this page is created from a template.
Mentors
---
Mentors should review reports for their project(s) and sign them off on
the Incubator wiki page. Signing off reports shows that you are
following the project - projects that are not signed may raise alarms
for the Incubator PMC.
Incubator PMC
march incubator report
Looks like HTrace is scheduled to report in March, but I don't think we got a notification. Due date is Wednesday, March 4th: https://wiki.apache.org/incubator/March2015
[REPORT] HTrace March 2015
Hi Folks, Please see below, this has been added to the wiki HTraceHTrace is a tracing framework intended for use with distributed systemswritten in java.HTrace has been incubating since 2014-11.Three most important issues to address in the move towards graduation: 1. Continue to grow the HTrace community 2. Continue to develop and release stable HTrace incubating artifacts 3. Continue to explore the integration of the HTrace framework into other Apache productsAny issues that the Incubator PMC (IPMC) or ASF Board wish/need to beaware of?NoHow has the community developed since the last report?There has been a bunch of mailing list activity relating directlyto issue 3 above e.g. better integration of HTrace into HBase/HDFS.HTrace is being represented at ApacheCon 2015 NA in April with a presentationIntroducing Apache HTrace: An End-to-End Tracing Framework for Distributed Systems - Colin McCabe, Cloudera - http://sched.co/2P8QHow has the project developed since the last report?The codebase has seen about 30 odd commits since last reporting.Jira continues to see activity which is encouraging as HTrace communityprogresses towards next incubating release.Date of last release: 2015-20-01 htrace-3.1.0-incubatingWhen were the last committers or PMC members elected?Abraham Elmahrek was elected to become an HTrace committer onWed, 11 Feb, 2-15.Signed-off-by: [ ](htrace) Jake Farrell [ ](htrace) Todd Lipcon [X](htrace) Lewis John Mcgibbney [ ](htrace) Andrew Purtell [ ](htrace) Billie Rinaldi [ ](htrace) Michael StackShepherd/Mentor notes: Ta Lewis -- *Lewis*
Re: [REPORT] HTrace March 2015
Looks good. Thanks +1 On Thu, Feb 26, 2015 at 12:41 PM, Stack st...@duboce.net wrote: Thanks Lewis. +1 St.Ack On Thu, Feb 26, 2015 at 10:58 AM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Folks, Please see below, this has been added to the wiki HTraceHTrace is a tracing framework intended for use with distributed systemswritten in java.HTrace has been incubating since 2014-11.Three most important issues to address in the move towards graduation: 1. Continue to grow the HTrace community 2. Continue to develop and release stable HTrace incubating artifacts 3. Continue to explore the integration of the HTrace framework into other Apache productsAny issues that the Incubator PMC (IPMC) or ASF Board wish/need to beaware of?NoHow has the community developed since the last report?There has been a bunch of mailing list activity relating directlyto issue 3 above e.g. better integration of HTrace into HBase/HDFS.HTrace is being represented at ApacheCon 2015 NA in April with a presentationIntroducing Apache HTrace: An End-to-End Tracing Framework for Distributed Systems - Colin McCabe, Cloudera - http://sched.co/2P8QHow has the project developed since the last report?The codebase has seen about 30 odd commits since last reporting.Jira continues to see activity which is encouraging as HTrace communityprogresses towards next incubating release.Date of last release: 2015-20-01 htrace-3.1.0-incubatingWhen were the last committers or PMC members elected?Abraham Elmahrek was elected to become an HTrace committer onWed, 11 Feb, 2-15.Signed-off-by: [ ](htrace) Jake Farrell [ ](htrace) Todd Lipcon [X](htrace) Lewis John Mcgibbney [ ](htrace) Andrew Purtell [ ](htrace) Billie Rinaldi [ ](htrace) Michael StackShepherd/Mentor notes: Ta Lewis -- *Lewis*
Re: HTrace for Nutch 2.x Search Stack
Hi Lewis, The results of TRACE level logging will give you messages from the various system components according to whatever those developers thought relevant for exposing from the application. Assembling the HTrace spans from a single trace will give you something similar, but structured according to the major boundaries within the system. The original intention is for tracing a single request through multiple RPC's across multiple systems. At the very least, you'd see these boundary points and the amount of time spent in each component. Each component's developers are free to use the HTrace API to attach annotations to their trace spans. These annotations are arbitrary messages, accompanied by timestamps, included by the whims of the projects' developers. I guess the big difference is that trace spans collected by HTrace are a bit more structured than simple log messages. There's a parent-child relationship between spans, and this hierarchy exists across threads within a process and across processes within a distributed application. Maybe someone else can do a better job explaining... -n On Thu, Feb 26, 2015 at 12:57 PM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Folks, Right now we are able to initiate TRACE within Log4j file in Nutch 2.X [0]. Nutch 2.X offer storage abstraction for storage of Webpage and Host data through use of Apache Gora. Gora backend support includes - Apache Avro 1.7.6 - Apache Hadoop 1.2.1 and 2.5.2 - Apache HBase 0.98.8-hadoop2 - Apache Cassandra 2.0.2 - Apache Solr 4.10.3 - MongoDB 2.6.X - Apache Accumlo 1.5.1 Parts of the Nutch 2.X search stack include Nutch, Gora, optionally Solr and/or Elastic Search for indexing context and optionally Hadoop for running it all on. As a developer of Nutch and Gora am very keen to see a more verbose tracing mechanism (potentially with some visualization) for debugging our search stack deployments. Can I please kick off conversation on what the difference and benefits of H Trace are over log4j TRACE level logging. Once I understand the above, I would like to obtain advice on the integration points for having HTrace as the tracing mechanism for the stack I describe above. Thanks Lewis [0] https://github.com/apache/nutch/blob/2.x/conf/log4j.properties -- *Lewis*
Re: [REPORT] HTrace March 2015
+1 by me. On Thu, Feb 26, 2015 at 10:58 AM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Folks, Please see below, this has been added to the wiki HTraceHTrace is a tracing framework intended for use with distributed systemswritten in java.HTrace has been incubating since 2014-11.Three most important issues to address in the move towards graduation: 1. Continue to grow the HTrace community 2. Continue to develop and release stable HTrace incubating artifacts 3. Continue to explore the integration of the HTrace framework into other Apache productsAny issues that the Incubator PMC (IPMC) or ASF Board wish/need to beaware of?NoHow has the community developed since the last report?There has been a bunch of mailing list activity relating directlyto issue 3 above e.g. better integration of HTrace into HBase/HDFS.HTrace is being represented at ApacheCon 2015 NA in April with a presentationIntroducing Apache HTrace: An End-to-End Tracing Framework for Distributed Systems - Colin McCabe, Cloudera - http://sched.co/2P8QHow has the project developed since the last report?The codebase has seen about 30 odd commits since last reporting.Jira continues to see activity which is encouraging as HTrace communityprogresses towards next incubating release.Date of last release: 2015-20-01 htrace-3.1.0-incubatingWhen were the last committers or PMC members elected?Abraham Elmahrek was elected to become an HTrace committer onWed, 11 Feb, 2-15.Signed-off-by: [ ](htrace) Jake Farrell [ ](htrace) Todd Lipcon [X](htrace) Lewis John Mcgibbney [ ](htrace) Andrew Purtell [ ](htrace) Billie Rinaldi [ ](htrace) Michael StackShepherd/Mentor notes: Ta Lewis -- *Lewis*
Re: Trace HBase/HDFS with HTrace
perhaps a double detach.
It turned out to be double detach of NullScope singleton and I filed
HTRACE-119.
I got the error on meta assignments on regionserver startup without
starting trace spans..
Thanks,
Masatake
On 2/26/15 09:58, Colin P. McCabe wrote:
Hmm. Looking at that error, my guess would be that there is an
incorrect usage of TraceScope#detach going on somewhere in hbase...
perhaps a double detach. But I could be wrong. We added some code
recently to catch issues like this.
best,
Colin
On Wed, Feb 25, 2015 at 12:28 AM, Masatake Iwasaki
iwasak...@oss.nttdata.co.jp wrote:
I tried hbase-1 built against today's htrace-3.2.0-SNAPSHOT (with quick fix
to TestHTraceHooks).
I got the error below in regionserver log.
I will dig this tomorrow.::
2015-02-25 00:18:29,270 ERROR [RS_OPEN_META-centos7:16201-0]
htrace.Tracer: Tried to detach trace span null but it has already been
detached.
2015-02-25 00:18:29,271 ERROR [RS_OPEN_META-centos7:16201-0]
handler.OpenRegionHandler: Failed open of region=hbase:meta,,1.1588230740,
starting to roll back the global memstore size.
java.lang.RuntimeException: Tried to detach trace span null but it has
already been detached.
at org.apache.htrace.Tracer.clientError(Tracer.java:61)
at org.apache.htrace.TraceScope.detach(TraceScope.java:57)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1559)
at
org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:94)
at
org.apache.hadoop.hbase.regionserver.HRegion.writeRegionOpenMarker(HRegion.java:910)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4911)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4874)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4845)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4801)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4752)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
On 2/24/15 18:27, Colin P. McCabe wrote:
Thanks for trying this, Mastake. I've got HDFS working on my cluster
with tracing and LocalFileSpanReceiver. Did you try using HBase +
HDFS with LocalFileSpanReceiver? Be sure to use a build including
HTRACE-112 since LFSR was kind of busted prior to that.
I'm going to do a longer writeup about getting HDFS + HBase working
with other span receivers just as soon as I finish stomping a few more
bugs.
best,
Colin
On Tue, Feb 24, 2015 at 12:04 PM, Masatake Iwasaki
iwasak...@oss.nttdata.co.jp wrote:
Hi,
Thanks for trying this. I am sorry for late reply.
I tried this today
by hbase-1.0.1-SANPSHOT built with
{{-Dhadoop-two.version=2.7.0-SNAPSHOT}}
in pseudo distributed cluster
but failed to get end-to-end trace.
I checked that
* tracing works for both of hbase and hdfs,
* hbase runs with 2.7.0-SNAPSHOT jar of hadoop.
When I did do put with tracing on,
I saw span named FSHLog.sync with annotations such as
syncing writer and writer synced.
The code for tracing in FSHLog worked at least.
I'm still looking into this.
If it turned out that tracing spans are not reached to
actual HDFS writer thread in HBase, I will file a JIRA.
# We need hadoop-2.6.0 or higher in order to trace HDFS.
# Building hbase from source with {{-Dhadoop-two.version=2.6.0}}
# is straight forward way to do this
# because the binary release of hbase-1.0.0 bundles hadoop-2.5.1 jars.
Masatake
On 2/11/15 08:56, Nick Dimiduk wrote:
Hi Joshua,
In theory there's nothing special for you to do. Just issue your query
to
HBase with tracing enabled. The active span will go through HBase, down
into HDFS, and back again. You'll need both systems collecting spans
into
the same place so that you can report on the complete trace tree.
I've not recently tested the end-to-end, but I believe it's all there.
If
not, it's a bug -- this is an intended use case. Can you give it a try
and let us know how it goes?
FYI, 0.99.x are preview releases of HBase and not for production use.
Just
so you know :)
-n
On Wednesday, February 11, 2015, Chunxu Tang chunxut...@gmail.com
wrote:
Hi all,
Now I’m exploiting HTrace to trace request level data flows in HBase
and
HDFS. I have successfully traced HBase and HDFS by using HTrace,
respectively.
After that, I combine HBase and HDFS together and I want to just send a
PUT/GET request to
Re: [REPORT] HTrace March 2015
+1. Thanks, Lewis. C. On Thu, Feb 26, 2015 at 10:58 AM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Folks, Please see below, this has been added to the wiki HTraceHTrace is a tracing framework intended for use with distributed systemswritten in java.HTrace has been incubating since 2014-11.Three most important issues to address in the move towards graduation: 1. Continue to grow the HTrace community 2. Continue to develop and release stable HTrace incubating artifacts 3. Continue to explore the integration of the HTrace framework into other Apache productsAny issues that the Incubator PMC (IPMC) or ASF Board wish/need to beaware of?NoHow has the community developed since the last report?There has been a bunch of mailing list activity relating directlyto issue 3 above e.g. better integration of HTrace into HBase/HDFS.HTrace is being represented at ApacheCon 2015 NA in April with a presentationIntroducing Apache HTrace: An End-to-End Tracing Framework for Distributed Systems - Colin McCabe, Cloudera - http://sched.co/2P8QHow has the project developed since the last report?The codebase has seen about 30 odd commits since last reporting.Jira continues to see activity which is encouraging as HTrace communityprogresses towards next incubating release.Date of last release: 2015-20-01 htrace-3.1.0-incubatingWhen were the last committers or PMC members elected?Abraham Elmahrek was elected to become an HTrace committer onWed, 11 Feb, 2-15.Signed-off-by: [ ](htrace) Jake Farrell [ ](htrace) Todd Lipcon [X](htrace) Lewis John Mcgibbney [ ](htrace) Andrew Purtell [ ](htrace) Billie Rinaldi [ ](htrace) Michael StackShepherd/Mentor notes: Ta Lewis -- *Lewis*
