Re: [DISCUSS] Subproject proposal

[Colin McCabe]

I would potentially be interested in continue to be involved with HTrace as a 
subproject.

The vision behind HTrace was always to have a single trace system that unified 
all of Hadoop.  So you could see what Accumulo was doing and how that affected 
HDFS, or what Phoenix was doing that affected HBase and HDFS, etc. etc.  This 
has sort of been built several times internally by companies running services 
based on Hadoopy projects, but never really made its way into open source in a 
meaningful way.  I thought we had a good shot at that, but maybe we needed to 
start earlier and have more resources.  We especially lacked full-time 
developers and people to evangelize the client.

I think it makes the most sense for HTrace to be a subproject of either Apache 
Hadoop or Apache Skywalking.  Skywalking in particular seems interesting since 
its goals are very similar to HTrace's -- to be a one-stop shop including 
tracing clients, visualization, and storage.  Perhaps HTraced could be useful 
to them for improving that "first 15 minute experience".  It's easy to start up 
and doesn't require managing a separate storage or query system.

I'm not so sure about HTrace being a subproject of Accumulo.  It seems like 
Accumulo is really focused on being a storage system, not so much on being a 
platform.  It would be weird for HBase or HDFS to depend on something that was 
a subproject of Accumulo, for example.

best,
Colin


On Wed, Mar 14, 2018, at 17:35, Michael Wall wrote:
> I am interested.  I am not thinking about it as subproject under Accumulo
> though, just to be clear.  Just looked at Skywalking for the first time,
> seems intriguing.
> 
> On Wed, Mar 14, 2018 at 7:32 PM Mike Drob  wrote:
> 
> > On Wed, Mar 14, 2018, 2:26 PM Billie Rinaldi 
> > wrote:
> >
> > > In the active thread "[VOTE] Retire HTrace from Incubation" Christopher
> > > Tubbs brought up the idea to make HTrace a subproject of an existing TLP.
> >
> > This would mitigate the issues of the community being inactive and the core
> > > instrumentation library not requiring ongoing development.
> >
> >
> > Does moving to a subproject out another tlp necessitate changing Java
> > package names prior to release? That would put a damper on user adoption
> > again.
> >
> > It's a choice we could make now (assuming we were able to find a TLP
> > > willing to adopt HTrace
> >
> > as a subproject),
> >
> > The Skywalking podling expressed some interest in the vote thread.
> >
> >
> >
> > or we could allow HTrace to retire and then revisit the
> > > subproject idea at a future time if someone becomes interested in
> > patching
> > > and releasing a new version of HTrace.
> > >
> > > So far, the people who have expressed interest in being involved with
> > > HTrace as a possible subproject are Christopher, Masatake, and myself. Is
> > > anyone else in the community interested in this idea?
> > >
> >

Re: Current status?

[Colin McCabe]

On Fri, Dec 15, 2017, at 11:31, Stack wrote:
> Yeah. Its not looking good. No work done since the Mike Drob prompted
> thread on to-attic or not.
> 
> If not progress in March of next year, lets put it out of its misery? I say
> this because I have a (totally ridiculous probably) thought that I can make
> some progress here in new year.
> 
> If that too much to ask, I won't raise objection of others want to send
> htrace to the attic.

That makes sense.  I think we should be clear that the progress we would like 
to see is in downstream integration, rather than major changes to the library 
itself.  Using current versions of the library in HBase and Accumulo would be a 
good start.

best,
Colin

> 
> Thanks,
> S
> 
> On Sun, Dec 10, 2017 at 6:32 AM, John D. Ament 
> wrote:
> 
> > Dear HTrace Podling,
> >
> > I was wondering what your current status is.  There is no report filed,
> > and very low on list activity.  Is it time to begin retirement discussions?
> >
> > John
> >

Re: HTrace activity in HBase and Hadoop

[Colin McCabe]

Cool.  I'm asking about the JIRA now (and hopefully I can get a few more
folks to join the mailing list to explain for themselves).

cheers,
Colin


On Mon, Oct 2, 2017, at 02:46, Raam Rosh-Hai wrote:
> Thanks for the update Colin, can you link to the S3 connectors issues? I
> might be able to dedicate some time to it in the coming weeks.
> 
> Cheers,
> Raam
> 
> On 1 October 2017 at 03:04, Colin McCabe <cmcc...@apache.org> wrote:
> 
> > Hi all,
> >
> > I wanted to post a bit about some HTrace activity that's been going on
> > lately.  Some of this stuff is not as visible, because it's going on in
> > other git repos, rather than in HTrace itself.  But it's actually just
> > as important, or even more important, than changes to the HTrace git
> > repo itself.
> >
> > In HBASE-18601, Tamas Penzes is working on updating the version of
> > HTrace that HBase uses to HTrace 4.x.  This is very important for
> > getting HBase and HDFS on the same version (version 4) and for letting
> > us tell end-to-end tracing stories.  The JIRA has been around for a
> > while, but the patch has been worked on a lot recently, and hopefully
> > we'll be able to get it in soon.
> >
> > In Hadoop, we've been talking a bit about how to better integrate
> > tracing into the s3 connectors.  This work hasn't really started, but I
> > think it will happen and be very worthwhile.
> >
> > best,
> > Colin
> >

HTrace activity in HBase and Hadoop

[Colin McCabe]

Hi all,

I wanted to post a bit about some HTrace activity that's been going on
lately.  Some of this stuff is not as visible, because it's going on in
other git repos, rather than in HTrace itself.  But it's actually just
as important, or even more important, than changes to the HTrace git
repo itself.

In HBASE-18601, Tamas Penzes is working on updating the version of
HTrace that HBase uses to HTrace 4.x.  This is very important for
getting HBase and HDFS on the same version (version 4) and for letting
us tell end-to-end tracing stories.  The JIRA has been around for a
while, but the patch has been worked on a lot recently, and hopefully
we'll be able to get it in soon.

In Hadoop, we've been talking a bit about how to better integrate
tracing into the s3 connectors.  This work hasn't really started, but I
think it will happen and be very worthwhile.

best,
Colin

Re: [DISCUSS] Attic podling Apache HTrace?

[Colin McCabe]

On Wed, Sep 13, 2017, at 09:01, Mike Drob wrote:
> Hi folks,
> 
> Want to pull this conversation back into people's minds to summarize and
> reposit some questions.
> 
> It looks like there is consensus that 1) HTrace is generally useful but
> 2)
> needs to refocus on more specific use cases rather than trying to be a
> general purpose tool and 3) suffers from being niche on niche.
> 
> There have been lots of folks volunteering to contribute, but I don't see
> any resulting JIRA or mailing list activity. We all can have the best of
> intentions while still not having the necessary time available to follow
> through, myself included.
> 
> Projects can still be successful outside of the ASF; retiring as a
> podling
> does not preclude others from picking up the works. As one example, I
> have
> seen many vibrant communities run as github projects with a core set of
> dedicated contributors. I don't believe that living in the ASF makes it
> any
> easier or harder for other Apache projects such as Hadoop to integrate
> with
> us, so long as the licensing remains compatible. In the future, if a
> diverse community springs up again and finds the desire to come back, I
> don't think that will be a problem to re-incubate at that point either.

Hi Mike,

I'm on vacation this week, so I apologize for the brevity (and delayed
responses).

I think it would make more sense to fold HTrace into Hadoop if we wanted
to disband the podling.  At a minimum, there is value in allowing Hadoop
to use the trace systems that are out there (Zipkin, OpenTracing, etc.)
in a software-configurable way.

But I still hope we can find people interested in moving tracing forward
in hadoop. Let's circle back on this in a little bit...

Colin


> 
> Thoughts?
> Mike
> 
> 
> On Thu, Aug 31, 2017 at 3:32 AM, Raam Rosh-Hai <r...@findhotel.net>
> wrote:
> 
> > Thank you Colin for your response,
> > I was actually referring to Adrian's comment on making Htrace and Zipkin
> > play more nicely together, I have been using Htrace with zipkin for a while
> > now and I am doing pretty well but I was wondering if there are some
> > glaring pain points that should be attended.
> >
> > On 30 August 2017 at 23:41, Colin McCabe <cmcc...@apache.org> wrote:
> >
> > > On Mon, Aug 21, 2017, at 02:09, Raam Rosh-Hai wrote:
> > > > I tend to agree that zipkin should be used as a frontend and I am
> > pretty
> > > > sure I will have some time to advance that in the coming weeks if one
> > of
> > > > the experts could create a few tasks.
> > >
> > > Hi Raam,
> > >
> > > If you're interested in this, check out the Zipkin trace sink.  That
> > > allows you to send HTrace spans to Zipkin.
> > >
> > > >
> > > > I think no one touched the obvious thing that tracing lacks any kind of
> > > > hype, it's just not glamours and in order to want to use such a
> > framework
> > > > you need to be versed in this world and to actually suffer from
> > > > distributed
> > > > debugging.
> > >
> > > Yeah, I think this is an area where some publicity and outreach would be
> > > a good thing :)
> > >
> > > It would help a lot if we could identify some core use-cases, like Stack
> > > said, and make sure everything works great for those.
> > >
> > > Colin
> > >
> > >
> > > >
> > > > so yes, I think this project deserves one last push before putting it
> > in
> > > > the attic and I would be happy to complete a few issues in order to
> > make
> > > > this happen.
> > > >
> > > >
> > > >
> > > > On 19 August 2017 at 03:55, Adrian Cole <adrian.f.c...@gmail.com>
> > wrote:
> > > >
> > > > > > Thanks Adrian for the editorial on the landscape. Helps, especially
> > > > > coming
> > > > > > from yourself.
> > > > > we aim to please
> > > > >
> > > > > > Given current state of the project, a retrofit to come up on OT is
> > > not
> > > > > the
> > > > > > solution to the topic-at-hand (and besides I have a colored opinion
> > > on
> > > > > > taking on the API of another after spending a bunch of time
> > recently
> > > > > > undoing our mistake letting third-party Interfaces and Classes show
> > > > > through
> > > > > > in hbase).
> > > > > sensible for any api highly disconnected from

Re: Podling Report Reminder - September 2017

[Colin McCabe]

Thanks, Stack!  You're the best.

+1.



On Thu, Sep 7, 2017, at 15:10, Stack wrote:
> Ok. I pushed it (and signed off on the report too... You want to Billie?)
> 
> Thanks,
> S
> 
> On Thu, Sep 7, 2017 at 3:04 PM, Billie Rinaldi  wrote:
> 
> > +1. Thanks, Stack :)
> >
> > On Thu, Sep 7, 2017 at 2:44 PM, Stack  wrote:
> >
> > > Report is due. How is this? I'd like to put it up today since we are late
> > > already.
> > > Thanks,
> > > St.Ack
> > >
> > >
> > >
> > >
> > >
> > > HTrace
> > >
> > > HTrace is a tracing framework intended for use with distributed systems
> > > written in java.
> > >
> > > HTrace has been incubating since 2014-11-11.
> > >
> > > Three most important issues to address in the move towards graduation:
> > >
> > >  1. Continue to grow the Apache HTrace community
> > >  2. Continue to explore projects integrating Apache HTrace
> > >  3. Continue to develop and release stable Apache HTrace incubating
> > > artifacts
> > >
> > > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> > aware
> > > of?
> > >
> > >  In a DISCUSSION thread on our dev list [1], one of us suggested
> > >  moving the podling htrace to the attic because of inactivity.
> > >
> > >  A healthy back and forth suggested that we are not
> > >  dead yet, that we cut back our scope radically, and
> > >  that we give it another try before calling it quits.
> > >
> > > 1. https://s.apache.org/9ybr
> > >
> > > How has the community developed since the last report?
> > >
> > >  Mailing lists continue quiet (bar the above noted thread).
> > >
> > > How has the project developed since the last report?
> > >
> > >  No development.
> > >
> > > How would you assess the podling's maturity? Please feel free to add your
> > > own
> > > commentary.
> > >
> > >  [ ] Initial setup
> > >  [ ] Working towards first release
> > >  [x] Community building
> > >  [ ] Nearing graduation
> > >  [ ] Other:
> > >
> > > Date of last release:
> > >
> > >  2016-10-06
> > >
> > > When were the last committers or PPMC members elected?
> > >
> > >  2016-10-03
> > >
> > > On Tue, Sep 5, 2017 at 6:40 PM,  wrote:
> > >
> > > > Dear podling,
> > > >
> > > > This email was sent by an automated system on behalf of the Apache
> > > > Incubator PMC. It is an initial reminder to give you plenty of time to
> > > > prepare your quarterly board report.
> > > >
> > > > The board meeting is scheduled for Wed, 20 September 2017, 10:30 am
> > PDT.
> > > > The report for your podling will form a part of the Incubator PMC
> > > > report. The Incubator PMC requires your report to be submitted 2 weeks
> > > > before the board meeting, to allow sufficient time for review and
> > > > submission (Wed, September 06).
> > > >
> > > > Please submit your report with sufficient time to allow the Incubator
> > > > PMC, and subsequently board members to review and digest. Again, the
> > > > very latest you should submit your report is 2 weeks prior to the board
> > > > meeting.
> > > >
> > > > Thanks,
> > > >
> > > > The Apache Incubator PMC
> > > >
> > > > Submitting your Report
> > > >
> > > > --
> > > >
> > > > Your report should contain the following:
> > > >
> > > > *   Your project name
> > > > *   A brief description of your project, which assumes no knowledge of
> > > > the project or necessarily of its field
> > > > *   A list of the three most important issues to address in the move
> > > > towards graduation.
> > > > *   Any issues that the Incubator PMC or ASF Board might wish/need to
> > be
> > > > aware of
> > > > *   How has the community developed since the last report
> > > > *   How has the project developed since the last report.
> > > > *   How does the podling rate their own maturity.
> > > >
> > > > This should be appended to the Incubator Wiki page at:
> > > >
> > > > https://wiki.apache.org/incubator/September2017
> > > >
> > > > Note: This is manually populated. You may need to wait a little before
> > > > this page is created from a template.
> > > >
> > > > Mentors
> > > > ---
> > > >
> > > > Mentors should review reports for their project(s) and sign them off on
> > > > the Incubator wiki page. Signing off reports shows that you are
> > > > following the project - projects that are not signed may raise alarms
> > > > for the Incubator PMC.
> > > >
> > > > Incubator PMC
> > > >
> > >
> >

Re: [DISCUSS] Attic podling Apache HTrace?

[Colin McCabe]

On Mon, Aug 21, 2017, at 02:09, Raam Rosh-Hai wrote:
> I tend to agree that zipkin should be used as a frontend and I am pretty
> sure I will have some time to advance that in the coming weeks if one of
> the experts could create a few tasks.

Hi Raam,

If you're interested in this, check out the Zipkin trace sink.  That
allows you to send HTrace spans to Zipkin.

> 
> I think no one touched the obvious thing that tracing lacks any kind of
> hype, it's just not glamours and in order to want to use such a framework
> you need to be versed in this world and to actually suffer from
> distributed
> debugging.

Yeah, I think this is an area where some publicity and outreach would be
a good thing :)

It would help a lot if we could identify some core use-cases, like Stack
said, and make sure everything works great for those.

Colin


> 
> so yes, I think this project deserves one last push before putting it in
> the attic and I would be happy to complete a few issues in order to make
> this happen.
> 
> 
> 
> On 19 August 2017 at 03:55, Adrian Cole  wrote:
> 
> > > Thanks Adrian for the editorial on the landscape. Helps, especially
> > coming
> > > from yourself.
> > we aim to please
> >
> > > Given current state of the project, a retrofit to come up on OT is not
> > the
> > > solution to the topic-at-hand (and besides I have a colored opinion on
> > > taking on the API of another after spending a bunch of time recently
> > > undoing our mistake letting third-party Interfaces and Classes show
> > through
> > > in hbase).
> > sensible for any api highly disconnected from the ecosystem,
> > especially without practice yet.
> >
> > > I appreciate the higher-level point made by Andrew, that it is hard to
> > > thread a cross-cutting library across the Hadoop landscape whether
> > because
> > > releases happen on the geologic time scale or that there is little by way
> > > of coordination.
> > I think this is indeed leading a path towards focus, eg the H in Htrace :)
> >
> > > Can we do a focused 'win' like Colin suggests? E.g. hook up hbase and
> > hdfs
> > > end-to-end with connection to a viewer (zipkin? Or text dumps in a
> > > webpage?). A while back I had a go at the hbase side but it was burning
> > up
> > > the hours just getting it hooked up w/ tests to scream if any spans were
> > > broken in a refactor. I had to put it aside.
> > Incidentally, I wouldn't necessarily say Zipkin is ready out of box
> > because htrace UI and query is more advanced (in some ways due to some
> > data storage options we have available). So, something like this could
> > be a move of focus which would require investment on the other side to
> > avail features needed, or discuss how to upgrade into them (ex if
> > using hbase storage, certain queries would work). It is fair to say
> > zipkin has  a great devops pipeline, we are good at fixing things. At
> > the same time, we are imperfect in impl and inexperienced in hadoop
> > ecosystem. Having some way to join together could be really
> > beneficial, at the cost of up-front effort (due to model, UI and
> > storage differences). I would be happy to direct time, though would
> > need some help because of my irrelevance in the data services space
> > (something this might correct!)
> >
> > > Like the rest of you, my time is a little occupied elsewhere these times
> > so
> > > I can't revive the project, not at the moment at least.
> > ack
> >

Re: [DISCUSS] Attic podling Apache HTrace?

[Colin McCabe]

On Thu, Aug 17, 2017, at 14:40, Andrew Purtell wrote:
> > That's not the issue.  We already have HTrace integration with Hadoop
> RPC, such that a Hadoop RPC creates a span.
> 
> This is an issue. I'm glad Hadoop RPC is covered, but nobody but Hadoop
> uses it. Likewise, HBase RP These are not general purpose RPC stacks by
> any stretch. There are some of those around. Some have tracing built in.
> They take some of the oxygen out of the room. I think that is a fair
> point when thinking about the viability of a podling that sees little activity
> as it is.

Yeah-- maybe we should integrate HTrace into HBase RPC as well.

I don't think RPC-specific trace systems have been a strong competitors.
 Since the RPC landscape is so fragmented, those systems tend to not get
used by many people.  Our strongest open source competitors, OpenTracing
and OpenZipkin, support multiple RPC systems.  (Zipkin originally was
specific to Finagle, but that is no longer true.)

> I didn't come here to suggest HTrace go away, though. I came to raise a
> few points on why adoption and use of HTrace has very likely suffered from
> usability problems. These problems are still not completely resolved.
> Stack describes HTrace integration with HBase as broken. My experience has 
> been
> I have to patch POMs, and patch HDFS, HBase, and Phoenix code, to get
> anything that works at all. I also sought to tie some of those problems
> to ecosystem issues because I know it is hard. For what it's worth, thanks.

I think you make some very good points about the difficulty of doing
cross-project coordination.  One thing that really held back HTrace 4.0
was that it was originally scheduled to be part of Hadoop 2.8-- and the
Hadoop 2.8 release was delayed for a really, really long time, to the
point when it almost became a punchline.  So people had to use vendor
releases to get HTrace 4, because those were the only releases with new
Hadoop code.

Colin


> 
> 
> 
> On Thu, Aug 17, 2017 at 2:21 PM, Colin McCabe <cmcc...@apache.org> wrote:
> 
> > On Thu, Aug 17, 2017, at 12:25, Andrew Purtell wrote:
> > > What about OpenTracing (http://opentracing.io/)? Is this the successor
> > > project to ZipKin? In particular grpc-opentracing (
> > > https://github.com/grpc-ecosystem/grpc-opentracing) seems to finally
> > > fulfill in open source the tracing architecture described in the Dapper
> > > paper.
> >
> > OpenTracing is essentially an API which sits on top of another tracing
> > system.
> >
> > So you can instrument your code with the OpenTracing library, and then
> > have that send the trace spans to OpenZipkin.
> >
> > Here are some thoughts here about this topic from a Zipkin developer:
> > https://gist.github.com/wu-sheng/b8d51dda09d3ce6742630d1484fd55
> > c7#what-is-the-relationship-between-zipkin-and-opentracing
> > .  Probably Adrian Cole can chime in here as well.
> >
> > In general the OpenTracing folks have been friendly and respectful.  (If
> > any of them are reading this, I apologize for not following some of the
> > discussions on gitter more thoroughly-- my time is just split so many
> > ways right now!)
> >
> > >
> > > If one takes a step back and looks at all of the hand rolled RPC stacks
> > > in
> > > the Hadoop ecosystem it's a mess. It is a heavier lift but getting
> > > everyone
> > > migrated to a single RPC stack - gRPC - would provide the unified tracing
> > > layer envisioned by HTrace. The tracing integration is then done exactly
> > > in
> > > one place. In contrast HTrace requires all of the components to sprinkle
> > > spans throughout the application code.
> > >
> >
> > That's not the issue.  We already have HTrace integration with Hadoop
> > RPC, such that a Hadoop RPC creates a span.  Integration with any RPC
> > system is actually very straightforward-- you just add two fields to the
> > base RPC request definition, and patch the RPC system to use them.
> >
> > Just instrumenting RPC is not sufficient.  You need programmers to add
> > explicit span annotations to your code so that you can have useful
> > information beyond what a program like wireshark would find.  Things
> > like what disk is a request hitting, what HBase PUT is an HDFS write
> > associated with, and so forth.
> >
> > Also, this is getting off topic, but there is a new RPC system every
> > year or two.  Java-RMI, CORBA, Thrift, Akka, SOAP, KRPC, Finagle, GRPC,
> > REST/JSON, etc.  They all have advantages and disadvantages.  For
> > example, GRPC depends on protobuf-- and Hadoop has a lot of deployment
> > and performance prob

Re: [DISCUSS] Attic podling Apache HTrace?

[Colin McCabe]

On Thu, Aug 17, 2017, at 12:25, Andrew Purtell wrote:
> What about OpenTracing (http://opentracing.io/)? Is this the successor
> project to ZipKin? In particular grpc-opentracing (
> https://github.com/grpc-ecosystem/grpc-opentracing) seems to finally
> fulfill in open source the tracing architecture described in the Dapper
> paper.

OpenTracing is essentially an API which sits on top of another tracing
system.

So you can instrument your code with the OpenTracing library, and then
have that send the trace spans to OpenZipkin.

Here are some thoughts here about this topic from a Zipkin developer: 
https://gist.github.com/wu-sheng/b8d51dda09d3ce6742630d1484fd55c7#what-is-the-relationship-between-zipkin-and-opentracing
.  Probably Adrian Cole can chime in here as well.

In general the OpenTracing folks have been friendly and respectful.  (If
any of them are reading this, I apologize for not following some of the
discussions on gitter more thoroughly-- my time is just split so many
ways right now!)

> 
> If one takes a step back and looks at all of the hand rolled RPC stacks
> in
> the Hadoop ecosystem it's a mess. It is a heavier lift but getting
> everyone
> migrated to a single RPC stack - gRPC - would provide the unified tracing
> layer envisioned by HTrace. The tracing integration is then done exactly
> in
> one place. In contrast HTrace requires all of the components to sprinkle
> spans throughout the application code.
> 

That's not the issue.  We already have HTrace integration with Hadoop
RPC, such that a Hadoop RPC creates a span.  Integration with any RPC
system is actually very straightforward-- you just add two fields to the
base RPC request definition, and patch the RPC system to use them.

Just instrumenting RPC is not sufficient.  You need programmers to add
explicit span annotations to your code so that you can have useful
information beyond what a program like wireshark would find.  Things
like what disk is a request hitting, what HBase PUT is an HDFS write
associated with, and so forth.

Also, this is getting off topic, but there is a new RPC system every
year or two.  Java-RMI, CORBA, Thrift, Akka, SOAP, KRPC, Finagle, GRPC,
REST/JSON, etc.  They all have advantages and disadvantages.  For
example, GRPC depends on protobuf-- and Hadoop has a lot of deployment
and performance problems with the protobuf-java library.  I wish GPRC
luck, but I think it's good for people to experiment with different
libraries.  It doesn't make sense to try to force everyone to use one
thing, even if we could.

> The Hadoop ecosystem is always partially at odds with itself, if for no
> other reason than there is no shared vision among the projects. There are
> no coordinated releases. There isn't even agreement on which version of
> shared dependencies to use (hence the recurring pain in various places
> with
> downstream version changes of protobuf, guava, jackson, etc. etc).
> Therefore HTrace is severely constrained on what API changes can be made.
> Unfortunately the different major versions of HTrace do not interoperate
> at
> all. And are not even source compatible. While is not unreasonable at all
> for a project in incubation, when combined with the inability of the
> Hadoop
> ecosystem to coordinate releases as a cross-cutting dependency ships a
> new
> version, this has reduced the utility of HTrace to effectively nil for
> the
> average user. I am sorry to say that. Only a commercial Hadoop vendor or
> power user can be expected to patch and build a stack that actually
> works.

One correction: The different major versions of HTrace are indeed source
code compatible.  You can build an application that can use both HTrace
3 and HTrace 4.  This was absolutely essential for us because of the
version skew issues you mention.

> On Thu, Aug 17, 2017 at 11:04 AM, lewis john mcgibbney  
> wrote:
> 
> > Hi Mike,
> > I think this is a fair question. We've probably all been associated with
> > projects which just don't really make it. It would appear that HTrace is
> > one of them. This is not to say that there is nothing going on with the
> > tracing effort generally (as there is) but it looks like HTrace as a
> > project may be headed to the Attic.
> > I suppose the response to this thread will determine what happens...

Thanks, Lewis.

I think maybe we should try to identify the top tracing priorities for
HBase and HDFS and see how HTrace / OpenTracing / OpenZipkin could fit
into those.  Just start from a nice crisp set of requirements, like
Stack suggested, and think about how we could make those a reality.  If
we can advance the state of tracing in hadoop, that will be a good thing
for our users, even if htrace goes to the attic.  I've been mostly
working on Apache Kafka these days but I could drop by to brainstorm.

best,
Colin


> > Lewis
> > ​​
> >
> >
> > On Wed, Aug 16, 2017 at 10:01 AM, <
> > dev-digest-h...@htrace.incubator.apache.org> wrote:
> >
> > >
> > > From: Mike Drob 

Re: [DISCUSS] Attic podling Apache HTrace?

[Colin McCabe]

Thanks for bringing this up, Mike.

The original vision for HTrace was trying to unify a bunch of disparate
Hadoop components with a unified tracing layer.  This would allow us to
debug slowness or odd behavior in a much better way.  We started from
that vision and deduced the need to build a frontend API (htrace-core),
backend data store (htrace-hbase, htrace-htraced, etc.), and web UI
(htrace-web).

I still think that vision is valid, but achieving it was a lot harder
than we expected, for a couple of reasons.

First of all, I think building all those components needed someone (or
maybe several someones) to work on it full time.  We tried to do it part
time with a few HDFS and HBase committers.  Ultimately this didn't scale
as much as we needed it to.

Secondly, we were hoping for a lot of buy-in from Hadoop vendors and big
tech companies that used Hadoop.  Unfortunately, we didn't really get
that.  The Hadoop vendors were preoccupied with other things.  Big tech
companies seem to mostly developed their own internal systems using bits
and pieces of open source.  I think this is another area where we just
needed more budget.  In retrospect, having meetups and reaching out to
potential users is something we needed to do.  There are some other
projects that have been a lot better with this than we have.

I think we should try to refocus on some core use-cases.  Basically,
decide what we want to achieve and find the shortest path to that.  If
that involves using other projects, then that's fine-- as long as they
are open source projects compatible with the ideals of the ASF.

Off the top of my head, I can think of a few core use-cases:

* Why is my HDFS request slow?   Figure out if there are disk issues or
network issues.

* Why is my HBase request slow?  Follow HBase requests into the HDFS
layer.

* Who is making the most requests to HDFS?

* What average speed is Hadoop getting from its S3 requests?  How often
do we hit our local caches, versus going over the network?

best,
Colin


On Thu, Aug 17, 2017, at 10:04, Stack wrote:
> On Wed, Aug 16, 2017 at 10:00 AM, Mike Drob  wrote:
> 
> > Hi folks,
> >
> > Want to bring up a potentially uncofortable topic for some. Is it time to
> > retire/attic the project?
> >
> > We've seen a minimal amount of activity in the past year. The last release
> > had two bug fixes, and had been pending for several months before somebody
> > reminded me to push the artifacts to subversion from the staging directory.
> >
> > I'd love to see a renewed set of activity here, but I don't think there is
> > a ton of interest going on.
> >
> > HBase is still on version 3. So is Accumulo, I think. Hadoop is on 4.1,
> > which is a good sign, but I haven't heard much from them recently. I
> > definitely do no think we are at the point where a lack of releases and
> > activity is a sign of super advanced maturity and stability.
> >
> > Your thoughts?
> 
> 
> Thanks Mike for starting this thread.
> 
> Activity over the last year is here [1].
> 
> Is there any testimony other than evangelizing presentations on how
> htrace
> has provided a benefit?
> 
> HTrace needs a bit of work. In order of import:
> 
> 1. A complete viewer (punt and use zipkin instead?)
> 2. Hooked up systems that tell wholesome trace stories: hdfs is
> incomplete,
> hbase is broke, accumulo/unknown, phoenix/custom-htrace... who else?
> 3. Work needs to be done so an operator can easily enable/disable trace
> and
> easily obtain views without impinging upon general perf
> 
> It could do w/ an API cleanup (v5.0.0?) and study of the fact that it is
> painstaking manual work adding it into a system (and that it is
> subsequently easily damaged by code movement). It needs a particular type
> of barker to drive it cross-project since the cross-project realm is when
> it starts to come into its own (and each project in its turn will resist
> since the benefit not immediate), etc.
> 
> None of the above is under active dev.
> 
> St.Ack
> 
> 1. https://github.com/apache/incubator-htrace/graphs/commit-activity
> 
> 
> 
> > Mike
> >

Re: podling report draft

[Colin McCabe]

Thanks, Stack.  I have a login (as Colin McCabe), but it still showed up
as immutable for me... hmm.

cheers,
C.

On Thu, Jun 8, 2017, at 11:56, Stack wrote:
> Done.
> 
> (Did you login? Do you have a login?)
> 
> Go easy Colin,
> M
> 
> On Thu, Jun 8, 2017 at 10:58 AM, Colin McCabe <cmcc...@apache.org> wrote:
> 
> > Hmm, the wiki page shows up as immutable for me.  Does somehow have edit
> > permissions? https://wiki.apache.org/incubator/June2017
> >
> >
> >
> > On Wed, Jun 7, 2017, at 21:54, Stack wrote:
> > > LGTM
> > >
> > > On Wed, Jun 7, 2017 at 12:19 PM, Colin McCabe <cmcc...@apache.org>
> > wrote:
> > >
> > > > Here's a draft of the podling report.  Let me know if I left out
> > > > anything...
> > > >
> > > > -
> > > > HTrace is a tracing framework intended for use with distributed systems
> > > > written in java.
> > > >
> > > > HTrace has been incubating since 2014-11-11.
> > > >
> > > > Three most important issues to address in the move towards graduation:
> > > >
> > > >   1. Continue to grow the Apache HTrace community
> > > >   2. Continue to explore projects integrating Apache HTrace
> > > >   3. Continue to develop and release stable Apache HTrace incubating
> > > > artifacts
> > > >
> > > > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> > > > aware of?
> > > >
> > > >   There are no issues requiring board attention at this time.
> > > >
> > > > How has the community developed since the last report?
> > > >
> > > >   Mailing lists have been quiet.
> > > >
> > > > How has the project developed since the last report?
> > > >
> > > >   We are on track to make an HTrace 4.3 release.
> > > >
> > > > How would you assess the podling's maturity?
> > > > Please feel free to add your own commentary.
> > > >
> > > >   [ ] Initial setup
> > > >   [ ] Working towards first release
> > > >   [x] Community building
> > > >   [ ] Nearing graduation
> > > >   [ ] Other:
> > > >
> > > > Date of last release:
> > > >
> > > >   2016-10-06
> > > >
> > > > When were the last committers or PPMC members elected?
> > > >
> > > >   2016-10-03
> > > >
> >

Re: podling report draft

[Colin McCabe]

Hmm, the wiki page shows up as immutable for me.  Does somehow have edit
permissions? https://wiki.apache.org/incubator/June2017



On Wed, Jun 7, 2017, at 21:54, Stack wrote:
> LGTM
> 
> On Wed, Jun 7, 2017 at 12:19 PM, Colin McCabe <cmcc...@apache.org> wrote:
> 
> > Here's a draft of the podling report.  Let me know if I left out
> > anything...
> >
> > -
> > HTrace is a tracing framework intended for use with distributed systems
> > written in java.
> >
> > HTrace has been incubating since 2014-11-11.
> >
> > Three most important issues to address in the move towards graduation:
> >
> >   1. Continue to grow the Apache HTrace community
> >   2. Continue to explore projects integrating Apache HTrace
> >   3. Continue to develop and release stable Apache HTrace incubating
> > artifacts
> >
> > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> > aware of?
> >
> >   There are no issues requiring board attention at this time.
> >
> > How has the community developed since the last report?
> >
> >   Mailing lists have been quiet.
> >
> > How has the project developed since the last report?
> >
> >   We are on track to make an HTrace 4.3 release.
> >
> > How would you assess the podling's maturity?
> > Please feel free to add your own commentary.
> >
> >   [ ] Initial setup
> >   [ ] Working towards first release
> >   [x] Community building
> >   [ ] Nearing graduation
> >   [ ] Other:
> >
> > Date of last release:
> >
> >   2016-10-06
> >
> > When were the last committers or PPMC members elected?
> >
> >   2016-10-03
> >

podling report draft

[Colin McCabe]

Here's a draft of the podling report.  Let me know if I left out
anything...

-
HTrace is a tracing framework intended for use with distributed systems
written in java.

HTrace has been incubating since 2014-11-11.

Three most important issues to address in the move towards graduation:

  1. Continue to grow the Apache HTrace community
  2. Continue to explore projects integrating Apache HTrace
  3. Continue to develop and release stable Apache HTrace incubating
artifacts

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

  There are no issues requiring board attention at this time.

How has the community developed since the last report?

  Mailing lists have been quiet.

How has the project developed since the last report?

  We are on track to make an HTrace 4.3 release.

How would you assess the podling's maturity?
Please feel free to add your own commentary.

  [ ] Initial setup
  [ ] Working towards first release
  [x] Community building
  [ ] Nearing graduation
  [ ] Other:

Date of last release:

  2016-10-06

When were the last committers or PPMC members elected?

  2016-10-03

Re: [RESULT][VOTE] HTrace-4.3.0-incubating RC3

[Colin McCabe]

No worries, Mike.  Thanks for driving this!

cheers,
Colin


On Fri, Jun 2, 2017, at 12:59, Mike Drob wrote:
> Vote passes with +3
> * Masatake Iwasaki
> * Elliott Clark
> * Colin McCabe
> 
> Will submit to IPMC shortly.
> 
> Apologies for letting this slip for so long; other schedules got in the
> way
> of being able to work on this.
> 
> On Wed, May 24, 2017 at 12:29 PM, Stack <st...@duboce.net> wrote:
> 
> > I wouldn't worry about it. Ship the thing.
> >
> > I had trouble making it pass all tests on mac. Will did in in time for next
> > release.
> >
> > St.Ack
> >
> > On Wed, May 24, 2017 at 10:23 AM, Colin McCabe <cmcc...@apache.org> wrote:
> >
> > > I always thought that the deadline represented a minimum period, rather
> > > than a maximum.  In other words, the deadline was around to prevent
> > > people from putting out a release in 6 hours or whatever that had some
> > > undiscovered flaws because people didn't have a chance to examine it.
> > >
> > > I don't remember us ever re-doing a vote because the period had been
> > > exceeded without enough votes coming in.  But looking at
> > > https://www.apache.org/foundation/voting.html , it doesn't seem to
> > > settle the issue one way or another.  Does anyone know if there is a
> > > formal rule for this?
> > >
> > > I guess to be on the safe side, you could start a new vote and have
> > > people re-cast.
> > >
> > > cheers,
> > > Colin
> > >
> > >
> > > On Tue, May 23, 2017, at 18:22, Mike Drob wrote:
> > > > A question about protocol here... Two of the votes have come in past
> > the
> > > > deadline, but without them we wouldn't have had enough votes to get
> > > > quorum
> > > > on the release. Can we still accept them and move forward, or do we
> > need
> > > > to
> > > > start over with a new vote? I'd really like to avoid the second option
> > > > since getting enough votes here has been tenuous.
> > > >
> > > > I'm also making a note for myself for future votes that seven days is
> > > > probably a better window than five.
> > > >
> > > > Mike
> > > >
> > > > On Tue, May 23, 2017 at 7:36 PM, Colin McCabe <cmcc...@apache.org>
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Verified checksums
> > > > > Build from source using Maven 3.0.4 (with -Pnative)
> > > > > Successfully ran unit tests
> > > > >
> > > > >
> > > > >
> > > > > On Mon, May 22, 2017, at 01:18, Elliott Clark wrote:
> > > > > > +1
> > > > > >
> > > > > > verified signatures.
> > > > > > build from source.
> > > > > > Checked that everything looked to be in the correct locations.
> > > > > >
> > > > > > On Sun, May 21, 2017 at 3:49 PM, Masatake Iwasaki <
> > > > > > iwasak...@oss.nttdata.co.jp> wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > * verified signature and mds.
> > > > > > > * built and installed from source tarball by maven-3.0.4.
> > > > > > > * built hadoop trunk against htrace-4.3.0-incubating.
> > > > > > > * verified that spans in HDFS works by HTracedSpanReceiver and
> > > htraced.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Masatake Iwasaki
> > > > > > >
> > > > > > >
> > > > > > > On 5/17/17 05:40, Mike Drob wrote:
> > > > > > >
> > > > > > >> Hi Devs,
> > > > > > >>
> > > > > > >> Please consider the following release candidate for Apache
> > HTrace
> > > > > > >> 4.3.0-incubating!
> > > > > > >>
> > > > > > >> http://people.apache.org/~mdrob/htrace-4.3.0-incubating-rc3/
> > > > > > >>
> > > > > > >> The jars have been staged here:
> > > > > > >>
> > > > > > >> https://repository.apache.org/content/repositories/
> > > > > orgapachehtrace-1029
> > > > > > >>
> > > > > > >> There's a few minor fixes in this release:
> > > > > > >>
> > > > > > >> ** Bug
> > > > > > >>  * [HTRACE-387] - htrace-c should have a test dependency on
> > > htrace
> > > > > > >> -htraced
> > > > > > >>  * [HTRACE-393] - Binary annotations are being shown as
> > > base64 in
> > > > > the
> > > > > > >> zipkin frontend
> > > > > > >>  * [HTRACE-398] - Enforce requirement to build with Maven
> > > 3.0.4
> > > > > from
> > > > > > >> HTRACE-236
> > > > > > >>
> > > > > > >> ** Improvement
> > > > > > >>  * [HTRACE-386] - htrace-c: add functions for retrieving the
> > > span
> > > > > ID
> > > > > > >> to
> > > > > > >> the public API
> > > > > > >>
> > > > > > >> RC History:
> > > > > > >> RC0 - Discarded due to user error.
> > > > > > >> RC1 - Missing some version string updates, vote failed.
> > > > > > >> RC2 - Discarded due to missing HTRACE-398
> > > > > > >> RC3 - Current
> > > > > > >>
> > > > > > >> This vote will run for 5 days ( Sun May 21 21:30:00 UTC 2017 )
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Mike
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > >
> > >
> >

Re: [VOTE] HTrace-4.3.0-incubating RC3

[Colin McCabe]

I always thought that the deadline represented a minimum period, rather
than a maximum.  In other words, the deadline was around to prevent
people from putting out a release in 6 hours or whatever that had some
undiscovered flaws because people didn't have a chance to examine it.

I don't remember us ever re-doing a vote because the period had been
exceeded without enough votes coming in.  But looking at
https://www.apache.org/foundation/voting.html , it doesn't seem to
settle the issue one way or another.  Does anyone know if there is a
formal rule for this?

I guess to be on the safe side, you could start a new vote and have
people re-cast.

cheers,
Colin


On Tue, May 23, 2017, at 18:22, Mike Drob wrote:
> A question about protocol here... Two of the votes have come in past the
> deadline, but without them we wouldn't have had enough votes to get
> quorum
> on the release. Can we still accept them and move forward, or do we need
> to
> start over with a new vote? I'd really like to avoid the second option
> since getting enough votes here has been tenuous.
> 
> I'm also making a note for myself for future votes that seven days is
> probably a better window than five.
> 
> Mike
> 
> On Tue, May 23, 2017 at 7:36 PM, Colin McCabe <cmcc...@apache.org> wrote:
> 
> > +1
> >
> > Verified checksums
> > Build from source using Maven 3.0.4 (with -Pnative)
> > Successfully ran unit tests
> >
> >
> >
> > On Mon, May 22, 2017, at 01:18, Elliott Clark wrote:
> > > +1
> > >
> > > verified signatures.
> > > build from source.
> > > Checked that everything looked to be in the correct locations.
> > >
> > > On Sun, May 21, 2017 at 3:49 PM, Masatake Iwasaki <
> > > iwasak...@oss.nttdata.co.jp> wrote:
> > >
> > > > +1
> > > >
> > > > * verified signature and mds.
> > > > * built and installed from source tarball by maven-3.0.4.
> > > > * built hadoop trunk against htrace-4.3.0-incubating.
> > > > * verified that spans in HDFS works by HTracedSpanReceiver and htraced.
> > > >
> > > > Thanks,
> > > > Masatake Iwasaki
> > > >
> > > >
> > > > On 5/17/17 05:40, Mike Drob wrote:
> > > >
> > > >> Hi Devs,
> > > >>
> > > >> Please consider the following release candidate for Apache HTrace
> > > >> 4.3.0-incubating!
> > > >>
> > > >> http://people.apache.org/~mdrob/htrace-4.3.0-incubating-rc3/
> > > >>
> > > >> The jars have been staged here:
> > > >>
> > > >> https://repository.apache.org/content/repositories/
> > orgapachehtrace-1029
> > > >>
> > > >> There's a few minor fixes in this release:
> > > >>
> > > >> ** Bug
> > > >>  * [HTRACE-387] - htrace-c should have a test dependency on htrace
> > > >> -htraced
> > > >>  * [HTRACE-393] - Binary annotations are being shown as base64 in
> > the
> > > >> zipkin frontend
> > > >>  * [HTRACE-398] - Enforce requirement to build with Maven 3.0.4
> > from
> > > >> HTRACE-236
> > > >>
> > > >> ** Improvement
> > > >>  * [HTRACE-386] - htrace-c: add functions for retrieving the span
> > ID
> > > >> to
> > > >> the public API
> > > >>
> > > >> RC History:
> > > >> RC0 - Discarded due to user error.
> > > >> RC1 - Missing some version string updates, vote failed.
> > > >> RC2 - Discarded due to missing HTRACE-398
> > > >> RC3 - Current
> > > >>
> > > >> This vote will run for 5 days ( Sun May 21 21:30:00 UTC 2017 )
> > > >>
> > > >> Thanks,
> > > >> Mike
> > > >>
> > > >>
> > > >
> >

Re: [VOTE] HTrace-4.3.0-incubating RC3

[Colin McCabe]

+1

Verified checksums
Build from source using Maven 3.0.4 (with -Pnative)
Successfully ran unit tests



On Mon, May 22, 2017, at 01:18, Elliott Clark wrote:
> +1
> 
> verified signatures.
> build from source.
> Checked that everything looked to be in the correct locations.
> 
> On Sun, May 21, 2017 at 3:49 PM, Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp> wrote:
> 
> > +1
> >
> > * verified signature and mds.
> > * built and installed from source tarball by maven-3.0.4.
> > * built hadoop trunk against htrace-4.3.0-incubating.
> > * verified that spans in HDFS works by HTracedSpanReceiver and htraced.
> >
> > Thanks,
> > Masatake Iwasaki
> >
> >
> > On 5/17/17 05:40, Mike Drob wrote:
> >
> >> Hi Devs,
> >>
> >> Please consider the following release candidate for Apache HTrace
> >> 4.3.0-incubating!
> >>
> >> http://people.apache.org/~mdrob/htrace-4.3.0-incubating-rc3/
> >>
> >> The jars have been staged here:
> >>
> >> https://repository.apache.org/content/repositories/orgapachehtrace-1029
> >>
> >> There's a few minor fixes in this release:
> >>
> >> ** Bug
> >>  * [HTRACE-387] - htrace-c should have a test dependency on htrace
> >> -htraced
> >>  * [HTRACE-393] - Binary annotations are being shown as base64 in the
> >> zipkin frontend
> >>  * [HTRACE-398] - Enforce requirement to build with Maven 3.0.4 from
> >> HTRACE-236
> >>
> >> ** Improvement
> >>  * [HTRACE-386] - htrace-c: add functions for retrieving the span ID
> >> to
> >> the public API
> >>
> >> RC History:
> >> RC0 - Discarded due to user error.
> >> RC1 - Missing some version string updates, vote failed.
> >> RC2 - Discarded due to missing HTRACE-398
> >> RC3 - Current
> >>
> >> This vote will run for 5 days ( Sun May 21 21:30:00 UTC 2017 )
> >>
> >> Thanks,
> >> Mike
> >>
> >>
> >

Re: HTrace 4.3.0 RC1

[Colin McCabe]

P.S. it's probably worth doing a new RC with the version changes.  It
looks pretty good aside from that... I'll run a few more tests with
htraced and the web UI when the new RC is out.

Colin


On Mon, May 15, 2017, at 11:33, Colin McCabe wrote:
> Thanks for driving this, Mike!
> 
> Like Masatake noted, there are a few places still referring to 4.2:
> * htrace-htraced/pom.xml
> * htrace-c/src/CMakeLists.txt
> 
> cheers,
> Colin
> 
> On Mon, May 8, 2017, at 13:46, Mike Drob wrote:
> > Hi Devs,
> > 
> > Based on recent mailing list discussions, I've posted the first release
> > candidate here: (rc0 had to be discarded due to an error on my part)
> > 
> > http://people.apache.org/~mdrob/htrace-4.3.0-incubating-rc1/
> > 
> > The jars have been staged here:
> > 
> > https://repository.apache.org/content/repositories/orgapachehtrace-1026
> > 
> > There's a few minor fixes in this release:
> > 
> > ** Bug
> > * [HTRACE-387] - htrace-c should have a test dependency on
> > htrace-htraced
> > * [HTRACE-393] - Binary annotations are being shown as base64 in the
> > zipkin frontend
> > 
> > ** Improvement
> > * [HTRACE-386] - htrace-c: add functions for retrieving the span ID
> > to
> > the public API
> > 
> > 
> > This vote will run for 5 days ( Sat May 13 21:00:00 UTC 2017 )
> > 
> > Thanks,
> > Mike

Re: 4.3.0 to maven

[Colin McCabe]

I think the C changes were just exposing additional APIs, so there
should be no backwards compatibility implications.

best,


On Mon, May 8, 2017, at 12:48, Mike Drob wrote:
> Had a chance to look at this in more detail. From the Java side we should
> be safe to do 4.2.1. From the C side... I don't know what kind of
> guarantees we offer there. Looks like addition of APIs, but I don't know
> what C communities and users expect.
> 
> Colin and Masatake - since you were involved in the change, can you
> comment?
> 
> Mike
> 
> On Mon, May 8, 2017 at 9:14 AM, Raam Rosh-Hai  wrote:
> 
> > Doesn't seem like any api changes were made so I am for the sneaking in a
> > minor version
> >

Re: 4.3.0 to maven

[Colin McCabe]

On Thu, May 4, 2017, at 18:07, Mike Drob wrote:
> I can make time for running through the release next week. Any unresolved
> JIRA issues will be bulk moved to 4.4 on Monday with this plan.

+1

> 
> I'm assuming somebody had already done the check on whether this needs to
> be a full 4.3 or can sneak in as a 4.2.1? Seem like we have a chance to
> not
> increment version number if change set is small and bug fix only.

I don't have a strong preference between 4.2.1 and 4.3.  I wonder what
you guys think?

best,
Colin

> 
> Mike
> 
> On Thu, May 4, 2017, 2:54 PM Colin McCabe <cmcc...@apache.org> wrote:
> 
> > Hi Raam,
> >
> > We haven't made an official 4.3 release yet.  I think we should,
> > although a lot of us are busy on other Apache projects at the moment.
> > Any interest in picking this up, Stack, Mike, et al?
> >
> > It looks like the main fixes were HTRACE-393 to the zipkin connector,
> > and some stuff with htrace-c.  I also remember having a discussion about
> > including the sub-millisecond granularity timing patch in the new
> > release, although maybe we should push that off.
> >
> > best,
> > Colin
> >
> > On Tue, May 2, 2017, at 03:00, Raam Rosh-Hai wrote:
> > > sorry my mistake, didn't notice the direct link to apache, but still,
> > > what
> > > about maven central :D?
> > >
> > > On 2 May 2017 at 11:56, Raam Rosh-Hai <r...@findhotel.net> wrote:
> > >
> > > > Hi Guys,
> > > > I was wondering when do you plan to release 4.3.0 to maven?
> > > > my akka tracing project relies on the fixes for 4.3.0 and now people
> > have
> > > > to build that version themselves.
> > > > for reference this the PR: https://github.com/FindHotel/
> > > > akka-stream-trace/pull/2
> > > > (doesn't work since it's not released yet)
> > > >
> > > > is there anything I can do?
> > > >
> > > > Cheers,
> > > > Raam
> > > >
> >

Re: incubator report / podling status

[Colin McCabe]

On Mon, Mar 6, 2017, at 10:56, Billie Rinaldi wrote:
> On Fri, Mar 3, 2017 at 8:21 AM, Stack <st...@duboce.net> wrote:
> 
> > On Fri, Mar 3, 2017 at 4:13 PM, Billie Rinaldi <bil...@apache.org> wrote:
> >
> > > I posted the report to https://wiki.apache.org/incubator/March2017 and
> > > signed off. Other mentors, please sign off as well.
> > >
> > >
> > Done
> >
> >
> >
> > > There has been some discussion in the Accumulo community about
> > contributing
> > > its span receiver back to HTrace. We know it works well, and that would
> > > make configuring and versioning it easier. I haven't had time to work on
> > > it, though, which is why I haven't brought it up. Would HTrace consider
> > > accepting an Accumulo span receiver, if we could find someone to work on
> > > it?
> > >
> > >
> > Yes. Makes sense. What is special about this receiver?
> >
> 
> Nearly all Apache Accumulo instances are configured to send traces to
> themselves using this mechanism. There's a tracer process that spans are
> sent to via Thrift, and the tracer inserts the spans into Accumulo. The
> tracer location(s) are discovered via ZooKeeper. So, pretty normal?
> https://github.com/apache/accumulo/tree/master/server/tracer/src/main/java/org/apache/accumulo/tracer

Yeah, we should add that SpanReceiver to HTrace.

> 
> When testing out HDFS and Accumulo tracing together, I found I had to add
> a
> configuration property that would optionally allow the receiver to drop
> spans of length 0ms, as HDFS was generating a huge number of these. (I
> realize dropping spans is not recommended, but I was still able to do
> some
> interpretation of parentless spans when the 0ms spans were missing.)

This should be fixed in the latest HDFS.  Unfortunately, the Hadoop 2.8
release got eaten by a grue, but if you try Hadoop 3.0 alpha, it should
be there.  Incidentally, updating Accumulo to HTrace 4.x would be one of
the best things we could do to help HTrace...

cheers,
Colin

> 
> Billie
> 
> 
> >
> > St.Ack
> >
> >
> > > On Thu, Mar 2, 2017 at 4:54 PM, Colin McCabe <cmcc...@apache.org> wrote:
> > >
> > > > On Thu, Mar 2, 2017, at 13:01, Stack wrote:
> > > > > On Thu, Mar 2, 2017 at 8:19 PM, Billie Rinaldi <bil...@apache.org>
> > > > wrote:
> > > > >
> > > > > > HTrace's quarterly report is slightly overdue (due yesterday). I've
> > > > pasted
> > > > > > a bare bones draft below based on the last report. Does anyone have
> > > > > > something to add?
> > > > > >
> > > > > > Thank you Billie. +1 on the report. It conveys that little has
> > > happened
> > > > > since our last report.
> > > > >
> > > > >
> > > > > > We should have a discussion about HTrace's status. We were talking
> > > > about
> > > > > > the possibility of graduation over a year ago. Do you think we need
> > > to
> > > > grow
> > > > > > the community further before proposing graduation?
> > > > > >
> > > > > >
> > > > > We should. There has been little progress of late. Lets talk about
> > > this.
> > > > > Most of us seem to be off occupied on other stuff with little time
> > over
> > > > > for
> > > > > moving HTrace forward.
> > > > >
> > > > > St.Ack
> > > >
> > > > Thanks for speaking up, Billie and Stack.  Agree it's been a little
> > > > quiet lately.  I've been busy on Apache Kafka recently.  There was some
> > > > stuff I wanted to push on the HTrace side but I didn't get a chance to
> > > > post it yet.
> > > >
> > > > Colin
> > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > -
> > > > > > HTrace is a tracing framework intended for use with distributed
> > > systems
> > > > > > written in java.
> > > > > >
> > > > > > HTrace has been incubating since 2014-11-11.
> > > > > >
> > > > > > Three most important issues to address in the move towards
> > > graduation:
> > > > > >
> > > > > >   1. Continue to grow the Apache HTrace community
> > > > > >   2. Continue to explore projects integr

Re: [DISCUSS] OpenTracing API implementation

[Colin McCabe]

Hi Konstantin,

Just to be clear, you're proposing an adapter that allows people to use
OpenTracing as the span creation API, and HTrace SpanReceivers to store
spans?  That seems like an interesting idea and I don't think anyone
would object to adding it.  It's just a matter of someone committing to
write and maintain it.

best,
Colin


On Mon, Mar 6, 2017, at 04:26, Konstantin Gribov wrote:
> Hi, folks.
> 
> Are you interested in OpenTracing API (OT) implementation with HTrace as
> tracer target?
> 
> As I saw in [1] there was a bit of interest for OT but from different
> angle
> (community-wise).
> 
> Zipkin has both it's own Dapper-inpired API and OT API impl which gives
> good way for API to be vendor neutral. Also Uber is opensourcing its
> Jaeger
> (which is native OT impl).
> 
> I think, implementing OT API as separate module could bring more users to
> HTrace and improve its adaption.
> 
> If PMC is interested in it, I could give it a try.
> 
> [1]:
> https://lists.apache.org/thread.html/373c91d58dcb9c87e29ff4cb2ad4d339ec228fdf6155b2e6a09c23f6@1463677907@%3Cdev.htrace.apache.org%3E
> 
> -- 
> 
> Best regards,
> Konstantin Gribov

Re: incubator report / podling status

[Colin McCabe]

On Thu, Mar 2, 2017, at 13:01, Stack wrote:
> On Thu, Mar 2, 2017 at 8:19 PM, Billie Rinaldi  wrote:
> 
> > HTrace's quarterly report is slightly overdue (due yesterday). I've pasted
> > a bare bones draft below based on the last report. Does anyone have
> > something to add?
> >
> > Thank you Billie. +1 on the report. It conveys that little has happened
> since our last report.
> 
> 
> > We should have a discussion about HTrace's status. We were talking about
> > the possibility of graduation over a year ago. Do you think we need to grow
> > the community further before proposing graduation?
> >
> >
> We should. There has been little progress of late. Lets talk about this.
> Most of us seem to be off occupied on other stuff with little time over
> for
> moving HTrace forward.
> 
> St.Ack

Thanks for speaking up, Billie and Stack.  Agree it's been a little
quiet lately.  I've been busy on Apache Kafka recently.  There was some
stuff I wanted to push on the HTrace side but I didn't get a chance to
post it yet.

Colin

> 
> 
> 
> 
> -
> > HTrace is a tracing framework intended for use with distributed systems
> > written in java.
> >
> > HTrace has been incubating since 2014-11-11.
> >
> > Three most important issues to address in the move towards graduation:
> >
> >   1. Continue to grow the Apache HTrace community
> >   2. Continue to explore projects integrating Apache HTrace
> >   3. Continue to develop and release stable Apache HTrace incubating
> > artifacts
> >
> > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> > aware of?
> >
> >   There are no issues requiring board attention at this time.
> >
> > How has the community developed since the last report?
> >
> >   Mailing lists have been quiet.
> >
> > How has the project developed since the last report?
> >
> >   Development has been quiet.
> >
> > How would you assess the podling's maturity?
> > Please feel free to add your own commentary.
> >
> >   [ ] Initial setup
> >   [ ] Working towards first release
> >   [x] Community building
> >   [ ] Nearing graduation
> >   [ ] Other:
> >
> > Date of last release:
> >
> >   2016-10-06
> >
> > When were the last committers or PPMC members elected?
> >
> >   2016-10-03
> >

Re: Not able to use HBaseSpanReceiver

[Colin McCabe]

On Tue, Dec 13, 2016, at 03:15, Mihir Monani wrote:
> Hi,
> 
> I am using below version of HBase and HTrace :-
> 
> HBase 0.98.x
> 
> HTrace 3.1.0

Hi Mihir,

That's a very old version of HTrace, from two years ago.  I really
recommend using modern versions, with a setup like this: 
http://blog.cloudera.com/blog/2015/12/new-in-cloudera-labs-apache-htrace-incubating/

> 
> I am trying to store spans in HBase itself but its not storing any data
> in
> HBase  There are no errors or warnings. Also when I start HBase , it will
> also print log line , something like this  " successfully loaded
> SpanReceiver."

What's the goal?  To store spans in HBase or to view tracing of HBase
itself?  If you are tracing HBase itself, you should expect to see that
log message.  It is not a good idea to trace hbase if you are also using
HBase to store the traces themselves (there is an infinite recursion
issue here).

best,
Colin


> 
> I am following this documentation for configuration :-
> 
> https://github.com/apache/incubator-htrace/tree/3.1/htrace-hbase
> 
> Can any one help me out?
> 
> Thanks,
> Mihir Monani

Re: report due today

[Colin McCabe]

Thanks, Stack!  Agree that it has been a little quiet in the last month.

I'm working on a patch to port the native stuff to Mac OS X, but there
are a few annoying details that prevent me from posting just yet.

I like the writeup below.

best,
Colin

On Wed, Dec 7, 2016, at 13:47, Stack wrote:
> Its been quiet of late. How about this:
> 
> HTraceHTrace is a tracing framework intended for use with distributed
> systemswritten in java.HTrace has been incubating since
> 2014-11-11.Three most important issues to address in the move towards
> graduation:
> 
>   1. Continue to grow the Apache HTrace community  2. Continue to
> explore projects integrating Apache HTrace  3. Continue to develop and
> release stable Apache HTrace incubating artifacts
> 
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> aware
> of?
> 
>   There are no issues requiring board attention at this time.
> 
> How has the community developed since the last report? Mailing lists have
> been quiet. How has the project developed since the last report?
> 
>   Release htrace-4.2.0
> 
> 
> Date of last release:  htrace-4.2.0 October 6th, 2016When were the
> last committers or PMC members elected?
> 
>  +Mike Drob (mdrob) October 3rd, 2016Signed-off-by:  [ ](htrace) Jake
> Farrell  [ ](htrace) Todd Lipcon  [ ](htrace) Lewis John Mcgibbney  [
> ](htrace) Andrew Purtell  [ ](htrace) Billie Rinaldi  [ ](htrace)
> Michael StackShepherd/Mentor notes:
> 
> 
> 
> On Wed, Dec 7, 2016 at 1:22 PM, Billie Rinaldi  wrote:
> 
> > Anyone want to put together a podling report for HTrace at
> > https://wiki.apache.org/incubator/December2016?
> >
> > Billie
> >

Re: Trace through MapReduce?

[Colin McCabe]

P.S. If you're interested in contributing, adding this MapReduce ID to
spans would be a cool project.  Also, converting accumulo to the latest
version of HTrace (4.0) would be great.

Colin


On Sat, Nov 26, 2016, at 11:20, Colin McCabe wrote:
> Hi Dylan,
> 
> Thanks for trying out HTrace!  We haven't added HTrace support to
> MapReduce yet.  Since MapReduce involves very long-running jobs, there
> is some discussion about the best way to add HTrace support to it. It
> doesn't really fit into the "one trace per request" model that HDFS
> uses.  One promising proposal is to add a tag to all spans that are
> created during a given mapreduce job, that contains an ID which can be
> traced back to the MR job.
> 
> best,
> Colin
> 
> 
> On Sat, Nov 26, 2016, at 05:17, Dylan Hutchison wrote:
> > Hi folks,
> > 
> > I am using HTrace 3 with Accumulo.  I would like to trace through a
> > MapReduce program that uses Accumulo Input/Output formats.  Has anyone
> > done
> > this?  I am using Hadoop 2.7.2, HTrace 3.1.0, Accumulo 1.8.0.
> > 
> > I confirm HTrace 3 is working with client java programs that scan
> > Accumulo.
> > 
> > 
> > I am not sure if Hadoop tracing is working. I added the ZooTraceClient
> > configuration to Hadoop and added the relevant Accumulo jars to Hadoop's
> > classpath, but I don't know if it worked.  (I see a new trace entry
> > called
> > ClientNamenodeProtocol that I never saw before, but it's not proof that
> > Hadoop tracing is working.)
> > 
> > I don't think the trace is being wrapped around the MapReduce mechanisms
> > that exec Mappers and Reducers over Yarn.
> > 
> > Maybe I can make it work by detaching the trace?  Would HTrace work if I
> > detach a trace from one process, record the trace ID, send the trace ID
> > to
> > the mappers and reducers, and then re-attach at the mapper and reducer
> > processes?
> > 
> > Cheers, Dylan

Re: Trace through MapReduce?

[Colin McCabe]

Hi Dylan,

Thanks for trying out HTrace!  We haven't added HTrace support to
MapReduce yet.  Since MapReduce involves very long-running jobs, there
is some discussion about the best way to add HTrace support to it. It
doesn't really fit into the "one trace per request" model that HDFS
uses.  One promising proposal is to add a tag to all spans that are
created during a given mapreduce job, that contains an ID which can be
traced back to the MR job.

best,
Colin


On Sat, Nov 26, 2016, at 05:17, Dylan Hutchison wrote:
> Hi folks,
> 
> I am using HTrace 3 with Accumulo.  I would like to trace through a
> MapReduce program that uses Accumulo Input/Output formats.  Has anyone
> done
> this?  I am using Hadoop 2.7.2, HTrace 3.1.0, Accumulo 1.8.0.
> 
> I confirm HTrace 3 is working with client java programs that scan
> Accumulo.
> 
> 
> I am not sure if Hadoop tracing is working. I added the ZooTraceClient
> configuration to Hadoop and added the relevant Accumulo jars to Hadoop's
> classpath, but I don't know if it worked.  (I see a new trace entry
> called
> ClientNamenodeProtocol that I never saw before, but it's not proof that
> Hadoop tracing is working.)
> 
> I don't think the trace is being wrapped around the MapReduce mechanisms
> that exec Mappers and Reducers over Yarn.
> 
> Maybe I can make it work by detaching the trace?  Would HTrace work if I
> detach a trace from one process, record the trace ID, send the trace ID
> to
> the mappers and reducers, and then re-attach at the mapper and reducer
> processes?
> 
> Cheers, Dylan

Re: [VOTE] HTrace 4.2.0-Incubating RC0

[Colin McCabe]

Great work, Mike!  And thanks for helping with the permissions issues,
Stack.

I would also add that this release also fixes some build frustrations
for users of jdk8 (the javadoc linting warnings were breaking the build
for them).

* Checked md5sum
* Did a clean build with jdk8 and ran all unit tests
* Ran htraced locally and looked at the web UI a bit.

I did encounter https://issues.apache.org/jira/browse/HTRACE-387 when
doing a quick test on this release.  It's an existing bug, so I don't
think it merits a new R  But if we do do a new RC, it would be nice.

+1

best,
Colin


On Thu, Sep 15, 2016, at 20:58, Mike Drob wrote:
> HTrace devs,
> 
> I've posted the first release candidate here:
> http://people.apache.org/~mdrob/htrace-4.2.0-incubating-rc0/
> 
> Signed by my key, available here:
> http://pgp.mit.edu/pks/lookup?op=vindex=0x3E48C0C6EF362B9E
> 
> The jars have been staged
> here:https://repository.apache.org/content/repositories/orgapachehtrace-1017
> 
> Signed by Stack's key, available here:
> https://www.apache.org/dist/incubator/htrace/KEYS
> 
> These artifacts are based on commit 7c0ae16 available in the git
> repository.
> 
> There's a lot of great stuff in this release, including:
>  * Improved developer documentation and javadocs
>  * Bug fixes to correct TraceRunnable/Callable construction pulling
> incorrect values.
>  * A tracing wrapper around SchedulerExecutorService
> 
> Full release notes available at:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315924=12334920
> 
> 
> Please vote one of:
> [ ] +1 - I have verified and accept...
> [ ] +0 - I have reservations, but not strong enough to vote against...
> [ ] -1 - Because..., I do not accept...
> ... these artifacts as the 4.2.0-incubating release of Apache HTrace.
> 
> 
> The vote will run for 5 days, ending Tuesday, September 20 23:59:59 US
> Eastern Time.
> 
> Cheers,
> Mike

Re: c++ support

[Colin McCabe]

I filed https://issues.apache.org/jira/browse/HTRACE-386 to take a look
at this.

best,


On Wed, Sep 14, 2016, at 12:43, Colin McCabe wrote:
> Oh yeah... I guess I should have included that :P
> 
> Thanks, Mike.
> 
> C.
> 
> 
> On Wed, Sep 14, 2016, at 12:42, Mike Drob wrote:
> > And in case you are unsure of where to find our JIRA instance - here is a
> > link!
> > 
> > https://issues.apache.org/jira/browse/HTRACE
> > 
> > On Wed, Sep 14, 2016 at 2:39 PM, Colin McCabe <cmcc...@apache.org> wrote:
> > 
> > > Hi Byung-Gon,
> > >
> > > Hmm.  The lack of an API for getting the current trace ID does look like
> > > an omission!  I think we should add this.  It shouldn't be too
> > > difficult.
> > >
> > > I would say, open a JIRA so we can take a look.
> > >
> > > best,
> > > Colin
> > >
> > > On Tue, Sep 13, 2016, at 14:57, Byung-Gon Chun wrote:
> > > > Hi,
> > > >
> > > > We've been using HTrace for Java. This really worked well. Thanks!
> > > >
> > > > We hope to trace C++ code with HTrace as well. But it looks like C++ 
> > > > does
> > > > not expose all the features we need (e.g., getting a span id). What's 
> > > > the
> > > > best way to approach this?
> > > >
> > > > Thanks.
> > > > Byung-Gon Chun
> > >

Re: HTrace API comments

[Colin McCabe]

On Mon, Sep 12, 2016, at 08:32, Roberto Attias wrote:
> Hi Colin,
> see inline
>
>
> **From:** Colin McCabe <cmcc...@apache.org> **To:**
> dev@htrace.incubator.apache.org; Roberto Attias
> <roberto_att...@yahoo.com> **Cc:** John D. Ament
> <johndam...@apache.org>; Jake Farrell <jfarr...@apache.org>; Ted
> Dunning <ted.dunn...@gmail.com> **Sent:** Sunday, September 11, 2016
> 10:03 PM **Subject:** Re: HTrace API comments
>
> On Sat, Sep 10, 2016, at 20:04, Roberto Attias wrote:
> > Hello,I have some comment/concerns regarding the HTrace API, and was
> > wondering whether extensions/changes would be considered. I'm
> > listing the
> > most important here, if there is interest we can discuss more in
> > detail.
>
> Welcome, Roberto!

Sorry for the delay in responses.  You have certainly given us a lot to
think about... thanks for your thoughtful comments.

> >
> > 1) From the HTrace Developer Guide:
> >
> >
> >
> > TraceScope objects manage the lifespan of Span objects. When a
> > TraceScope
> > is created, it often comes with an associated Span object. When this
> > scope is closed, the Span will be closed as well. “Closing”
> > the scope
> > means that the span is sent to a SpanReceiver for processing.
> >
> >
> > One of the implications of this model is the fact that nested
> > spans (for
> > example instrumenting nested function calls) will be delivered
> > to the
> > receiver in reverse order (as the innermost function completes
> > before the
> > outermost. This may introduce more complexity on the logic in
> > the span
> > receiver.
>
> Hmm.  While I would never say never, in the existing span
> receivers, we
> haven't found that delivering the spans in this order results in any
> extra complexity.  What you want is a span sink that aggregates all
> the
> spans together, and supports querying spans by various things like ID,
> time, etc.  This is typically a distributed database like HBase, Kudu,
> etc.  There isn't any performance or simplicity advantage to
> delivering
> spans in time order to these databases (as far as I know, at least).
>
> The advantage is not on the storage front, but rather on the
> consumer side.
> For example, consider a hypothetical messaging application. A sender
> client may
> send a message to a server, the server storing the message until a
> receiver client
> logs-in to consume pending messages. Say a span captures the function
> that sends
> the message, a span captures the time spent between when the message
> is received by
> the server and consumed from it, a span captures the function that
> receives the message
> on the receiver client side. This may be a long lasting (days)
> interaction, but a consumer
> will not be able to access any of the temporary information until the
> whole transaction
> is completed. Similarly, you mention later a UI visualization issue.

I feel like there are two separate issues here:
1. The order that spans get sent to the SpanReceiver
2. Whether spans get sent prior to being closed, or whether we wait
   until they're closed to send them.

I don't think #1 is that important.  The SpanReceivers inherently
receive a stream of spans from many different threads.  Even if the
spans are in order with respect to a single thread, they will not be in
time order across all threads.

#2 is a tradeoff.  If you send out spans prior to closing them, then you
#have to do at least two sends-- since you need to update the span with
#the new end time.  My proposal for addressing #2 is to have a timeout
#beyond which we send the unclosed span.  I believe this is a good
#tradeoff which avoids sending short-lived spans twice, but still allows
#long-lived spans to work well.

> Of course, in a distributed system, just because node A sends
> out a span
> before some other node B doesn't mean that node A's spans will arrive
> before B's in the distributed database.  And also, multiple
> threads and
> nodes will be sending spans to the database, so the input to the
> database will not be in strictly ascending time order anyway.
>
> >
> > Also, the fact that information about a span is not delivered
> > until the
> > span is closed, relies on the program not terminating abruptly.
> > In Java
> > this is not so much of a problem, but in C what happens if a
> > series of
> > nested function calls is instrumented with spans, and the innermost
> > function crashes? As far as I can tell none of the span is
> > delivered.
> > This makes the use of the tracing API unreliable for bug analysis.
>
> I definitely agree that it is frustrating when a progra

Re: Infinite Loop generating Dependency-reduced POM

[Colin McCabe]

I believe this is one reason why projects sometimes wrap Maven in
downloader scripts-- to ensure that they get the right version of Maven,
and avoid having to think about the quirks of any other version.  For
example, OpenZipkin has this:
https://github.com/openzipkin/zipkin/blob/master/mvnw

As much as I hate to add any more complexity to the build system, if new
contributors keep stumbling on problems like HTRACE-234, we might have
to.

I really wish someone would figure out a fix for HTRACE-234. 
Unfortunately, it's pretty darn difficult...



On Wed, Sep 14, 2016, at 13:40, Mike Drob wrote:
> Thanks, Colin. Switching to maven 3.0.4 solved it for me.
> 
> On Wed, Sep 14, 2016 at 3:02 PM, Colin McCabe <cmcc...@apache.org> wrote:
> 
> > Hi Mike,
> >
> > I believe this is HTRACE-234.
> >
> > https://issues.apache.org/jira/browse/HTRACE-234
> >
> > The workaround is to use Maven 3.0.4 rather than a later version.  :(
> >
> > Unfortunately, this appears to result from a bug in the maven-shade
> > plugin.  I wish we had a better workaround, but right now we don't...
> >
> > best,
> > Colin
> >
> >
> > On Wed, Sep 14, 2016, at 12:59, Mike Drob wrote:
> > > Hey folks,
> > >
> > > I've run into an issue where when I am building htrace the process gets
> > > stuck creating a dep-reduced pom for htrace-hbase module over and over
> > > again. I'm not sure which plugin is even asking for this pom file, or
> > > what
> > > it gets used for/how it is generated. If anybody has seen anything like
> > > this, or has any advice, I'd be very appreciative.
> > >
> > > I can reproduce it consistently with
> > > mvn clean package -pl htrace-hbase -am -DskipTests
> > >
> > > My default maven/java details are here. Switching to java 7 did not
> > > change
> > > things.
> > >
> > > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > > 2015-11-10T10:41:47-06:00)
> > > Maven home: /opt/apache-maven-3.3.9
> > > Java version: 1.8.0_101, vendor: Oracle Corporation
> > > Java home: /usr/lib/jvm/java-8-oracle/jre
> > > Default locale: en_US, platform encoding: UTF-8
> > > OS name: "linux", version: "3.13.0-95-generic", arch: "amd64", family:
> > > "unix"
> > >
> > > When running with debug logging turned on, I see the following set of
> > > lines
> > > printed:
> > >
> > > [DEBUG] Extension realms for project
> > > org.apache.htrace:htrace-hbase:jar:4.2.0-incubating-SNAPSHOT: (none)
> > > [DEBUG] Looking up lifecyle mappings for packaging jar from
> > > ClassRealm[plexus.core, parent: null]
> > > [DEBUG] Extension realms for project
> > > org.apache.htrace:htrace:pom:4.2.0-incubating-SNAPSHOT: (none)
> > > [DEBUG] Looking up lifecyle mappings for packaging pom from
> > > ClassRealm[plexus.core, parent: null]
> > > [DEBUG] Extension realms for project org.apache:apache:pom:17: (none)
> > > [DEBUG] Looking up lifecyle mappings for packaging pom from
> > > ClassRealm[plexus.core, parent: null]
> > > [DEBUG] building maven31 dependency graph for
> > > org.apache.htrace:htrace-hbase:jar:4.2.0-incubating-SNAPSHOT
> > > [DEBUG] Dependency collection stats: {ConflictMarker.analyzeTime=0,
> > > ConflictMarker.markTime=0, ConflictMarker.nodeCount=48,
> > > ConflictIdSorter.graphTime=0, ConflictIdSorter.topsortTime=0,
> > > ConflictIdSorter.conflictIdCount=28,
> > > ConflictIdSorter.conflictIdCycleCount=0, ConflictResolver.totalTime=0,
> > > ConflictResolver.conflictItemCount=21,
> > > DefaultDependencyCollector.collectTime=1,
> > > DefaultDependencyCollector.transformTime=0}
> > > [DEBUG] org.apache.htrace:htrace-hbase:jar:4.2.0-incubating-SNAPSHOT
> > > [DEBUG]
> > > org.apache.htrace:htrace-core4:jar:4.2.0-incubating-SNAPSHOT:provided
> > > [DEBUG]   commons-logging:commons-logging:jar:1.1.1:provided
> > > [DEBUG]
> > > org.apache.htrace:htrace-core4:jar:tests:4.2.0-incubating-SNAPSHOT:test
> > > [DEBUG]   com.fasterxml.jackson.core:jackson-core:jar:2.4.0:test
> > > [DEBUG]   com.fasterxml.jackson.core:jackson-databind:jar:2.4.0:test
> > > [DEBUG]
> > > com.fasterxml.jackson.core:jackson-annotations:jar:2.4.0:test
> > > [DEBUG]junit:junit:jar:4.11:test
> > > [DEBUG]org.apache.hbase:hbase-common:jar:1.1.2:provided
> > > [DEBUG]   com.google.protobuf:protobuf-java:jar:2.5.0:provided
> > > [DEBUG]org.apache.hbase:hbase-client:jar:1.1.2:provided
> > > [DEBUG]org.apache.hbase:hbase-testing-util:jar:1.1.2:test
> > > [INFO] Dependency-reduced POM written at:
> > > /home/mdrob/workspace/htrace/htrace-hbase/dependency-reduced-pom.xml
> > >
> > >
> > > Thanks,
> > > Mike
> >

Re: Infinite Loop generating Dependency-reduced POM

[Colin McCabe]

Hi Mike,

I believe this is HTRACE-234.

https://issues.apache.org/jira/browse/HTRACE-234

The workaround is to use Maven 3.0.4 rather than a later version.  :(

Unfortunately, this appears to result from a bug in the maven-shade
plugin.  I wish we had a better workaround, but right now we don't...

best,
Colin


On Wed, Sep 14, 2016, at 12:59, Mike Drob wrote:
> Hey folks,
> 
> I've run into an issue where when I am building htrace the process gets
> stuck creating a dep-reduced pom for htrace-hbase module over and over
> again. I'm not sure which plugin is even asking for this pom file, or
> what
> it gets used for/how it is generated. If anybody has seen anything like
> this, or has any advice, I'd be very appreciative.
> 
> I can reproduce it consistently with
> mvn clean package -pl htrace-hbase -am -DskipTests
> 
> My default maven/java details are here. Switching to java 7 did not
> change
> things.
> 
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T10:41:47-06:00)
> Maven home: /opt/apache-maven-3.3.9
> Java version: 1.8.0_101, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-8-oracle/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.13.0-95-generic", arch: "amd64", family:
> "unix"
> 
> When running with debug logging turned on, I see the following set of
> lines
> printed:
> 
> [DEBUG] Extension realms for project
> org.apache.htrace:htrace-hbase:jar:4.2.0-incubating-SNAPSHOT: (none)
> [DEBUG] Looking up lifecyle mappings for packaging jar from
> ClassRealm[plexus.core, parent: null]
> [DEBUG] Extension realms for project
> org.apache.htrace:htrace:pom:4.2.0-incubating-SNAPSHOT: (none)
> [DEBUG] Looking up lifecyle mappings for packaging pom from
> ClassRealm[plexus.core, parent: null]
> [DEBUG] Extension realms for project org.apache:apache:pom:17: (none)
> [DEBUG] Looking up lifecyle mappings for packaging pom from
> ClassRealm[plexus.core, parent: null]
> [DEBUG] building maven31 dependency graph for
> org.apache.htrace:htrace-hbase:jar:4.2.0-incubating-SNAPSHOT
> [DEBUG] Dependency collection stats: {ConflictMarker.analyzeTime=0,
> ConflictMarker.markTime=0, ConflictMarker.nodeCount=48,
> ConflictIdSorter.graphTime=0, ConflictIdSorter.topsortTime=0,
> ConflictIdSorter.conflictIdCount=28,
> ConflictIdSorter.conflictIdCycleCount=0, ConflictResolver.totalTime=0,
> ConflictResolver.conflictItemCount=21,
> DefaultDependencyCollector.collectTime=1,
> DefaultDependencyCollector.transformTime=0}
> [DEBUG] org.apache.htrace:htrace-hbase:jar:4.2.0-incubating-SNAPSHOT
> [DEBUG]
> org.apache.htrace:htrace-core4:jar:4.2.0-incubating-SNAPSHOT:provided
> [DEBUG]   commons-logging:commons-logging:jar:1.1.1:provided
> [DEBUG]
> org.apache.htrace:htrace-core4:jar:tests:4.2.0-incubating-SNAPSHOT:test
> [DEBUG]   com.fasterxml.jackson.core:jackson-core:jar:2.4.0:test
> [DEBUG]   com.fasterxml.jackson.core:jackson-databind:jar:2.4.0:test
> [DEBUG]
> com.fasterxml.jackson.core:jackson-annotations:jar:2.4.0:test
> [DEBUG]junit:junit:jar:4.11:test
> [DEBUG]org.apache.hbase:hbase-common:jar:1.1.2:provided
> [DEBUG]   com.google.protobuf:protobuf-java:jar:2.5.0:provided
> [DEBUG]org.apache.hbase:hbase-client:jar:1.1.2:provided
> [DEBUG]org.apache.hbase:hbase-testing-util:jar:1.1.2:test
> [INFO] Dependency-reduced POM written at:
> /home/mdrob/workspace/htrace/htrace-hbase/dependency-reduced-pom.xml
> 
> 
> Thanks,
> Mike

Re: c++ support

[Colin McCabe]

Hi Byung-Gon,

Hmm.  The lack of an API for getting the current trace ID does look like
an omission!  I think we should add this.  It shouldn't be too
difficult.

I would say, open a JIRA so we can take a look.

best,
Colin

On Tue, Sep 13, 2016, at 14:57, Byung-Gon Chun wrote:
> Hi,
> 
> We've been using HTrace for Java. This really worked well. Thanks!
> 
> We hope to trace C++ code with HTrace as well. But it looks like C++ does
> not expose all the features we need (e.g., getting a span id). What's the
> best way to approach this?
> 
> Thanks.
> Byung-Gon Chun

Re: HTrace API comments

[Colin McCabe]

On Sat, Sep 10, 2016, at 20:04, Roberto Attias wrote:
> Hello,I have some comment/concerns regarding the HTrace API, and was
> wondering whether extensions/changes would be considered. I'm listing the
> most important here, if there is interest we can discuss more in detail.

Welcome, Roberto!

> 
> 1) From the HTrace Developer Guide: 
> 
> 
> 
> TraceScope objects manage the lifespan of Span objects. When a TraceScope
> is created, it often comes with an associated Span object. When this
> scope is closed, the Span will be closed as well. “Closing” the scope
> means that the span is sent to a SpanReceiver for processing.
> 
> 
> One of the implications of this model is the fact that nested spans (for
> example instrumenting nested function calls) will be delivered to the
> receiver in reverse order (as the innermost function completes before the
> outermost. This may introduce more complexity on the logic in the span
> receiver. 

Hmm.  While I would never say never, in the existing span receivers, we
haven't found that delivering the spans in this order results in any
extra complexity.  What you want is a span sink that aggregates all the
spans together, and supports querying spans by various things like ID,
time, etc.  This is typically a distributed database like HBase, Kudu,
etc.  There isn't any performance or simplicity advantage to delivering
spans in time order to these databases (as far as I know, at least).

Of course, in a distributed system, just because node A sends out a span
before some other node B doesn't mean that node A's spans will arrive
before B's in the distributed database.  And also, multiple threads and
nodes will be sending spans to the database, so the input to the
database will not be in strictly ascending time order anyway.

> 
> Also, the fact that information about a span is not delivered until the
> span is closed, relies on the program not terminating abruptly. In Java
> this is not so much of a problem, but in C what happens if a series of
> nested function calls is instrumented with spans, and the innermost
> function crashes? As far as I can tell none of the span is delivered.
> This makes the use of the tracing API unreliable for bug analysis.

I definitely agree that it is frustrating when a program crashes with
spans which are buffered.  This can happen in both Java and .. although
our out-of-the-box handling of shutdown hooks is better in Java.  This
problem is difficult to avoid completely for a few different reasons:

1. As you commented, we don't output spans until they're complete...
i.e., closed.
2. Without buffering, we end up doing an RPC per span, which is too
costly in real-world systems

I would also add, one thing that is frustrating sometimes is how very
long-running spans don't show up for a while in the GUI.

> 
> Would you consider a change where each API call produces at least one
> event sent to the SpanReceiver? 

It would be interesting to think about giving users (or maybe
spanreceivers?) the option of receiving the same span twice: once when
it was first opened, and once when it was completed.  Or maybe having
spans which were uncompleted for a certain amount of time sent out, to
better avoid losing them in a crash.

We'd have to think carefully about this to avoid overwhelming users with
configuration knobs.  And we'd also have to document that SpanReceivers
would have to be able to handle receiving the same span twice. 
Hopefully the consistency implications don't get too tricky.

> 
> 2) HTrace has a concept of spans having one or more parents.  This
> allows, for example, to capture the fact that a process makes an RPC call
> to another.  However, there is no information about when within the span
> the caller calls the callee. A caller span may have two child spans,
> representing the fact that it made two RPC calls, but the order in which
> those were made is lost in the model (using the timestamps associated to
> the begin of the callee spans is not feasible, as there may be different
> RPC latencies, or simply the clocks may not be aligned. Also, the only
> relation captured by the API is between blocks. 

In your example, is the caller span making the two RPCs in parallel?  If
so, it might be appropriate to say that the spans don't have a
well-defined ordering.  Certainly we don't have any guarantees about
which one will be processed first.  Which one was initiated first
doesn't seem very interesting-- unless I'm missing something.

> 
> I propose a more general API with a concept of spans and  points
> (timestamped sets of annotations), and cause-effect relationship among
> points. an RPC call can be represented as a point in the caller span
> marked as cause, and a  (begin) point in the callee span marked as
> effect. This is very flexible and allow to capture all sorts of
> relationship, not just parent child. for example, a DMA operation may be
> initiated in a block  and captured as a point, the completion captured as
> a point 

Re: Podling Report Reminder - September 2016

[Colin McCabe]

Thanks, Lewis and Jake.

+1 (non-binding)

best,


On Wed, Sep 7, 2016, at 13:14, Jake Farrell wrote:
> Looks good with a couple slight modifications made, updated and signed
> off
> on
> 
> -Jake
> 
> On Wed, Sep 7, 2016 at 2:22 PM, lewis john mcgibbney 
> wrote:
> 
> > Done, Please scope out and sign off or add.
> > Thanks Jake and others.
> > Lewis
> >
> > On Wed, Sep 7, 2016 at 10:28 AM, <
> > dev-digest-h...@htrace.incubator.apache.org> wrote:
> >
> > >
> > > dev Digest 7 Sep 2016 17:28:19 - Issue 180
> > >
> > > Topics (messages 831 through 831)
> > >
> > > Re: Podling Report Reminder - September 2016
> > > 831 by: Jake Farrell
> > >
> > > Administrivia:
> > >
> > > -
> > > To post to the list, e-mail: dev@htrace.incubator.apache.org
> > > To unsubscribe, e-mail: dev-digest-unsubscribe@htrace.
> > incubator.apache.org
> > > For additional commands, e-mail: dev-digest-help@htrace.
> > > incubator.apache.org
> > >
> > > --
> > >
> > >
> > >
> > > -- Forwarded message --
> > > From: Jake Farrell 
> > > To: "dev@htrace.incubator.apache.org" 
> > > Cc:
> > > Date: Wed, 7 Sep 2016 13:28:14 -0400
> > > Subject: Re: Podling Report Reminder - September 2016
> > > Our podling report is due today, anyone that has not put it together yet
> > > want to give it a shot?
> > >
> > > -Jake
> > >
> > >
> > >
> > > On Sun, Sep 4, 2016 at 5:46 PM,  wrote:
> > >
> > > > Dear podling,
> > > >
> > > > This email was sent by an automated system on behalf of the Apache
> > > > Incubator PMC. It is an initial reminder to give you plenty of time to
> > > > prepare your quarterly board report.
> > > >
> > > > The board meeting is scheduled for Wed, 21 September 2016, 10:30 am
> > PDT.
> > > > The report for your podling will form a part of the Incubator PMC
> > > > report. The Incubator PMC requires your report to be submitted 2 weeks
> > > > before the board meeting, to allow sufficient time for review and
> > > > submission (Wed, September 07).
> > > >
> > > > Please submit your report with sufficient time to allow the Incubator
> > > > PMC, and subsequently board members to review and digest. Again, the
> > > > very latest you should submit your report is 2 weeks prior to the board
> > > > meeting.
> > > >
> > > > Thanks,
> > > >
> > > > The Apache Incubator PMC
> > > >
> > > > Submitting your Report
> > > >
> > > > --
> > > >
> > > > Your report should contain the following:
> > > >
> > > > *   Your project name
> > > > *   A brief description of your project, which assumes no knowledge of
> > > > the project or necessarily of its field
> > > > *   A list of the three most important issues to address in the move
> > > > towards graduation.
> > > > *   Any issues that the Incubator PMC or ASF Board might wish/need to
> > be
> > > > aware of
> > > > *   How has the community developed since the last report
> > > > *   How has the project developed since the last report.
> > > >
> > > > This should be appended to the Incubator Wiki page at:
> > > >
> > > > http://wiki.apache.org/incubator/September2016
> > > >
> > > > Note: This is manually populated. You may need to wait a little before
> > > > this page is created from a template.
> > > >
> > > > Mentors
> > > > ---
> > > >
> > > > Mentors should review reports for their project(s) and sign them off on
> > > > the Incubator wiki page. Signing off reports shows that you are
> > > > following the project - projects that are not signed may raise alarms
> > > > for the Incubator PMC.
> > > >
> > > > Incubator PMC
> > > >
> > >
> > >
> >
> >
> > --
> > http://home.apache.org/~lewismc/
> > @hectorMcSpector
> > http://www.linkedin.com/in/lmcgibbney
> >

Re: Time for another release?

[Colin McCabe]

On Fri, Aug 26, 2016, at 07:06, Mike Drob wrote:
> HTrace devs,
> 
> It's been a while since our last update, and it looks like there are
> several good fixes slated for the 4.2 version. What do you all think
> about
> releasing soon?

+1.

> 
> There's a few issues that I think are close to wrapping up, or could be
> included soon, but I didn't see any blockers in JIRA. The only question
> in
> my mind is how close we are on HTRACE-376
> .

I think it's ready to submit.  It makes HTrace more generally useful,
extending it to use-cases where you need finer granularlity than
milliseconds-- and it does it in a fully backwards-compatible way.  The
4.2 release probably would go more smoothly if we bumped HTRACE-376 to
4.3, though.

cheers,
Colin

> 
> Meanwhile, can somebody create a 4.3 version in JIRA so that incomplete
> issues have a place to go?
> 
> Thanks,
> Mike

Re: Nike Wingtips

[Colin McCabe]

Thanks, Sean.  I hadn't seen this before.  It seems very similar to
Zipkin in most respects.  It has the Zipkin-style separation between
TraceId and SpanId, as well as 64-bit random IDs for both.

It seems like an odd choice to include duration as well as start time
and end time.  Can't duration be calculated from these other two?  Did
Zipkin do that too?  I can't remember.

Spans have a userId, which I assume is kind of similar to our TracerId. 
Trying to identify who did what.

I'm surprised that there are no arbitrary key/value annotations
permitted.  They mention this as TBD.

It looks like everything is global in Tracer.java so better be prepared
for have traced libraries stomp on each other's configuration.

Probably the most interesting choice is the decision to hard-code
everything to go to the Java logging framework.  I guess the idea is you
can use "grep" or something to separate the trace JSON from the logfiles
later, and then run your existing analytics stack on that data.  Or
maybe Nike has their java logging hooked up to something more
sophisticated than a file-based sink (thinking about it more, this is
probably the rationale.)

Should HTrace implement a log4j trace sink?  It would certainly be easy
for us to implement, and maybe useful for people who want to experiment.
 We have our stable JSON API, so it's just a matter of outputting that
information.

cheers,
Colin


On Tue, Jul 26, 2016, at 13:55, Sean Busbey wrote:
> did folks see that Nike has opensourced a distributed tracing system
> based on Dapper?
> 
> https://github.com/Nike-Inc/wingtips
> 
> 
> -- 
> busbey

Re: Website Branding Issues

[Colin McCabe]

Hi John,

We have added the disclaimer to the site.  Sorry for the delay.

best,
Colin

On Tue, Jul 12, 2016, at 13:06, Colin McCabe wrote:
> I posted a patch on HTRACE-377 to update the markdown files which are
> used to generate the site.  Once this is committed, we'll re-generate
> the site.
> 
> best,
> Colin
> 
> 
> On Sat, Jul 9, 2016, at 08:17, John D. Ament wrote:
> > Ping.  When can this be expected to be resolved?
> > 
> > On 2016-07-05 15:42 (-0400), Colin McCabe <cmcc...@apache.org> wrote: 
> > > Hi John,
> > > 
> > > Hmm.  Looks like you are correct.  Although we do identify ourselves as
> > > "an Apache Incubator project," we don't have the exact text of the
> > > incubator disclaimer, as described on
> > > http://incubator.apache.org/guides/branding.html .  Thanks for bringing
> > > this to our attention.  I filed
> > > https://issues.apache.org/jira/browse/HTRACE-377 to resolve this issue.
> > > 
> > > regards,
> > > Colin
> > > 
> > > 
> > > On Fri, Jul 1, 2016, at 14:55, johndam...@apache.org wrote:
> > > > Dear podling,
> > > > 
> > > > During a recent audit of podling websites, your podling was identified 
> > > > as
> > > > not including the incubating disclaimer.  This disclaimer is required on
> > > > all podling websites, releases and announcements, to clarify that your
> > > > project may not be in compliance with all ASF processes.
> > > > 
> > > > Please review the Incubator's branding guide:
> > > > http://incubator.apache.org/guides/branding.html
> > > > 
> > > > The full list of observations for all projects can be found at:
> > > > https://wiki.apache.org/incubator/BrandingAuditJune2016
> > > > 
> > > > If you have any questions, feel free to respond to this email or email
> > > > our general@ mailing list.  Please note that I am not subscribed to your
> > > > mailing list.
> > > 

Re: Website Branding Issues

[Colin McCabe]

I posted a patch on HTRACE-377 to update the markdown files which are
used to generate the site.  Once this is committed, we'll re-generate
the site.

best,
Colin


On Sat, Jul 9, 2016, at 08:17, John D. Ament wrote:
> Ping.  When can this be expected to be resolved?
> 
> On 2016-07-05 15:42 (-0400), Colin McCabe <cmcc...@apache.org> wrote: 
> > Hi John,
> > 
> > Hmm.  Looks like you are correct.  Although we do identify ourselves as
> > "an Apache Incubator project," we don't have the exact text of the
> > incubator disclaimer, as described on
> > http://incubator.apache.org/guides/branding.html .  Thanks for bringing
> > this to our attention.  I filed
> > https://issues.apache.org/jira/browse/HTRACE-377 to resolve this issue.
> > 
> > regards,
> > Colin
> > 
> > 
> > On Fri, Jul 1, 2016, at 14:55, johndam...@apache.org wrote:
> > > Dear podling,
> > > 
> > > During a recent audit of podling websites, your podling was identified as
> > > not including the incubating disclaimer.  This disclaimer is required on
> > > all podling websites, releases and announcements, to clarify that your
> > > project may not be in compliance with all ASF processes.
> > > 
> > > Please review the Incubator's branding guide:
> > > http://incubator.apache.org/guides/branding.html
> > > 
> > > The full list of observations for all projects can be found at:
> > > https://wiki.apache.org/incubator/BrandingAuditJune2016
> > > 
> > > If you have any questions, feel free to respond to this email or email
> > > our general@ mailing list.  Please note that I am not subscribed to your
> > > mailing list.
> > 

Re: Website Branding Issues

[Colin McCabe]

Hi John,

Hmm.  Looks like you are correct.  Although we do identify ourselves as
"an Apache Incubator project," we don't have the exact text of the
incubator disclaimer, as described on
http://incubator.apache.org/guides/branding.html .  Thanks for bringing
this to our attention.  I filed
https://issues.apache.org/jira/browse/HTRACE-377 to resolve this issue.

regards,
Colin


On Fri, Jul 1, 2016, at 14:55, johndam...@apache.org wrote:
> Dear podling,
> 
> During a recent audit of podling websites, your podling was identified as
> not including the incubating disclaimer.  This disclaimer is required on
> all podling websites, releases and announcements, to clarify that your
> project may not be in compliance with all ASF processes.
> 
> Please review the Incubator's branding guide:
> http://incubator.apache.org/guides/branding.html
> 
> The full list of observations for all projects can be found at:
> https://wiki.apache.org/incubator/BrandingAuditJune2016
> 
> If you have any questions, feel free to respond to this email or email
> our general@ mailing list.  Please note that I am not subscribed to your
> mailing list.

Re: Apache HTrace question

[Colin McCabe]

Hi Nuc,

Welcome!  If you would like to try out HTrace on a real system, Hadoop
is the best choice.

There is some information about setting up HTrace+Hadoop here:
http://blog.cloudera.com/blog/2015/12/new-in-cloudera-labs-apache-htrace-incubating/

If you would like to take a look at the code for HTrace in Hadoop, a
good starting point is here:
https://github.com/apache/hadoop/commit/892ade689f9bcce76daae8f66fc00a49bee8548e

Perhaps one point that might be confusing is that in general, HTrace is
agnostic to the choice of RPC systems.  Also, when HTrace is integrated
into an RPC system, the source code for that integration will be in the
repository where the RPC system is implemented, not in the HTrace code
repository.  In general the integration with RPC is pretty simple-- just
a matter of adding an optional field with an extra 128 bits for a parent
span ID.

best,
Colin


On Sat, Jun 25, 2016, at 04:23, Nuc Andrei wrote:
> Hi,
> 
> I am working on a thesis called “Tracing in distributed systems” and saw
> your project.
> 
> I have a short question. Is there any code example with multiple nodes
> involved in the tracing flow? Based on tests found for htrace-core
> component, I did not manage to build an example that could work in a
> distributed environment.
> 
> Thank you.
> 
> Regards
> Andrei Nuc,
> Technical University of Cluj-Napoca
> Romania.
> 

Re: Span start and end times in nano time

[Colin McCabe]

Hi Pranavan,
 
Thanks for the background information.  I filed
https://issues.apache.org/jira/browse/HTRACE-376 to continue this
discussion.  Definitely agree that more granularity would be nice, at
least as an option.  Also added you to the contributor role on JIRA
(that lets you post patches and so forth if you want).
 
best,
Colin
 
 
On Tue, Jun 14, 2016, at 20:52, Pranavan Theivendiram wrote:
> Hi Colin,
>
> I am working on a GSoC project for Apache Phoenix. The requirement
> came from an issue -
> https://issues.apache.org/jira/browse/PHOENIX-2178  . I will open a
> JIRA and keep the progress.
>
> Thanks
>
> *T. Pranavan*
> **Junior Consultant *| Department of Computer Science & Engineering
> ,University of Moratuwa*
> *Mobile| *0775136836
>
> On 15 June 2016 at 01:32, Colin McCabe <cmcc...@apache.org> wrote:
>> Hi Pranavan,
>>
>>  I'm curious how your original email about $subject was
>>  generated; can
>>  you elaborate on that?  Usually such emails are characteristic
>>  of spam,
>>  but it sounds like yours might not be, based on your response.
>>
>>  The big question about adding nanosecond time support is how
>>  it would
>>  show up in the span JSON.  The easiest way to do it might be
>>  adding a
>>  new field in addition to the existing "milliseconds since the epoch"
>>  field.
>>
>>  Nanosecond-granularity support sounds like a generally useful addition--
>>  definitely open a JIRA if you are interested in this.
>>
>>  best,
>>  Colin
>>
>>
>>  On Mon, Jun 13, 2016, at 22:57, Pranavan Theivendiram wrote:
>>  > Hi Colin,
>>  >
>>  > I am planning to introduce some new methods inside Span interface
>>  > and going to add some variables for nano time inside  MilliSPlan.
>>  > Eventually, it may affect some other code base in minor scale.
>>  > Shall I go ahead with that?
>>  >
>>  > Thanks
>>  >
>> > *T. Pranavan*
>>  > *Junior Consultant | Department of Computer Science & Engineering
>>  > ,University of Moratuwa*
>>  > *Mobile| *0775136836
>>  >
>>  > On 14 June 2016 at 10:32, Colin McCabe <cmcc...@apache.org> wrote:
>>  >
>>  > > Hi Pranavan,
>>  > >
>>  > > Have you tried $solution?
>>  > >
>>  > > Best,
>>  > > Colin
>>  > >
>>  > > On Mon, Jun 13, 2016, at 21:38, Pranavan Theivendiram wrote:
>>  > > >Hi Devs,
>>  > > >
>>  > > > I need $subject. Can anyone help me?
>>  > > >
>>  > > > Thanks
>>  > > > *T. Pranavan*
>>  > > > *Junior Consultant | Department of Computer Science &
>>  > > > Engineering
>>  > > > ,University of Moratuwa*
>>  > > > *Mobile| *0775136836
>>  > >
 

Re: HTrace question

[Colin McCabe]

Just to be completely clear, you can't use HTrace with any distribution
of Hadoop, only with a distribution that supports the specific
instrumentation that HTrace requires.  It's a feature like any other. 
In a similar fashion, for example, you cannot use snapshots or HDFS high
availability with Hadoop 1.x or HDP 1, but you can with Hadoop 2.x or
CDH 5.  Not every Hadoop distribution supports every feature or project.

We are working on better upstream documentation (there has been some
progress on that in the HTrace 4.2 time-frame, and hopefully there will
be even more).

I think it's likely that Hortonworks will support these features in the
future and I am not trying to advocate for any particular vendor.  But
to be concrete about your options, basically you have two choices: using
Cloudera's distribution of Hadoop, or using the upstream distribution of
Hadoop and building it yourself.  The Cloudera docs are here:
http://blog.cloudera.com/blog/2015/12/new-in-cloudera-labs-apache-htrace-incubating/

best,
Colin


On Thu, Jun 9, 2016, at 16:47, Krishnanand Khambadkone wrote:
> Colin, Thank you for the quick response.  My question was really a
> general one.  How would I use the Apache version of HTrace with any
> distribution of hadoop really which happens to be Hortonworks in my case.
>  Is there any documentation that describes how to set it up and how to
> see the metrics using the GUI. 
> 
> On Thursday, June 9, 2016 4:39 PM, Colin McCabe <cmcc...@apache.org>
> wrote:
>  
> 
>  Hi Krishnanand,
> 
> Thanks for looking at HTrace!  This list is for questions about the
> Apache project.  For questions about Hortonworks, please consult the
> Hortonworks mailing lists or bulletin boards.
> 
> With regard to Apache Hadoop, HTrace support is included in the upcoming
> Hadoop 2.8 release.  Unfortunately, the 2.8 release is taking longer
> than expected to get out (in fact, it now appears that Hadoop 3.0 might
> beat it!)
> 
> Cloudera's distribution has support for HTrace because this was
> backported from 2.8.  I do not know if Hortonworks has backported these
> patches or not-- you would have to look that up.  Similarly, I do not
> know if Hortonworks has a binary distribution of HTraced-- if they do, I
> haven't heard about it.
> 
> best,
> Colin
> 
> 
> On Thu, Jun 9, 2016, at 13:57, Krishnanand Khambadkone wrote:
> > Hi,  Is there any user guide on how to install htrace on a Hortonworks
> > HDP 2.3 cluster and access the GUI
> 
>   

Re: HTrace question

[Colin McCabe]

Hi Krishnanand,

Thanks for looking at HTrace!  This list is for questions about the
Apache project.  For questions about Hortonworks, please consult the
Hortonworks mailing lists or bulletin boards.

With regard to Apache Hadoop, HTrace support is included in the upcoming
Hadoop 2.8 release.  Unfortunately, the 2.8 release is taking longer
than expected to get out (in fact, it now appears that Hadoop 3.0 might
beat it!)

Cloudera's distribution has support for HTrace because this was
backported from 2.8.  I do not know if Hortonworks has backported these
patches or not-- you would have to look that up.  Similarly, I do not
know if Hortonworks has a binary distribution of HTraced-- if they do, I
haven't heard about it.

best,
Colin


On Thu, Jun 9, 2016, at 13:57, Krishnanand Khambadkone wrote:
> Hi,  Is there any user guide on how to install htrace on a Hortonworks
> HDP 2.3 cluster and access the GUI

Re: Board Report?

[Colin McCabe]

We can get it by next week at the latest.  Sorry for the delay.

Best,
C.

On Fri, Jun 3, 2016, at 18:59, John D. Ament wrote:
> HTrace PPMC,
> 
> When can a report be expected?
> 
> John

Re: Community Growth through Connections with Distributed Tracing Workgroup + OpenTracing

[Colin McCabe]

There are a few folks around.  Here's a brief summary:

OpenTracing is a project which sits on top of other tracing projects,
and just provides an API.  It's kind of like Apache Gora, which provides
an API on top of multiple Key/Value stores.  Of course, this relies on
the different tracing projects having at least somewhat similar data
models.  Right now, the main backend for OpenTracing is Zipkin.  There
are also some folks interested in creating closed-source backends for
OpenTracing.

Both Zipkin and OpenTracing are open source.  However, they aren't
Apache.  I'm not sure what the governance model is-- that might be a
good thing to ask about.  I think they are handling everything through
GitHub for now.  They have been fairly responsive and reasonable in our
discussions, and generally seem like very nice folks.  Adrian Cole
(Zipkin) might be a good person to ask about OpenTracing.

It would be nice if we could cross-post some discussions between our
list and theirs.  However, I remember most of the discussion taking
place through Gitter, which might make that difficult.  I remember the
biggest point of contention being multiple parents-- there were some
folks who argued against the idea of allowing spans to have multiple
parents.  HTrace supports multi-parent, and Zipkin does not.  It would
be a good project for someone to write code to hook OpenTracing up to
HTrace.

There is also the Linux Foundation Diagnostic and Monitoring Workgroup. 
These guys are mostly interested in tracing things with extremely fine
granularity on single nodes.  There was a lot of discussion about things
like LTT (Linux Trace Toolkit), ftrace, tracing hooks inside the Linux
kernel, etc.  I gave a presentation at the Linux Tracing Summit in 2015.
 See http://tracingsummit.org/wiki/TracingSummit2015  We also talked
about things like _extremely_ low-level tracing using Intel's processor
hooks.  Mathieu Desnoyers might be a good person to ask about this
group.

I feel like fine-grained kernel-level tracing will stay separate from
higher-level tracing for some time yet.  It's pretty difficult to create
a system that hits both of those sweet spots at the same time.

best,
Colin


On Thu, May 19, 2016, at 10:11, Lewis John Mcgibbney wrote:
> Hi Folks,
> Are there any ongoing or planned engagements/collaborations ongoing with
> any of the above? I spoke with a couple of folks @ApacheCon last week and
> they were keen to connect with dev@ on a face-to-face basis to see what
> was
> going on.
> Is/are anyone from HTrace involved in these activities? If not then is
> this
> something that we want to engage in?
> I'm just throwing this out here to see what is going on.
> Thanks
> Lewis
> 
> -- 
> *Lewis*

Re: Obtaining HTrace Logging in Client Applications

[Colin McCabe]

I have gotten log output out of htrace4-core in Hadoop.

I used the line:
> log4j.rootLogger=TRACE,file,console

commons-logging is a kind of shim which is (usually but not always) on
top of log4j.

Perhaps try modifying your log4j settings?  Unfortunately it's hard to
give good advice about log4j since there are so many configurations. 
One thing that helps sometimes is using strace to find out which
log4j.properties file it's really opening.

best,
Colin


On Mon, May 16, 2016, at 21:02, Lewis John Mcgibbney wrote:
> Hi Folks,
> So I consume htrace-core4 in the Nuch Java application. I am however not
> able to see the HTrace DEBUG log statements.
> I am aware that HTrace uses commons-logging and I have tried adding
> commons-logging.properties on my classpath with the following contents
> 
> org.apache.commons.logging.Log=org.apache.commons.logging.impl.Log4JLogger
> log4j.configuration=log4j.properties
> LocalFileSpanReceiver.level=DEBUG
> 
> The log4j.properties file is in the same directory.
> I am unable to see any log statements from HTrace...
> Has anyone else had any luck?
> Thanks
> Lewis
> 
> -- 
> *Lewis*

Re: Seeing Spans in HTraced

[Colin McCabe]

Hi Lewis,

Thanks for trying this out.  The htraced buffer gets flushed after a
configurable amount of time.  It's also flushed if the SpanReceiver gets
shut down (like in a short lived command line program.). So I don't
think we should change the buffer size default.  Maybe we could document
this better, and/or change the buffer flush time default.

Best,
Colin

On Mon, May 16, 2016, at 19:34, Lewis John Mcgibbney wrote:
> Hi Folks,
> For test and development and also for new comers coming to HTrace I am
> not
> sure if it is realistic for such a high default buffer to be set for
> Htraced. Right now is appears that the default value is 16,777,216 bytes
> which is pretty large if one wishes to get a feel for the HTrace
> framework
> generally and HTraced specifically.
> Although I am easily able to work programmatically and hence change this
> default buffer value to anything I want. Everyone else may not be that
> way
> inclined and may instead just wish to, fr example, run the Docker image
> and
> see what HTrace actually does.
> Does anyone else have an opinion on this? My perspective from being
> @ApacheCon last week and from doing my HTrac'ing in Nutch presentation is
> that people just want to see what HTrac'ing looks like. They may not have
> an application which writes 16MB of spans to a buffer out of the box.
> Thanks
> Lewis
> 
> -- 
> *Lewis*

Re: HTraced Documentation

[Colin McCabe]

I filed htrace-374 to add more upstream documentation for htraced. 
Right now the best intro is probably my scale14x talk: 
https://www.socallinuxexpo.org/sites/default/files/presentations/2016.01_scale14x_introducing_apache_htrace.pdf

Best,
Colin

On Thu, May 12, 2016, at 16:26, Lewis John Mcgibbney wrote:
> Hi Folks,
> Where is the htraced documentation?
> Thanks
> Lewis
> 
> -- 
> *Lewis*

Re: Issues Building HTrace Master

[Colin McCabe]

Hmm.  The build works for me as well.  I also think the Go version would
be a good thing to check.  We should probably start setting a minimum Go
version to avoid build problems.

best,
Colin


On Thu, May 5, 2016, at 08:30, Stack wrote:
> It works for me Lewis. godep is failing. Go version? If you run
> ./htrace/htraced/go/gobuild.sh directly, the issue may be clearer?
> St.Ack
> 
> On Wed, May 4, 2016 at 9:00 PM, Lewis John Mcgibbney <
> lewis.mcgibb...@gmail.com> wrote:
> 
> > Hi Folks,
> > when I attempt to build master branch I get the following error
> >
> > lmcgibbn@LMC-032857 /usr/local/incubator-htrace(master) $ mvn -version
> > Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06;
> > 2015-04-22T04:57:37-07:00)
> > Maven home: /usr/local/apache-maven-3.3.3
> > Java version: 1.8.0_66, vendor: Oracle Corporation
> > Java home:
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"
> >
> > mvn clean install -DskipTests -DcreateDependencyReducedPom=false
> >
> > [INFO]
> > 
> > [INFO] Building htrace-htraced 4.2.0-incubating-SNAPSHOT
> > [INFO]
> > 
> > [INFO]
> > [INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ htrace-htraced
> > ---
> > [INFO] Deleting /usr/local/incubator-htrace/htrace-htraced/target
> > [INFO] Deleting /usr/local/incubator-htrace/htrace-htraced/go/build
> > (includes = [], excludes = [])
> > [INFO] Deleting /usr/local/incubator-htrace/htrace-htraced/go/pkg (includes
> > = [], excludes = [])
> > [INFO]
> > [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
> > htrace-htraced ---
> > [INFO]
> > [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @
> > htrace-htraced ---
> > [INFO] Using 'UTF-8' encoding to copy filtered resources.
> > [INFO] skip non existing resourceDirectory
> > /usr/local/incubator-htrace/htrace-htraced/src/main/resources
> > [INFO] Copying 3 resources
> > [INFO]
> > [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @
> > htrace-htraced ---
> > [INFO] Compiling 8 source files to
> > /usr/local/incubator-htrace/htrace-htraced/target/classes
> > [INFO]
> > [INFO] --- maven-antrun-plugin:1.7:run (go_compile) @ htrace-htraced ---
> > [WARNING] Parameter tasks is deprecated, use target instead
> > [INFO] Executing tasks
> >
> > main:
> >  [exec] Installing godep...
> >  [exec] godep: [WARNING]: godep should only be used inside a valid go
> > package directory and
> >  [exec] godep: [WARNING]: may not function correctly. You are probably
> > outside of your $GOPATH.
> >  [exec] godep: [WARNING]:Current Directory:
> > /usr/local/incubator-htrace/htrace-htraced/go
> >  [exec] godep: [WARNING]:$GOPATH:
> >
> > /usr/local/incubator-htrace/htrace-htraced/go/build:/usr/local/incubator-htrace/htrace-htraced/go
> >  [exec] /usr/local/incubator-htrace/htrace-htraced/go/build/godep
> > restore...
> >  [exec] Building 4.2.0-incubating-SNAPSHOT
> > [4a101e0d1c435937feba9fc876882ffbf71259b1]
> >  [exec] github.com/ugorji/go/codec
> >  [exec] htrace/conf
> >  [exec] github.com/alecthomas/units
> >  [exec] github.com/gorilla/context
> >  [exec] github.com/jmhodges/levigo
> >  [exec] github.com/alecthomas/kingpin
> >  [exec] github.com/gorilla/mux
> >  [exec] htrace/common
> >  [exec] htrace/test
> >  [exec] htrace/client
> >  [exec] htrace/htraced
> >  [exec] htrace/htracedTool
> >  [exec] # htrace/htracedTool
> >  [exec] usage: 6l [options] main.6
> >  [exec]   -1use alternate profiling code
> >  [exec]   -8assume 64-bit addresses
> >  [exec]   -B info
> >  [exec] define ELF NT_GNU_BUILD_ID note
> >  [exec]   -Ccheck Go calls to C code
> >  [exec]   -D addr
> >  [exec] data address
> >  [exec]   -E sym
> >  [exec] entry symbol
> >  [exec]   -I interp
> >  [exec] set ELF interp
> >  [exec]   -L dir
> >  [exec] add dir to library path
> >  [exec]   -H head
> >  [exec] header type
> >  [exec]   -Kadd stack underflow checks
> >  [exec]   -Oprint pc-line tables
> >  [exec]   -Qdebug byte-register code gen
> >  [exec]   -R rnd
> >  [exec] address rounding
> >  [exec]   -Scheck type signatures
> >  [exec]   -T addr
> >  [exec] text address
> >  [exec]   -Vprint version and exit
> >  [exec]   -Wdisassemble input
> >  [exec]   -X name value
> >  [exec] define string data
> >  [exec]   -Zclear stack frame on entry
> >  [exec]   -adisassemble output
> >  [exec]   -cdump call graph
> >  [exec]   -ddisable dynamic 

Re: Google Summer of Code project: Apache Kudu backend for Apache HTrace

[Colin McCabe]

Hmm, yeah, the summerofcode URL might need a GSoC login.  D'oh!

We're going to put up an HTRACE JIRA and a design doc soon.  In the
meantime, check out the COMDEV JIRA here:
https://issues.apache.org/jira/browse/COMDEV-192

best,
Colin


On Fri, Apr 22, 2016, at 16:08, Todd Lipcon wrote:
> Great! This should be a cool application.
> 
> I can't seem to access the link - getting a "ruh roh" 403. Any ideas?
> 
> -Todd
> 
> On Fri, Apr 22, 2016 at 1:24 PM, Colin McCabe <cmcc...@apache.org> wrote:
> 
> > On Fri, Apr 22, 2016, at 13:16, Colin McCabe wrote:
> > > Hi all,
> > >
> > > I'm happy to announce that I am going to be mentoring Nisala Mendis this
> > > year in a Google Summer of Code project!  We are going to be working on
> > > adding an Apache Kudu backend for Apache HTrace.
> > >
> > > I think this is a really exciting project-- Kudu will make a great
> > > backend storage engine for HTrace.  There are more details about the
> > > proposal here:
> > >
> > https://summerofcode.withgoogle.com/dashboard/organization/5694656234913792/proposal/4506009544425472/
> > >
> > > P.S. Thanks to everyone who submitted project proposals for HTrace and
> > > other Apache projects.  It was really tough to choose between them and
> > > we greatly appreciate the time that everyone spent on their proposals.
> > >
> > > cheers,
> > > Colin
> >
> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera

Re: Google Summer of Code project: Apache Kudu backend for Apache HTrace

[Colin McCabe]

On Fri, Apr 22, 2016, at 13:16, Colin McCabe wrote:
> Hi all,
> 
> I'm happy to announce that I am going to be mentoring Nisala Mendis this
> year in a Google Summer of Code project!  We are going to be working on
> adding an Apache Kudu backend for Apache HTrace.
> 
> I think this is a really exciting project-- Kudu will make a great
> backend storage engine for HTrace.  There are more details about the
> proposal here:
> https://summerofcode.withgoogle.com/dashboard/organization/5694656234913792/proposal/4506009544425472/
> 
> P.S. Thanks to everyone who submitted project proposals for HTrace and
> other Apache projects.  It was really tough to choose between them and
> we greatly appreciate the time that everyone spent on their proposals. 
> 
> cheers,
> Colin

Re: Mentors for Google Summer of Code 2016?

[Colin McCabe]

The mentor submission period is now closed.

thanks,
C.

On Tue, Apr 19, 2016, at 11:52, Colin McCabe wrote:
> Hi all,
> 
> Good news... I am going to be mentoring an HTrace project for Google
> Summer of Code 2016!
> 
> We got a lot of project submissions, though, and we have an additional
> HTrace project which we'd also like to find a mentor for.  If anyone is
> interested, it's not too late to become a mentor.  See
> https://community.apache.org/gsoc.html for information about applying.
> 
> Please reply soon if you're interested, since we're finalizing project
> assignments now.
> 
> best,
> Colin

Google Summer of Code project: Apache Kudu backend for Apache HTrace

[Colin McCabe]

Hi all,

I'm happy to announce that I am going to be mentoring Nisala Mendis this
year in a Google Summer of Code project!  We are going to be working on
adding an Apache Kudu backend for Apache HTrace.

I think this is a really exciting project-- Kudu will make a great
backend storage engine for HTrace.  There are more details about the
proposal here:
https://summerofcode.withgoogle.com/dashboard/organization/5694656234913792/proposal/4506009544425472/

P.S. Thanks to everyone who submitted project proposals for HTrace and
other Apache projects.  It was really tough to choose between them and
we greatly appreciate the time that everyone spent on their proposals. 

cheers,
Colin

Mentors for Google Summer of Code 2016?

[Colin McCabe]

Hi all,

Good news... I am going to be mentoring an HTrace project for Google
Summer of Code 2016!

We got a lot of project submissions, though, and we have an additional
HTrace project which we'd also like to find a mentor for.  If anyone is
interested, it's not too late to become a mentor.  See
https://community.apache.org/gsoc.html for information about applying.

Please reply soon if you're interested, since we're finalizing project
assignments now.

best,
Colin

Re: [ANNOUNCE] Apache HTrace 3.2.0 (Incubating) released

[Colin McCabe]

Awesome!  Thanks, Abe.  C.
On Jun 2, 2015 10:36 PM, Abraham Elmahrek a...@cloudera.com wrote:

 The Apache HTrace (Incubating) team is pleased to announce the release of
 HTrace 3.2.0.

 Apache HTrace (incubating) is a library that seeks to enable distributed
 tracing in Hadoop.

 This release improves the Java client with better error checking and
 setting parents in a span after the span is created. Also, some issues in
 the local file span receiver are fixed and work has started on a new C/C++
 native client.

 The release is available in maven:
 https://repo1.maven.org/maven2/org/apache/htrace/

 The full change log is available here:

 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315924version=12329181

 Your help and feedback is more than welcome. For more information on how to
 report problems
 and to get involved, visit the project website at
 http://incubator.apache.org/projects/htrace.html.

 The Apache HTrace (Incubating) Team

Re: [VOTE] HTrace 3.2.0 - Release Candidate 0

[Colin McCabe]

* Verified md5 and sha1 sums
* Ran mvn package -Pnative successfully (this includes tests)
* Did mvn install of HTrace 3.2.0 and built Hadoop 2.7.1 against it
* Verified that htraced worked with Hadoop 2.7.1 + HTrace 3.2

+1

best,
Colin

On Tue, Apr 28, 2015 at 2:46 PM, Abraham Elmahrek a...@cloudera.com wrote:
 I've posted our first release candidate here:

   http://people.apache.org/~abe/htrace/releases/3.2.0/

 The jars have been staged here:

   https://repository.apache.org/content/repositories/orgapachehtrace-1015

 This release improves the Java client with better error checking and
 setting parents in a span after the span is created. Also, some issues in
 the local file span receiver are fixed and work has started on a new C/C++
 native client.

 Please vote +1/0/-1 by Friday, May 1st, 2015.

 Thanks,
 Abe

 1.
 https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%203.2.0%20AND%20project%20%3D%20HTRACE%20AND%20status%20%3D%20Resolved%20ORDER%20BY%20priority%20DESC


 Release Notes - HTrace - Version 3.2.0

 ** Sub-task
 * [HTRACE-44] - Add htraced web UI search page skeleton
 * [HTRACE-45] - Add Span details page skeleton
 * [HTRACE-75] - UI should query htraced
 * [HTRACE-76] - Search page: Add +/- filters to search control
 * [HTRACE-77] - htraced gui: add pagination to the search page.
 * [HTRACE-78] - Search page: Enhance begin time and end time selection
 widget
 * [HTRACE-79] - Search page: Make list more tabular with sorting on
 front end
 * [HTRACE-108] - Search Page: Use server side pagination
 * [HTRACE-121] - Details page: Graph parents and children
 * [HTRACE-131] - Port spans.js in htrace-hbase to htraced
 * [HTRACE-134] - Port front end to marionette JS
 * [HTRACE-142] - Details page: Update graph
 * [HTRACE-146] - Search page: Remove client-side sorting and improve
 pagination

 ** Bug
 * [HTRACE-32] - Change span timeline annotations map to be a
 mapstring, string
 * [HTRACE-37] - htraced: serve index.html if the root URL was given
 * [HTRACE-47] - Add Content-Type header in REST server
 * [HTRACE-48] - use -L 1 in format.sh to make it work on macosx
 * [HTRACE-52] - FIgure out content-type handling in JSON API
 * [HTRACE-55] - Add Clean command to htrace go
 * [HTRACE-57] - Fix LocalFileSpanReceiver to avoid adding irrelevant
 wrapper object
 * [HTRACE-68] - Clean up LocalFileSpanReceiver and SpanReceiverBuilder
 a bit
 * [HTRACE-71] - format.sh: only format src/ directory
 * [HTRACE-90] - Remove Guava and shade other depdendencies in HTrace
 subprojects
 * [HTRACE-93] - Add null check to SamplerBuilder
 * [HTRACE-97] - Support both -D and --D when specifying conf vars on
 the command-line
 * [HTRACE-98] - Web Server should use PrefixHandler
 * [HTRACE-99] - log.go fails to create new log files
 * [HTRACE-100] - htraced: put query parameters in the URL, not in the
 GET request body
 * [HTRACE-101] - Add better error-checking to Java HTrace client
 * [HTRACE-103] - Reset unclosed spans after TestBadClient
 * [HTRACE-107] - Support for Greater Than operator in htraced
 * [HTRACE-109] - fix TestHTracedRESTReceiver unit test failures
 * [HTRACE-110] - Fix rat check failure due to
 dependency-reduced-pom.xml in htrace-hbase
 * [HTRACE-111] - HTrace Java client API fixes for 3.2
 * [HTRACE-112] - Fix LocalFileSpanReceiver to avoid BG thread and
 problems around close()
 * [HTRACE-115] - The htraced datastore should use uint64 for span ids
 rather than int64
 * [HTRACE-116] - htraced's data.store.clear option does not work
 * [HTRACE-117] - graph.go: sort children array to get deterministic
 output
 * [HTRACE-118] - Java API: support setting the parents of a span after
 the span is created
 * [HTRACE-119] - detach of NullScope singleton should not fail
 * [HTRACE-123] - fix graphviz functionality in the htrace command
 * [HTRACE-126] - BUILDING.txt should not say that godep is required,
 because we auto-install godep
 * [HTRACE-127] - fix ./build/htrace dumpAll
 * [HTRACE-128] - RAT check hangs on files under
 htrace-core/src/web/lib/rome-2.1.0
 * [HTRACE-132] - Trace#startSpan(String, TraceInfo) must not create
 spans with spanId == 0
 * [HTRACE-133] - HTracedRESTReceiver drops spans when close() is called
 * [HTRACE-140] - Web UI of htraced causes high CPU usage if search
 result is empty
 * [HTRACE-141] - Fix nits of swimlane view of htraced Web UI
 * [HTRACE-148] - Add deserialization utility method to MilliSpan for
 testing outside of htrace-core
 * [HTRACE-151] - htrace-core, htrace-htraced: Use shaded import paths
 * [HTRACE-152] - Fix TestHTracedRESTReceiver
 com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException

 ** Improvement
 * [HTRACE-30] - Add writeSpan REST verb to htraced
 * [HTRACE-33] - Add getUniqueLocalTraceFileName to 

Re: releasing native build artifacts

[Colin McCabe]

+1

C.

On Thu, Apr 23, 2015 at 12:15 PM, Lewis John Mcgibbney
lewis.mcgibb...@gmail.com wrote:
 Hi Colin,
 My understanding is that none of this is yet established on builds.a.o is
 this correct?
 https://builds.apache.org/view/H-L/view/HBase/
 Maybe it would be a nice opportunity to get that sorted out as well.
 Lewis

 On Thu, Apr 23, 2015 at 11:46 AM, Colin P. McCabe cmcc...@apache.org
 wrote:

 Hi all,

 Based on the responses in the earlier thread, it sounds like it's
 getting to be about that time again... time for a new HTrace release
 :)

 I've been thinking about what we should do with our native build
 artifacts.  With Java, we only ever have to build one version of
 everything, since the jar files can run on any architecture and
 supported OS.  But we now have a C client (which is only at an alpha
 level of stability, but still...) and the htraced server, which is
 also not Java.

 Do we want to do builds of this stuff for the common OS/arch
 configurations?  If we did a RHEL6 / x86_64 build and an Ubuntu 12.04
 / x86_64 build, that would cover most of our users, I think.  We could
 also do 32-bit builds, but I don't think anyone is actually using
 32-bit any more in big data land (and if they are, they need to stop
 :)

 This also means we would have 3 downloads available in the next release:
 * Just jar files + source
 * Jar files + source + RHEL6/x86_64 libhtrace.so + htraced
 * Jar files + source + Ubuntu 12.04/x86_64 libhtrace.so + htraced

 thoughts?

 Colin




 --
 *Lewis*

Re: HTrace GUI meetup

[Colin McCabe]

Great.  4/3 at 5PM PDT sounds like the time, then.  Lewis, does that
work for you?

On Fri, Mar 27, 2015 at 4:49 PM, Masatake Iwasaki
iwasak...@oss.nttdata.co.jp wrote:
 Hmm.  Sounds like May is unavailable on Tuesday.  Does Friday (4/3) at
 5PM PDT work for everyone?  Masatake, I'd especially like to talk to
 you about some ideas for the spans GUI...

 4/3 is ok for me.
 I'm in JST now and will join by Hangouts.
 I appreciate if you forward voice to Hangouts.

It might be a bit tough to have audio on both hangouts and webex.  We
tend to get the echo chamber effect when we try to do that.  I can
understand the desire not to make an international call, though.
Maybe let's just forget about the phone connection and just do
hangouts.  If we run into problems we can try something else at that
point.

I will try to get a hardline (i.e. non-wifi) so maybe the VoIP will work out.

cheers.
Colin



 Masatake



 On 3/28/15 03:19, Colin P. McCabe wrote:
 Hmm.  Sounds like May is unavailable on Tuesday.  Does Friday (4/3) at
 5PM PDT work for everyone?  Masatake, I'd especially like to talk to
 you about some ideas for the spans GUI...

 Abe, to respond to your points: We will post the minutes to the
 mailing list.  Also, obviously the actual implementation of any of
 these ideas will happen on JIRA through the usual process.  I view
 this as kind of similar to the meetings we do occasionally on Hadoop
 to coordinate a new feature that the community is working on.

 Also, on a semi-related note, I'm going to try to check some
 real-world span data into the repo to use when looking at the GUI.

 cheers,
 Colin


 On Thu, Mar 26, 2015 at 3:46 PM, Abraham Elmahrek a...@cloudera.com
 wrote:
 Hey Colin,

 I'm hugely +1 on this! I'd prefer Tuesday @ 5PM PST.

 There has been some discussion about doing this in the Sqoop community as
 well, so I thought I'd relay some of the ideas that popped up there (

 http://mail-archives.apache.org/mod_mbox/sqoop-dev/201502.mbox/%3CCAHUddLM98%3DM4N4qNGfpAThQ%2BEpRf0war0FN4WM%3D3T6t7Owt4nQ%40mail.gmail.com%3E
 ):

1. Meeting minutes are persisted (wiki) and communicated (mailing
 list)
2. No concrete decisions are made (we should run votes first so every
one can participate and make sure full context is provided some how)
3. Proper notice is given to the community and the meeting is globally
available (which it seems we have).

 I think HTrace is a different community and can run it however we see
 fit.
 But I hope the above helps at least for reference.

 -Abe

 On Thu, Mar 26, 2015 at 3:29 PM, Colin P. McCabe cmcc...@apache.org
 wrote:

 Hi all,

 There's been a lot of really great work on the HTrace GUI recently.
 Masatake's span visualization screen, Abe's work on the details page,
 and May's work come to mind.

 I was thinking, we should have a phone call to talk about the GUI.  I
 have some ideas that might be really cool.  Abe also came up with some
 mockups.

 Some time next week would work well for me.  Maybe next Tuesday (3/31)
 at 5pm, or next Friday (4/3) at 5pm PST?  (I realize 5pm PST is late
 in California but I'm trying to come up with something that works for
 all the time zones.)  Does that work for you guys?

 I'm thinking we can provide a plain ol' telephone dial-in number and
 use Google Hangouts for screen sharing.  (We could use Google Hangouts
 for voice as well, but in my experience, it's best to use regular
 phone for voice to avoid glitches.)

 best,
 Colin

Re: [VOTE] htrace-3.1.0, fifth release candidate

[Colin McCabe]

I verified the signature, untarred, ran mvn test with no problems.  I
did see that we are leaking a few jackson classes on to the CLASSPATH
in htrace-core:

jar tvf ./htrace-core/target/htrace-core-3.1.0.jar

  2487 Wed Dec 31 14:29:18 PST 2014
org/apache/htrace/fasterxml/jackson/core/util/MinimalPrettyPrinter.class
  8621 Wed Dec 31 14:29:18 PST 2014
org/apache/htrace/fasterxml/jackson/core/util/TextBuffer.class
  6419 Wed Dec 31 14:29:18 PST 2014
org/apache/htrace/fasterxml/jackson/core/util/VersionUtil.class
 0 Wed Dec 31 14:29:18 PST 2014 com/
 0 Wed Dec 31 14:29:18 PST 2014 com/fasterxml/
 0 Wed Dec 31 14:29:18 PST 2014 com/fasterxml/jackson/
 0 Wed Dec 31 14:29:18 PST 2014 com/fasterxml/jackson/databind/
   801 Wed Dec 31 14:29:18 PST 2014
com/fasterxml/jackson/databind/AbstractTypeResolver.class
  1506 Wed Dec 31 14:29:18 PST 2014
com/fasterxml/jackson/databind/AnnotationIntrospector$ReferenceProperty$Type.class
...

This could potentially be an issue because some of our users want to
use a different version of jackson than what we're using.  I posted a
fix at https://issues.apache.org/jira/browse/HTRACE-46.  Not sure if
this is worth spinning a new RC over.

Colin

On Wed, Dec 31, 2014 at 1:32 PM, Elliott Clark ecl...@apache.org wrote:
 +1
 Verified sig.
 un-tar'd the src tar.
 Verified that all tests pass
 mvn package creates go binaries.
 Those binaries run and listen on a port
 maven repository looks good and has the needed jars.

 On Wed, Dec 31, 2014 at 1:01 PM, Stack st...@duboce.net wrote:

 I've posted our fifth release candidate here (Thanks all for finding issues
 in previous RCs):

   http://people.apache.org/~stack/htrace-3.1.0RC4/

 Maven artifacts are available here:

  https://repository.apache.org/content/repositories/orgapachehtrace-1006

 RC4 has 4 fixes beyond RC3.

 It is a src-only tarball for now. Later we can add binary bundles to our
 release after we have better sense of what we as a project would like to
 deliver.

 This release is mainly a change of packaging from org.htrace to
 org.apache.htrace to suit our new home here in Apache Incubator but it does
 also includes 32 resolved issues [1] including the beginnings of an htrace
 daemon whose intent is to make it so there is a low barrier collecting
 cluster traces as well as a new flume receiver.

 Beware that Apache Incubator, org.apache.htrace is not compatible with its
 former self, org.htrace: the package name has changed but so has the JSON
 serialization format.

 Shall we make this release candidate our first incubator release?

 Lets keep the vote period short (We'll have to run another vote over in
 incubator general after this one if I understand the process properly).

 Please vote +1/0/-1 by Friday, January 2nd, 2015.

 Thanks,
 St.Ack

 1.

 https://issues.apache.org/jira/issues/?jql=project%20%3D%20HTRACE%20AND%20status%20%3D%20resolved%20AND%20fixVersion%20%3D%203.1.0%20ORDER%20BY%20issuetype%20DESC

Re: Created/Resolved JIRA actions to dev list?

[Colin McCabe]

+1.

Colin

On Fri, Dec 19, 2014 at 11:12 AM, Andrew Purtell apurt...@apache.org wrote:
 When watching HBase and HDFS (and other Hadoop) JIRAs, I rely on their
 configuration where create and resolution events are copied to dev@ to
 become aware of newly filed or resolved issues. Individual issues can be
 watched. This is a great middle ground between nothing and an issues@ list
 flood. Please consider it for HTrace.

 On Wed, Dec 17, 2014 at 2:43 PM, Colin McCabe cmcc...@alumni.cmu.edu
 wrote:

 I too am used to JIRAs going to the list.  I've been sending both
 dev@htrace.incubator.apache.org and issues@ to the same gmail label to get
 that setup for myself.  I would have no objection to merging the lists,
 although I don't feel strongly about it.

 Colin
 On Dec 16, 2014 6:08 PM, Jake Farrell jfarr...@apache.org wrote:

  Right now its setup to send jira notifications to
  iss...@htrace.incubator.apache.org, this helps keep the dev list a
 little
  cleaner and easier to follow. If everyone would like to switch and use
 the
  dev list for all jira notices then we can change it
 
  -Jake
 
  On Tue, Dec 16, 2014 at 5:59 PM, Nick Dimiduk ndimi...@gmail.com
 wrote:
  
   What does it take to setup the above? I like how HBase does it :)
  
 



 --
 Best regards,

- Andy

 Problems worthy of attack prove their worth by hitting back. - Piet Hein
 (via Tom White)

Re: Lets make our first release

[Colin McCabe]

+1

Colin

On Tue, Dec 16, 2014 at 9:40 AM, Billie Rinaldi bil...@apache.org wrote:
 I wouldn't mind getting HTRACE-6 in; that will be an easy one. I'm
 investigating HTRACE-5 today and will post whatever I discover.
 On Dec 15, 2014 9:33 PM, Stack st...@duboce.net wrote:

 Unless objection, I'll put up an RC in the next day or so (Maybe I should
 wait on Billie getting in what she needs for accumulo?).  The master branch
 is in a state that roughly equates to the 3.0.4 release caveat the new
 packaging and some extras like the 'site' and Colin's work in core but
 otherwise, its effectively the same functionality.

 Thanks,
 St.Ack

Re: What version should be the first apache htrace be? 4.0.0 or 1.0.0?

[Colin McCabe]

Makes sense to me.  +1 for 3.1.0-SNAPSHOT

best,
Colin

On Fri, Dec 5, 2014 at 11:56 PM, Elliott Clark ecl...@apache.org wrote:
 Most projects I have seen that changed their maven artifact org have
 continued from the old one (netty et al).
 +1 for using 3.1 for our first releases.

 On Fri, Dec 5, 2014 at 9:46 PM, Lewis John Mcgibbney 
 lewis.mcgibb...@gmail.com wrote:

 Hi Folks,
 +0 on keeping versioning going 3.1.X or 3.2
 I'll get more involved as the weeks go on.
 Have a great weekend folks
 Lewis

 On Fri, Dec 5, 2014 at 9:36 PM, Nick Dimiduk ndimi...@gmail.com wrote:

  I think going backwards to 1.0 would be confusing for any existing users.
  Maybe make the -incubating releases pickup 3.1.x with the intention of
 the
  first graduated release being 4.0.0. Could be seen as artificially
  inflating the version numbers, but I don't think that matters too much. I
  assume (prefer) we'll follow the guidelines of semantic versioning.
 
  -n
 
  On Friday, December 5, 2014, Colin McCabe cmcc...@alumni.cmu.edu
 wrote:
 
   I looked at
  
 
 http://incubator.apache.org/guides/releasemanagement.html#best-practice-versioning
   and it doesn't say whether we need to start at 1.  Hmm.
  
   I think either way could work.  There is stuff from org.htrace up on
   Maven central, but since we're moving to org.apache.htrace, we won't
   conflict if we choose to go back to 1.0.0.  I don't really have any
   preference between 1.0.0 or 4.0.0.
  
   best,
   Colin
  
   On Fri, Dec 5, 2014 at 7:47 PM, Stack st...@duboce.net
 javascript:;
   wrote:
org.htrace was at 3.0.4
   
The next release could be 4.0.0.
   
Or we could roll back and make it 1.0.0?
   
Any opinions out there?
   
Thanks,
St.Ack
  
 



 --
 *Lewis*