HTrace 4.1 release candidate 1

[Colin P. McCabe]

Hi all,

I've posted the first release candidate for HTrace 4.1 here:

http://people.apache.org/~cmccabe/htrace/releases/4.1.0/rc1/

The jars have been staged here:

https://repository.apache.org/content/repositories/orgapachehtrace-1021

HTrace 4.1 brings a lot of robustness improvements.  There were major
improvements to htraced and the web UI, as well as new metrics added.
There were numerous build fixups, and we added Docker support, to
ensure a repeatable build.

Check it out.  The vote will run for 5 days.

cheers,
Colin

Release Notes - HTrace - Version 4.1
** Bug
* [HTRACE-114] - Fix compilation error of htrace-hbase against hbase-1.0.0
* [HTRACE-238] - Change maven compiler source level to 1.7 to
match targetJdk
* [HTRACE-243] - Remove duplicate maven-assembly-plugin
configuration section in htrace-htraced/pom.xml
* [HTRACE-245] - NOTICE.txt: change "developed by The Apache
Software...” to "developed at The Apache Software...”
* [HTRACE-246] - HTrace WebApp not properly defined and therefore
not packaged into .war
* [HTRACE-248] - HTraced should gracefully shutdown if stopped
* [HTRACE-249] - Script and doc on how to publish website
* [HTRACE-251] - Fix "mvn clean" target
* [HTRACE-253] - Tracer loadSamplers and loadSpanReceivers logs
are too chatty
* [HTRACE-256] - Change the artifactId for htrace-core in branch
4.0 to be htrace-core4
* [HTRACE-257] - htrace-htraced: add web symlink rather than
generating programmatically
* [HTRACE-262] - Temporarily suppress doclint for Java 8 to
prevent build failure
* [HTRACE-266] - Make the CLIENT_REST_MAX_SPANS_AT_A_TIME_KEY
config key more consistent with other configs
* [HTRACE-267] - Move owl logo licensing information from NOTICE to LICENSE
* [HTRACE-268] - Remove Units and go-codec from LICENSE since they
are not contained in the source release
* [HTRACE-272] - TracerPool must not load multiple inscance of
same receiver class when a simple classname is given
* [HTRACE-279] - Fix issues where the HTracedSpanReceiver was
using the wrong JSON serialization for spans and add validation to
htraced REST ingest path
* [HTRACE-280] - htraced: add metrics about total spans added and
dropped per address
* [HTRACE-281] - htraced: add example/htraced-conf.xml
* [HTRACE-282] - htraced: reap spans which are older than a
configurable interval
* [HTRACE-283] - Heartbeater should wait for goroutine to finish on close
* [HTRACE-284] - htrace-htraced, htrace-flume: do not treat the
shaded version of commons-logging as provided
* [HTRACE-285] - htraced tool: fix query parsing and add query_test
* [HTRACE-289] - Fix TraceEnabled, etc. logger methods for
conditional logging
* [HTRACE-294] - htraced: fix some metrics issues
* [HTRACE-297] - htraced: avoid serializing spans to json unless
TRACE logging is enabled
* [HTRACE-300] - Reaper should be initialized before shards are activated
* [HTRACE-301] - htraced: fix unit tests that aren't waiting for
spans to be written, use semaphore for WrittenSpans
* [HTRACE-302] - htraced: Add admissions control to HRPC to limit
the number of incoming messages
* [HTRACE-304] - htraced: fix bug with GREATER_THAN queries
* [HTRACE-307] - htraced: queries sometimes return no results even
when many results exist due to confusion in iterator usage
* [HTRACE-311] - htraced: Fix logging to stdout via -Dlog.path=
* [HTRACE-316] - htrace-web: span.js issue: span ID string length
is 32, not 36
* [HTRACE-317] - Fix the documentation for adding tracing to an
application to reflect HTrace 4.x API changes
* [HTRACE-328] - htraced continues scanning in some cases even
when no more results are possible

** Improvement
* [HTRACE-129] - htraced: add /server/stats REST endpoint
* [HTRACE-156] - HTrace GUI: add about view
* [HTRACE-181] - gui: Split "about" screen
* [HTRACE-237] - Optimize htraced span receiver
* [HTRACE-239] - Add htrace/impl/TestZipkinSpanReceiver.java
* [HTRACE-260] - htrace-zipkin should not set the obsolete
duration field in thrift
* [HTRACE-271] - Add log4j.properties to all submodule tests
* [HTRACE-276] - Shade classes into org.apache.htrace.shaded
rather than org.apache.htrace
* [HTRACE-286] - htraced: improvements to logging, daemon startup,
and configuration
* [HTRACE-290] - htraced: Fix per-faculty log level settings and
add unit tests for conditional logging
* [HTRACE-291] - rename bin/htrace to bin/htracedTool
* [HTRACE-292] - "htracedTool version" should display the git
hash, and -Dgit.version option should be available for build
* [HTRACE-295] - htraced: setting span.expiry.ms to 0 should
disable span expiry
* [HTRACE-296] - htraced tests: make sure local settings for
HTRACED_WEB_DIR and HTRACE_CONF_DIR don't affect unit tests
* [HTRACE-298] - htraced: improve datastore serialization and metrics
* [HTRACE-303] - Add 

Re: state of the htrace

[Lewis John Mcgibbney]

Hi Colin,
Really cool. I was in Pasadena and sadly missed you and your presentation.
Maybe next time, please post here if you are presenting on HTrace again.
I wonder if anyone is planning on taking HTrace talks to ApacheCon?
Thanks

On Tue, Feb 2, 2016 at 4:51 PM,  wrote:

>
>
> Subject: state of the htrace
> Hi all,
>
> Happy (belated) New Year.  I've been a bit quiet since I went on
> vacation for a few weeks recently, and since I have some other
> projects going on.  But today I wanted to highlight some really cool
> stuff that's been happening in the HTrace and broader tracing
> community recently.
>
> Last weekend, I gave an "intro to htrace" talk this weekend at the
> Scale 14x Linux conference.  See
>
> https://www.socallinuxexpo.org/scale/14x/presentations/introducing-apache-htrace
> .  Also there is a video here:
> https://www.youtube.com/watch?v=t-TwCLwYIGE  It starts at 6:50, since
> the first few minutes of the recording are just me setting up the VGA
> connection and microphone :P  I thought this one went really well
> (especially the demo), and hopefully will get the word out to even
> more people.
>
> In HDFS, we've been exploring adding annotations to certain trace
> spans to collect even more useful information.  For example, Zhe Zhang
> posted a patch to HDFS-9576 to add "position" and "length" annotations
> to DFSInputStream#byteArrayRead, etc. spans.  This should give us data
> on things like the average and median read length in a set of HDFS
> requests.  Similarly, I posted HDFS-9674 which adds the "maximum write
> latency" to OpWriteBlock spans generated by the DataNode.  This is
> very handy when analyzing HDFS write pipelines.  I think we will see
> more of these really helpful annotations, and they will expand our
> ideas about what HTrace can do.
>
> A few weeks back, I wrote a blog post for my employer, Cloudera.  It
> talks about setting up HTrace and htraced on a CDH5.5 cluster.  See
>
> http://blog.cloudera.com/blog/2015/12/new-in-cloudera-labs-apache-htrace-incubating/
>  Hopefully HTrace can "bridge the chasm" between being a developer
> tool, and being a trusted ops tool.  We have some ways to go, but
> having these precompiled packages for CDH5.5 is a big step forward.
> (OK, I'll shut up about vendor stuff now...)
>
> Another really cool thing is that Sean Busbey and others in the YCSB
> community are working on integrating HTrace.  YCSB is a very popular
> benchmark for big data / Hadoopy workloads.  The github issue is here:
> https://github.com/brianfrankcooper/YCSB/issues/415
>
> cheers,
> Colin
>
>

Re: HTrace 4.1 release candidate 1

[Colin P. McCabe]

Thanks for looking at this, Lewis.

On Tue, Feb 2, 2016 at 6:25 PM, Lewis John Mcgibbney
 wrote:
> Hi Colin,
>
> Signatures Good
> Aggregated results of running DRAT over the release candidate
>
> Notes Binaries Archives Standards Apache Generated Unknown
> 0 0 0 142 118 0 15
> Unapproved licenses include
>
>
> /usr/local/drat/deploy/data/jobs/rat/1454465689433/input/bootstrap-theme.css
>
> /usr/local/drat/deploy/data/jobs/rat/1454465689433/input/bootstrap-theme.min.css
>   /usr/local/drat/deploy/data/jobs/rat/1454465689433/input/bootstrap.css
>   /usr/local/drat/deploy/data/jobs/rat/1454465689433/input/bootstrap.min.css
>   /usr/local/drat/deploy/data/jobs/rat/1454465688911/input/backbone-1.1.2.js
>   /usr/local/drat/deploy/data/jobs/rat/1454465688911/input/bootstrap.js
>   /usr/local/drat/deploy/data/jobs/rat/1454465688911/input/bootstrap.min.js
>   /usr/local/drat/deploy/data/jobs/rat/1454465688911/input/d3.min.js
>   /usr/local/drat/deploy/data/jobs/rat/1454465688911/input/jquery-2.1.4.js
>   /usr/local/drat/deploy/data/jobs/rat/1454465688911/input/moment-2.10.3.js
>   /usr/local/drat/deploy/data/jobs/rat/1454465688911/input/npm.js
>
> /usr/local/drat/deploy/data/jobs/rat/1454465688911/input/underscore-1.7.0.js
>   /usr/local/drat/deploy/data/jobs/rat/1454465688785/input/SpanProtos.java
>
> /usr/local/drat/deploy/data/jobs/rat/1454465689194/input/dependency-reduced-pom.xml
>
> /usr/local/drat/deploy/data/jobs/rat/1454465689194/input/dependency-reduced-pom.xml_02022016_1814
>
> I understand that the .css and .js files above are covered in LICENSE at
> the bottom however we need to address the following files
>
>   /usr/local/drat/deploy/data/jobs/rat/1454465688785/input/SpanProtos.java
>
> /usr/local/drat/deploy/data/jobs/rat/1454465689194/input/dependency-reduced-pom.xml
>
> /usr/local/drat/deploy/data/jobs/rat/1454465689194/input/dependency-reduced-pom.xml_02022016_1814

Hmm.  I think we talked about SpanProtos.java,
dependency-reduced-pom.xml, etc. during the previous release and
concluded that they are generated files, and hence exempt from the
license requirement according to
http://incubator.apache.org/guides/releasemanagement.html#notes-license-headers

>
> NOTICE includes
> Copyright 2015 The Apache Software Foundation
> This should be
> Copyright 2016 The Apache Software Foundation
>

OK

> There seems to be a bit on confusion between instructions for Building the
> code. We have the note in README.md and then a separate note within
> BUILDING.txt. We should probably resolve this and include them both in
> README.md
>

OK, I created HTRACE-342 to fix this.

> Build and tests pass fine.
>
> Typically the absence of the license header in the above files would be a
> -1 from me. I will wait to see how others review the candidate before
> VOTE'ing.
> Good job putting this together.

Thanks

best,
Colin

>
>
> On Tue, Feb 2, 2016 at 4:51 PM, > wrote:
>
>>
>> Hi all,
>>
>> I've posted the first release candidate for HTrace 4.1 here:
>>
>> http://people.apache.org/~cmccabe/htrace/releases/4.1.0/rc1/
>>
>> The jars have been staged here:
>>
>> https://repository.apache.org/content/repositories/orgapachehtrace-1021
>>
>> HTrace 4.1 brings a lot of robustness improvements.  There were major
>> improvements to htraced and the web UI, as well as new metrics added.
>> There were numerous build fixups, and we added Docker support, to
>> ensure a repeatable build.
>>
>> Check it out.  The vote will run for 5 days.
>>
>> cheers,
>> Colin
>>
>> Release Notes - HTrace - Version 4.1
>> ** Bug
>> * [HTRACE-114] - Fix compilation error of htrace-hbase against
>> hbase-1.0.0
>> * [HTRACE-238] - Change maven compiler source level to 1.7 to
>> match targetJdk
>> * [HTRACE-243] - Remove duplicate maven-assembly-plugin
>> configuration section in htrace-htraced/pom.xml
>> * [HTRACE-245] - NOTICE.txt: change "developed by The Apache
>> Software...” to "developed at The Apache Software...”
>> * [HTRACE-246] - HTrace WebApp not properly defined and therefore
>> not packaged into .war
>> * [HTRACE-248] - HTraced should gracefully shutdown if stopped
>> * [HTRACE-249] - Script and doc on how to publish website
>> * [HTRACE-251] - Fix "mvn clean" target
>> * [HTRACE-253] - Tracer loadSamplers and loadSpanReceivers logs
>> are too chatty
>> * [HTRACE-256] - Change the artifactId for htrace-core in branch
>> 4.0 to be htrace-core4
>> * [HTRACE-257] - htrace-htraced: add web symlink rather than
>> generating programmatically
>> * [HTRACE-262] - Temporarily suppress doclint for Java 8 to
>> prevent build failure
>> * [HTRACE-266] - Make the CLIENT_REST_MAX_SPANS_AT_A_TIME_KEY
>> config key more consistent with other configs
>> * [HTRACE-267] - Move owl logo licensing information from NOTICE to
>> LICENSE
>> * [HTRACE-268] - Remove Units and go-codec from LICENSE since they
>> are not contained in