Repository: incubator-htrace Updated Branches: refs/heads/master 45e69c888 -> 51dbeb835
HTRACE-350. Add a developer guide (Colin Patrick McCabe via iwasakims) Project: http://git-wip-us.apache.org/repos/asf/incubator-htrace/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-htrace/commit/51dbeb83 Tree: http://git-wip-us.apache.org/repos/asf/incubator-htrace/tree/51dbeb83 Diff: http://git-wip-us.apache.org/repos/asf/incubator-htrace/diff/51dbeb83 Branch: refs/heads/master Commit: 51dbeb8355b856cf27cf5d42d13ee75a7c53c2c5 Parents: 45e69c8 Author: Masatake Iwasaki <[email protected]> Authored: Wed Apr 6 01:42:44 2016 +0900 Committer: Masatake Iwasaki <[email protected]> Committed: Wed Apr 6 01:42:44 2016 +0900 ---------------------------------------------------------------------- README.md | 13 +- src/main/site/markdown/developer_guide.md | 396 +++++++++++++++++++++++++ src/main/site/markdown/index.md | 141 --------- src/main/site/site.xml | 2 +- 4 files changed, 406 insertions(+), 146 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-htrace/blob/51dbeb83/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 39a2331..7b11dab 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,13 @@ HTrace ====== -HTrace is a tracing framework for use with distributed systems. See the -documentation at [src/main/site/markdown/index.md] or at -http://htrace.incubator.apache.org for more details. +[HTrace](http://htrace.incubator.apache.org) is a tracing framework for use +with distributed systems. -For help on building the software, see [BUILDING.txt]. +For help on building the software, see [BUILDING.txt](./BUILDING.txt). + +For details about integrating HTrace into an application, see the [developer +guide](./src/main/site/markdown/developer_guide.md) + +See the [release guide](./src/main/site/markdown/building.md) for +information about making an HTrace release. http://git-wip-us.apache.org/repos/asf/incubator-htrace/blob/51dbeb83/src/main/site/markdown/developer_guide.md ---------------------------------------------------------------------- diff --git a/src/main/site/markdown/developer_guide.md b/src/main/site/markdown/developer_guide.md new file mode 100644 index 0000000..20b832b --- /dev/null +++ b/src/main/site/markdown/developer_guide.md @@ -0,0 +1,396 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# HTrace Developer Guide + +## Introduction +Apache HTrace is an open source framework for distributed tracing. It can be +used with both standalone applications and libraries. + +By adding HTrace support to your project, you will allow end-users to trace +their requests. In addition, any other project that uses HTrace can follow the +requests it makes to your project. That`s why we say HTrace is "end-to-end." + +HTrace was designed for use in big distributed systems such as the Apache +Hadoop Distributed Filesystem and the Apache HBase storage engine. However, +there is nothing Hadoop-specific about HTrace. It has no dependencies on +Hadoop, and is a useful building block for many distributed systems. + +## The HTrace Core Library +In order to use HTrace, your application must link against the appropriate core +library. HTrace`s core libraries have been carefully designed to minimize the +number of dependencies that each one pulls in. HTrace currently has Java, C, +and C++ support. + +HTrace guarantees that the API of core libraries will not change in an +incompatible way during a minor release. So if your application worked with +HTrace 4.1, it should continue working with HTrace 4.2 with no code changes. +(However HTrace 5 may change things, since it is a major release.) + +### Java +The Java library for HTrace is named htrace-core4.jar. This jar must appear on +your CLASSPATH in order to use tracing in Java. If you are using Maven, add +the following to your dependencyManagement section: + + <dependencyManagement> + <dependencies> + <dependency> + <groupId>org.apache.htrace</groupId> + <artifactId>htrace-core4</artifactId> + <version>4.1.0-incubating</version> + </dependency> + ... + </dependencies> + ... + </dependencyManagement> + +If you are using an alternate build system, use the appropriate configuration +for your build system. Note that it is not a good idea to shade htrace-core4, +because some parts of the code use reflection to load classes by name. + +### C +The C library for HTrace is named libhtrace.so. The interface for libhtrace.so +is described in [htrace.h](https://github.com/apache/incubator-htrace/blob/rel/4.1/htrace-c/src/core/htrace.h) + +As with all dynamically loaded native libraries, your application or library +must be able to locate libhtrace.so in order to use it. There are many ways to +accomplish this. The easiest way is to put libhtrace.so in one of the system +shared library paths. You can also use RPATH or LD\_LIBRARY\_PATH to alter the +search paths which the operating system uses. + +### C++ +The C++ API for HTrace is a wrapper around the C API. This approach makes it +easy to use HTrace with any dialect of C++ without recompiling the core +library. It also makes it easier for us to avoid making incompatible ABI +changes. + +The interface is described in +[htrace.hpp](https://github.com/apache/incubator-htrace/blob/rel/4.1/htrace-c/src/core/htrace.hpp) +the same as using the C API, except that you use htrace.hpp instead of +htrace.h. + +## Core Concepts +HTrace is based on a few core concepts. + +### Spans +[Spans](https://github.com/apache/incubator-htrace/blob/rel/4.1/htrace-core4/src/main/java/org/apache/htrace/core/Span.java) +in HTrace are lengths of time. A span has a beginning time in milliseconds, an +end time, a description, and many other fields besides. + +Spans have parents and children. The parent-child relationship between spans +is a little bit like a stack trace. For example, the span graph of an HDFS +"ls" request might look like this: + + ls + +--- FileSystem#createFileSystem + +--- Globber#glob + | +---- GetFileInfo + | +---- ClientNamenodeProtocol#GetFileInfo + | +---- ClientProtocol#GetFileInfo + +--- listPaths + +---- ClientNamenodeProtocol#getListing + +---- ClientProtocol#getListing + +"ls" has several children, "FileSystem#createFileSystem", "Globber#glob", and +"listPaths". Those spans, in turn, have their own children. + +Unlike in a traditional stack trace, the spans in HTrace may be in different +processes or even on different computers. For example, +ClientProtocol#getListing is done on the NameNode, whereas +ClientNamenodeProtocol#getListing happens inside the HDFS client. These are +usually on different computers. + +Each span has a unique 128-bit ID. Because the space of 128-bit numbers is so +large, HTrace can use random generation to avoid collisions. + +### Scopes +[TraceScope](https://github.com/apache/incubator-htrace/blob/rel/4.1/htrace-core4/src/main/java/org/apache/htrace/core/TraceScope.java) +objects manage the lifespan of Span objects. When a TraceScope is created, it +often comes with an associated Span object. When this scope is closed, the +Span will be closed as well. "Closing" the scope means that the span is sent +to a SpanReceiver for processing. We will talk more about what that means +later. For now, just think of closing a TraceScope as similar to closing a +file descriptor-- the natural thing to do when the TraceScope is done. + +HTrace tracks whether a trace scope is active in the current thread by using +thread-local data. This approach makes it easier to add HTrace to existing +code, by avoiding the need to pass around context objects. + +TraceScopes lend themselves to the try... finally pattern of management: + + TraceScope computationScope = tracer.newScope("CalculateFoo"); + try { + calculateFoo(); + } finally { + computationScope.close(); + } + +Any trace spans created inside calculateFoo will automatically have the +CalculateFoo trace span we have created here as their parents. We don`t have +to do any additional work to set up the parent/child relationship because the +thread-local data takes care of it. + +In Java7, the +[try-with-resources](https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html) +idiom may be used to accomplish the same thing without a finally block: + + try (TraceScope computationScope = tracer.newScope("CalculateFoo")) { + calculateFoo(); + } + +The important thing to remember is to close the scope when you are done with it. + +Note that in the C++ API, the destructor of the htrace::Scope object will +automatically close the span. + + htrace::Scope(tracer_, "CalculateFoo"); + calculateFoo(); + +TraceScope are associatewith particular threads. If you want to pass a trace +scope to another thread, you must detach it from the current one first. We +will talk more about that later in this guide. + +### Tracers +[Tracers](https://github.com/apache/incubator-htrace/blob/rel/4.1/htrace-core4/src/main/java/org/apache/htrace/core/Tracer.java) +are the API for creating trace scope objects. You can see that in the example +above, we called the Tracer#newScope function to create a scope. + +It is difficult to trace every operation. The volume of trace span data that +would be generated would be extremely large! So we rely on sampling a subset +of all possible traces. Tracer objects contain Samplers. When you call +Tracer#newScope, the Tracer will consult that Sampler to determine if a new +span should be created, or if an empty scope which contains no span should be +returned. Note that if there is already a currently active span, the Tracer +will always create a child span, regardless of what the sampler says. This is +because we want to see the complete graph of every operation, not just "bits +and pieces." Tracer objects also manage the SpanReceiver objects which control +where spans are sent. + +A single process or library can have many Tracer objects. Each Tracer object +has its own configuration. One way of thinking of Tracer objects is that they +are similar to Log objects in log4j. Just as you might create a Log object for +the NameNode and one for the DataNode, we create a Tracer for the NameNode and +another Tracer for the DataNode. This allows users to control the sampling +rate for the DataNode and the NameNode separately. + +Unlike TraceScope and Span, Tracer objects are thread-safe. It is perfectly +acceptable (and even recommended) to have multiple threads calling +Tracer#newScope at once on the same Tracer object. + +The number of Tracer objects you should create in your project depends on the +structure of your project. Many applications end up creating a small number of +global Tracer objects. Libraries usually should not use globals, but associate +the Tracer with the current library context. + +### Wrappers +HTrace contains many helpful wrapper objects like +[TraceRunnable](https://github.com/apache/incubator-htrace/blob/rel/4.1/htrace-core4/src/main/java/org/apache/htrace/core/TraceRunnable.java) +[TraceCallable](https://github.com/apache/incubator-htrace/blob/rel/4.1/htrace-core4/src/main/java/org/apache/htrace/core/TraceCallable.java) +and +[TraceExecutorService](https://github.com/apache/incubator-htrace/blob/rel/4.1/htrace-core4/src/main/java/org/apache/htrace/core/TraceExecutorService.java) +These helper classes make it easier for you to create trace spans. Basically, +they act as wrappers around Tracer#newScope. + +## SpanReceivers +HTrace is a pluggable framework. We can configure where trace spans are sent +at runtime, by selecting the appropriate SpanReceiver. + + FoobarApplication + | + V + htrace-core4 + | + V + HTracedSpanReceiver + OR + LocalFileSpanReceiver + OR + StandardOutSpanReceiver + OR + ZipkinSpanReceiver + OR + ... + +As a developer integrating HTrace into your application, you do not need to +know what each and every SpanReceiver does-- only the ones you actually intend +to use. The nice thing is that users can use any span receiver with your +project, without any additional effort on your part. The span receivers are +decoupled from the core library. + +When using Java, you will need to add the jar file for whichever span receiver +you want to use to your CLASSPATH. (These span receivers are not added to the +CLASSPATH by default because they may have additional dependencies that not +every user wants.) For C and C++, a more limited set of span receivers is +available, but they are all integrated into libhtrace.so, so no additional +libraries are needed. + +## Configuration +Clearly, HTrace requires configuration. We need to control which SpanReceiver +is used, what the sampling rate is, and many other things besides. Luckily, as +we discussed earlier, the Tracer objects maintain this configuration +information for us. When we ask for a new trace scope, the Tracer knows what +configuration to use. + +This configuration comes from the HTraceConfiguration object that we supplied +to the Tracer#Builder earlier. In general, we want to configure HTrace the +same way we configure anything else in our distributed system. So we normally +create a subclass of HTraceConfiguration that accesses the appropriate +information in our existing configuration system. + +To make this a little more concrete, let`s suppose we are writing Bob`s +Distributed System. Being a pragmatic (not to mention lazy) guy, Bob has +decided to just use Java configuration properties for configuration. +So our Tracer#Builder invoation would look something like this: + + this.tracer = new Tracer.Builder("Bob"). + conf(new HTraceConfiguration() { + @Override + public String get(String key) { + return System.getProperty("htrace." + key); + } + + @Override + public String get(String key, String defaultValue) { + String ret = get(key); + return (ret != null) ? ret : defaultValue; + } + }). + build(); + +You can see that this configuration object maps every property starting in +"htrace." to an htrace property. So, for example, you would set the Java +system property value "htrace.span.receiver.classes" in order to control the +HTrace configuration key "span.receiver.classes". + +Of course, Bob probably should have been less lazy, and used a real +configuration system instead of using Java system properties. This is just a +toy example to illustrate how to integrate with an existing configuration +system. + +Bob might also have wanted to use different prefixes for different Tracer +objects. For example, in Hadoop you can configure the FsShell Tracer +separately from the NameNode Tracer, by setting +"fs.shell.htrace.span.receiver.classes". This is easy to control by changing +the HTraceConfiguration object that you pass to Tracer#Builder. + +Note that in C and C++, this system is a little different, based on explicitly +creating a configuration object prior to creating a tracer, rather than using a +callback-based system. + +## TracerPool +SpanReceiver objects often need to make a network connection to a remote +serveice or daemon. Usually, we don`t want to create more than one +SpanReceiver of each type in a particular process, so that we can optimize the +number of these connections that we have open. TracerPool objects allow us to +acheieve this. + +Each Tracer object belongs to a TracerPool. When a call to Tracer#Builder is +made which requests a specific SpanReceiver, we check the TracerPool to see if +there is already an instance of that SpanReceiver. If so, we simply re-use the +existing one rather than creating a new one. + +Normally, you don`t need to worry about TracerPools. However, if you have an +explicit need to create multiple SpanReceivers of the same type, you can do it +by using a TracerPool other than the default one, or by explicitly adding the +SpanReceiver to your Tracer once it has been created. This is not a very +common need. + +When the application terminates, we will attempt to close all currently open +SpanReceivers. You can attempt to close the SpanReceivers earlier than that by +calling tracer.getTracerPool().removeAndCloseAllSpanReceivers(). + +## Passing Span IDs over RPC +So far, we have described how to use HTrace inside a single process. However, +since we are dealing with distributed systems, a single process is not enough. +We need a way to send HTrace information across the network. + +Unlike some other tracing systems, HTrace works with many different RPC +systems. You do not need to change the RPC framework you are using in order to +use HTrace. You simply need to find a way to pass HTrace information using the +RPC framework that you`re already using. In most cases, what this boils down +to is figuring out a way to send the 128-bit parent ID of an operation over the +network as an optional field. + +Let`s say that Bob is writing the server side of his system. If the client +sent its parent ID over the wire, Bob might write code like this: + + BobRequestProto bp = ...; + SpanId spanId = (bp.hasSpanId()) ? bp.getSpanId() : SpanId.INVALID; + try (TraceScope scope = tracer.newScope("bobRequest", spanId)) { + doBobRequest(bp); + } + +By passing the spanId to Tracer#newScope, we ensure that any new span we create +will have a record of its parent. + +## Handling work done in multiple threads +Sometimes, you end up performing work for a single request in multiple threads. +How can we handle this? Certainly, we can use the same approach as we did in +the RPC case above. We can have the child threads create trace scopes which +use our parent ID object. SpanId objects are immtuable and easy to share +between threads. + + try (TraceScope bigScope = tracer.newScope("bigComputation")) { + SpanId bigScopeId = bigScope.getCurrentSpanId(); + Thread t1 = new Thread(new Runnable() { + @Override + public void run() { + TraceScope scope = (bigScopeId.isValid()) ? + tracer.newScope("bigComputationWorker", bigScopeId) : + tracer.newNullScope(); + try { + doWorkerStuff(); + } finally { + scope.close(); + } + } + }, "myWorker"); + t1.start(); + t1.join(); + } + +Note that in this case, the two threads are not sharing trace scopes. Instead, +we are setting up a new trace scope, which may have its own span, which has the +outer scope as a parent. Note that HTrace will be well-behaved even if the +outer scope may be closed before the inner one. The SpanId object of a +TraceScope continues to be valid even after a scope is closed. It`s just a +number, essentially-- and that number will not be reused by any other scope. + +Why do we make the worker Thread call newNullScope in the case where the outer +scope`s span id is invalid? Well, we don`t want to ever create the inner span +if the outer one does not exist. Calling newNullScope ensures that we get a +scope with no span, no matter what samplers are configured. + +What if we don`t want to create more than one span here? In that case, we need +to have some way of detaching the TraceScope from the parent thread, and +re-attaching it to the worker thread. Luckily, HTrace has an API for that. + + final TraceScope bigScope = tracer.newScope("bigComputation"); + bigScope.detach(); + Thread t1 = new Thread(new Runnable() { + @Override + public void run() { + bigScope.reattach(); + try { + doWorkerStuff(); + } finally { + bigScope.close(); + } + } + }, "myWorker"); + t1.start(); + t1.join(); + +Note that in this case, we don`t need to close the TraceScope in the containing +thread. It has already been closed by the worker thread. http://git-wip-us.apache.org/repos/asf/incubator-htrace/blob/51dbeb83/src/main/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/src/main/site/markdown/index.md b/src/main/site/markdown/index.md deleted file mode 100644 index 4db4244..0000000 --- a/src/main/site/markdown/index.md +++ /dev/null @@ -1,141 +0,0 @@ -<!--- - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. ---> - -Apache HTrace is an <a href="http://htrace.incubator.apache.org";>Apache Incubator</a> -project. To add HTrace to your project, see detail on how to add it as a -<a href="dependency-info.html">dependency</a>. Formerly, HTrace was available at `org.htrace`. - -* htrace-4.0.0-incubating published September 15th, 2015. [Download it!](http://www.apache.org/dyn/closer.cgi/incubator/htrace/) -* htrace-3.2.0-incubating published June 2nd, 2015. [Download it!](http://www.apache.org/dyn/closer.cgi/incubator/htrace/) -* We made our first release from Apache Incubator, htrace-3.1.0-incubating, January 20th, 2015. [Download it!](http://www.apache.org/dyn/closer.cgi/incubator/htrace/) - - -## API -Using HTrace requires adding some instrumentation to your application. -Before we get into the details, lets review our terminology. HTrace -borrows [Dapper's](http://research.google.com/pubs/pub36356.html) -terminology. - -<b>Span:</b> The basic unit of work. For example, sending an RPC is a new span, -as is sending a response to an RPC. Spans are identified by a unique 128-bit -ID. Spans also have other data, such as descriptions, key-value annotations, -the ID of the span that caused them, and tracer IDs. - -Spans are started and stopped, and they keep track of their timing -information. Once you create a span, you must stop it at some point -in the future. - -<b>TracerId:</b> Identifies the Tracer object which created a specific Span. -The TracerId should identify the source of the Span. "10.0.0.2/DataNode" is a -typical TracerID. - -<b>SpanReceiver:</b> SpanReceivers handle spans once they have been created. -Typically, Span Receivers send spans to an external data store to be -analyzed. HTrace comes with several standard span receivers, such as -`LocalFileSpanReceiver`. - -<b>Sampler:</b> Samplers determine when tracing should be enabled, and when it -should be disabled. The goal of HTrace is to trace an entire request. - -## How to add tracing to your application -To instrument your system you must: - -### Create a Tracer object -You can create a Tracer object via the `Tracer#Builder`. - - Tracer tracer = new Tracer.Builder("MyApp").conf(conf).build(); - -The `Tracer#Builder` will take care of creating the appropriate Sampler and -SpanReceiver objects, as well as the Tracer itself. If a SpanReceiver was -created, we will install a shutdown hook to close it when the JVM shuts down. - -### Attach additional information to your RPCs -In order to create the causal links necessary for a trace, HTrace needs to know -about the causal relationships between spans. The only information you need to -add to your RPCs is the 128-bit span ID. If tracing is enabled when you send an -RPC, attach the ID of the current span to the message. On the receiving end of -the RPC, check to see if the message has a span ID attached. If it does, start -a new trace scope with that span as a parent. - -### Wrap your thread changes -HTrace stores span information in java's ThreadLocals, which causes -the trace to be "lost" on thread changes. The only way to prevent -this is to "wrap" your thread changes. For example, if your code looks -like this: - - Thread t1 = new Thread(new MyRunnable()); - ... - -Just change it to look this: - - Thread t1 = new Thread(tracer.wrap(new MyRunnable(), "MyRunnable")); - -That's it! `Tracer.wrap()` takes two arguments -(a runnable or a callable and span description) -and if the current thread is a part of a trace, -returns a wrapped version of the runnable/callable. -The wrapped version of a runnable/callable -just knows about the span that created it and will start a new span -with given description in the new thread that is the child of the span that -created the runnable/callable. There may be situations in which a -simple `Tracer.wrap()` does not suffice. In these cases all you need -to do is keep a reference to the "parent span" (the span before the -thread change) and once you're in the new thread start a new span that -is the "child" of the parent span you stored. - -For example: - -Say you have some object representing a "put" operation. When the -client does a "put," the put is first added to a list so another -thread can batch together the puts. In this situation, you -might want to add another field to the Put class that could store the -current span at the time the put was created. Then when the put is -pulled out of the list to be processed, you can start a new span as -the child of the span stored in the Put. - -### Add custom spans and annotations -Once you've augmented your RPC's and wrapped the necessary thread -changes, you can add more spans and annotations wherever you want. -For example, you might do some expensive computation that you want to -see on your traces. In this case, -you could create a new trace scope which starts a new span internally -before the computation that you then close after the computation has -finished. It might look like this: - - TraceScope computationScope = tracer.newScope("Expensive computation."); - try { - //expensive computation here - } finally { - computationScope.close(); - } - -HTrace also supports key-value annotations on a per-trace basis. - -Example: - - scope.addKVAnnotation("faultyRecordCounter", Integer.toString(1)); - -## htrace-zipkin -htrace-zipkin provides the `SpanReceiver` implementation -which sends spans to [Zipkin](https://github.com/twitter/zipkin) collector. -You can build the uber-jar (htrace-zipkin-*-jar-withdependency.jar) for manual -setup as shown below. This uber-jar contains all dependencies except -htrace-core and its dependencies. - - $ cd htrace-zipkin - $ mvn compile assembly:single - -## htrace-hbase -See htrace-hbase for an Span Receiver implementation that writes HBase. - http://git-wip-us.apache.org/repos/asf/incubator-htrace/blob/51dbeb83/src/main/site/site.xml ---------------------------------------------------------------------- diff --git a/src/main/site/site.xml b/src/main/site/site.xml index 544db38..f2b329c 100644 --- a/src/main/site/site.xml +++ b/src/main/site/site.xml @@ -33,7 +33,7 @@ </bannerRight> <body> <menu name="Apache HTrace Project"> - <item name="Overview" href="index.html"/> + <item name="Developer Guide" href="developer_guide.html"/> <item name="License" href="license.html" /> <item name="Downloads" href="http://www.apache.org/dyn/closer.cgi/incubator/htrace/"; /> <item name="Distribution" href="dependency-management.html" />
