Author: buildbot
Date: Mon Jun 1 19:50:47 2015
New Revision: 953472
Log:
Staging update by buildbot for flume
Added:
websites/staging/flume/trunk/content/.doctrees/releases/1.6.0.doctree
(with props)
websites/staging/flume/trunk/content/_sources/releases/1.6.0.txt
websites/staging/flume/trunk/content/releases/1.6.0.html
Modified:
websites/staging/flume/trunk/content/ (props changed)
websites/staging/flume/trunk/content/.doctrees/FlumeDeveloperGuide.doctree
websites/staging/flume/trunk/content/.doctrees/FlumeUserGuide.doctree
websites/staging/flume/trunk/content/.doctrees/download.doctree
websites/staging/flume/trunk/content/.doctrees/environment.pickle
websites/staging/flume/trunk/content/.doctrees/index.doctree
websites/staging/flume/trunk/content/.doctrees/releases/index.doctree
websites/staging/flume/trunk/content/FlumeDeveloperGuide.html
websites/staging/flume/trunk/content/FlumeUserGuide.html
websites/staging/flume/trunk/content/_sources/FlumeDeveloperGuide.txt
websites/staging/flume/trunk/content/_sources/FlumeUserGuide.txt
websites/staging/flume/trunk/content/_sources/download.txt
websites/staging/flume/trunk/content/_sources/index.txt
websites/staging/flume/trunk/content/_sources/releases/index.txt
websites/staging/flume/trunk/content/documentation.html
websites/staging/flume/trunk/content/download.html
websites/staging/flume/trunk/content/index.html
websites/staging/flume/trunk/content/project-reports.html
websites/staging/flume/trunk/content/releases/1.5.2.html
websites/staging/flume/trunk/content/releases/index.html
websites/staging/flume/trunk/content/searchindex.js
Propchange: websites/staging/flume/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Mon Jun 1 19:50:47 2015
@@ -1 +1 @@
-1657863
+1682982
Modified:
websites/staging/flume/trunk/content/.doctrees/FlumeDeveloperGuide.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/FlumeUserGuide.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/download.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/environment.pickle
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/index.doctree
==============================================================================
Binary files - no diff available.
Added: websites/staging/flume/trunk/content/.doctrees/releases/1.6.0.doctree
==============================================================================
Binary file - no diff available.
Propchange:
websites/staging/flume/trunk/content/.doctrees/releases/1.6.0.doctree
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: websites/staging/flume/trunk/content/.doctrees/releases/index.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/FlumeDeveloperGuide.html
==============================================================================
--- websites/staging/flume/trunk/content/FlumeDeveloperGuide.html (original)
+++ websites/staging/flume/trunk/content/FlumeDeveloperGuide.html Mon Jun 1
19:50:47 2015
@@ -7,7 +7,7 @@
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
- <title>Flume 1.5.2 Developer Guide — Apache Flume</title>
+ <title>Flume 1.6.0 Developer Guide — Apache Flume</title>
<link rel="stylesheet" href="_static/flume.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
@@ -27,7 +27,7 @@
<link rel="top" title="Apache Flume" href="index.html" />
<link rel="up" title="Documentation" href="documentation.html" />
<link rel="next" title="Releases" href="releases/index.html" />
- <link rel="prev" title="Flume 1.5.2 User Guide" href="FlumeUserGuide.html"
/>
+ <link rel="prev" title="Flume 1.6.0 User Guide" href="FlumeUserGuide.html"
/>
</head>
<body>
<div class="header">
@@ -59,8 +59,8 @@
<div class="bodywrapper">
<div class="body">
- <div class="section" id="flume-1-5-2-developer-guide">
-<h1>Flume 1.5.2 Developer Guide<a class="headerlink"
href="#flume-1-5-2-developer-guide" title="Permalink to this
headline">¶</a></h1>
+ <div class="section" id="flume-1-6-0-developer-guide">
+<h1>Flume 1.6.0 Developer Guide<a class="headerlink"
href="#flume-1-6-0-developer-guide" title="Permalink to this
headline">¶</a></h1>
<div class="section" id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to
this headline">¶</a></h2>
<div class="section" id="overview">
@@ -293,6 +293,112 @@ properties:</p>
</pre></div>
</div>
</div>
+<div class="section" id="secure-rpc-client-thrift">
+<h5>Secure RPC client - Thrift<a class="headerlink"
href="#secure-rpc-client-thrift" title="Permalink to this headline">¶</a></h5>
+<p>As of Flume 1.6.0, Thrift source and sink supports kerberos based
authentication.
+The client needs to use the getThriftInstance method of <tt class="docutils
literal"><span class="pre">SecureRpcClientFactory</span></tt>
+to get hold of a <tt class="docutils literal"><span
class="pre">SecureThriftRpcClient</span></tt>. <tt class="docutils
literal"><span class="pre">SecureThriftRpcClient</span></tt> extends
+<tt class="docutils literal"><span class="pre">ThriftRpcClient</span></tt>
which implements the <tt class="docutils literal"><span
class="pre">RpcClient</span></tt> interface. The kerberos
+authentication module resides in flume-ng-auth module which is
+required in classpath, when using the <tt class="docutils literal"><span
class="pre">SecureRpcClientFactory</span></tt>. Both the client
+principal and the client keytab should be passed in as parameters through the
+properties and they reflect the credentials of the client to authenticate
+against the kerberos KDC. In addition, the server principal of the destination
+Thrift source to which this client is connecting to, should also be provided.
+The following example shows how to use the <tt class="docutils literal"><span
class="pre">SecureRpcClientFactory</span></tt>
+within a user’s data-generating application:</p>
+<div class="highlight-java"><div class="highlight"><pre><span
class="kn">import</span> <span class="nn">org.apache.flume.Event</span><span
class="o">;</span>
+<span class="kn">import</span> <span
class="nn">org.apache.flume.EventDeliveryException</span><span
class="o">;</span>
+<span class="kn">import</span> <span
class="nn">org.apache.flume.event.EventBuilder</span><span class="o">;</span>
+<span class="kn">import</span> <span
class="nn">org.apache.flume.api.SecureRpcClientFactory</span><span
class="o">;</span>
+<span class="kn">import</span> <span
class="nn">org.apache.flume.api.RpcClientConfigurationConstants</span><span
class="o">;</span>
+<span class="kn">import</span> <span
class="nn">org.apache.flume.api.RpcClient</span><span class="o">;</span>
+<span class="kn">import</span> <span
class="nn">java.nio.charset.Charset</span><span class="o">;</span>
+<span class="kn">import</span> <span
class="nn">java.util.Properties</span><span class="o">;</span>
+
+<span class="kd">public</span> <span class="kd">class</span> <span
class="nc">MyApp</span> <span class="o">{</span>
+ <span class="kd">public</span> <span class="kd">static</span> <span
class="kt">void</span> <span class="nf">main</span><span
class="o">(</span><span class="n">String</span><span class="o">[]</span> <span
class="n">args</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">MySecureRpcClientFacade</span> <span
class="n">client</span> <span class="o">=</span> <span class="k">new</span>
<span class="n">MySecureRpcClientFacade</span><span class="o">();</span>
+ <span class="c1">// Initialize client with the remote Flume agent's
host, port</span>
+ <span class="n">Properties</span> <span class="n">props</span> <span
class="o">=</span> <span class="k">new</span> <span
class="n">Properties</span><span class="o">();</span>
+ <span class="n">props</span><span class="o">.</span><span
class="na">setProperty</span><span class="o">(</span><span
class="n">RpcClientConfigurationConstants</span><span class="o">.</span><span
class="na">CONFIG_CLIENT_TYPE</span><span class="o">,</span> <span
class="s">"thrift"</span><span class="o">);</span>
+ <span class="n">props</span><span class="o">.</span><span
class="na">setProperty</span><span class="o">(</span><span
class="s">"hosts"</span><span class="o">,</span> <span
class="s">"h1"</span><span class="o">);</span>
+ <span class="n">props</span><span class="o">.</span><span
class="na">setProperty</span><span class="o">(</span><span
class="s">"hosts.h1"</span><span class="o">,</span> <span
class="s">"client.example.org"</span><span class="o">+</span><span
class="s">":"</span><span class="o">+</span> <span
class="n">String</span><span class="o">.</span><span
class="na">valueOf</span><span class="o">(</span><span
class="mi">41414</span><span class="o">));</span>
+
+ <span class="c1">// Initialize client with the kerberos authentication
related properties</span>
+ <span class="n">props</span><span class="o">.</span><span
class="na">setProperty</span><span class="o">(</span><span
class="s">"kerberos"</span><span class="o">,</span> <span
class="s">"true"</span><span class="o">);</span>
+ <span class="n">props</span><span class="o">.</span><span
class="na">setProperty</span><span class="o">(</span><span
class="s">"client-principal"</span><span class="o">,</span> <span
class="s">"flumeclient/[email protected]"</span><span
class="o">);</span>
+ <span class="n">props</span><span class="o">.</span><span
class="na">setProperty</span><span class="o">(</span><span
class="s">"client-keytab"</span><span class="o">,</span> <span
class="s">"/tmp/flumeclient.keytab"</span><span class="o">);</span>
+ <span class="n">props</span><span class="o">.</span><span
class="na">setProperty</span><span class="o">(</span><span
class="s">"server-principal"</span><span class="o">,</span> <span
class="s">"flume/[email protected]"</span><span
class="o">);</span>
+ <span class="n">client</span><span class="o">.</span><span
class="na">init</span><span class="o">(</span><span class="n">props</span><span
class="o">);</span>
+
+ <span class="c1">// Send 10 events to the remote Flume agent. That agent
should be</span>
+ <span class="c1">// configured to listen with an AvroSource.</span>
+ <span class="n">String</span> <span class="n">sampleData</span> <span
class="o">=</span> <span class="s">"Hello Flume!"</span><span
class="o">;</span>
+ <span class="k">for</span> <span class="o">(</span><span
class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span
class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span
class="o"><</span> <span class="mi">10</span><span class="o">;</span> <span
class="n">i</span><span class="o">++)</span> <span class="o">{</span>
+ <span class="n">client</span><span class="o">.</span><span
class="na">sendDataToFlume</span><span class="o">(</span><span
class="n">sampleData</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="n">client</span><span class="o">.</span><span
class="na">cleanUp</span><span class="o">();</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+
+<span class="kd">class</span> <span class="nc">MySecureRpcClientFacade</span>
<span class="o">{</span>
+ <span class="kd">private</span> <span class="n">RpcClient</span> <span
class="n">client</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">Properties</span> <span
class="n">properties</span><span class="o">;</span>
+
+ <span class="kd">public</span> <span class="kt">void</span> <span
class="nf">init</span><span class="o">(</span><span class="n">Properties</span>
<span class="n">properties</span><span class="o">)</span> <span
class="o">{</span>
+ <span class="c1">// Setup the RPC connection</span>
+ <span class="k">this</span><span class="o">.</span><span
class="na">properties</span> <span class="o">=</span> <span
class="n">properties</span><span class="o">;</span>
+ <span class="c1">// Create the ThriftSecureRpcClient instance by using
SecureRpcClientFactory</span>
+ <span class="k">this</span><span class="o">.</span><span
class="na">client</span> <span class="o">=</span> <span
class="n">SecureRpcClientFactory</span><span class="o">.</span><span
class="na">getThriftInstance</span><span class="o">(</span><span
class="n">properties</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="kd">public</span> <span class="kt">void</span> <span
class="nf">sendDataToFlume</span><span class="o">(</span><span
class="n">String</span> <span class="n">data</span><span class="o">)</span>
<span class="o">{</span>
+ <span class="c1">// Create a Flume Event object that encapsulates the
sample data</span>
+ <span class="n">Event</span> <span class="n">event</span> <span
class="o">=</span> <span class="n">EventBuilder</span><span
class="o">.</span><span class="na">withBody</span><span class="o">(</span><span
class="n">data</span><span class="o">,</span> <span
class="n">Charset</span><span class="o">.</span><span
class="na">forName</span><span class="o">(</span><span
class="s">"UTF-8"</span><span class="o">));</span>
+
+ <span class="c1">// Send the event</span>
+ <span class="k">try</span> <span class="o">{</span>
+ <span class="n">client</span><span class="o">.</span><span
class="na">append</span><span class="o">(</span><span
class="n">event</span><span class="o">);</span>
+ <span class="o">}</span> <span class="k">catch</span> <span
class="o">(</span><span class="n">EventDeliveryException</span> <span
class="n">e</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// clean up and recreate the client</span>
+ <span class="n">client</span><span class="o">.</span><span
class="na">close</span><span class="o">();</span>
+ <span class="n">client</span> <span class="o">=</span> <span
class="kc">null</span><span class="o">;</span>
+ <span class="n">client</span> <span class="o">=</span> <span
class="n">SecureRpcClientFactory</span><span class="o">.</span><span
class="na">getThriftInstance</span><span class="o">(</span><span
class="n">properties</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="kd">public</span> <span class="kt">void</span> <span
class="nf">cleanUp</span><span class="o">()</span> <span class="o">{</span>
+ <span class="c1">// Close the RPC connection</span>
+ <span class="n">client</span><span class="o">.</span><span
class="na">close</span><span class="o">();</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+</pre></div>
+</div>
+<p>The remote <tt class="docutils literal"><span
class="pre">ThriftSource</span></tt> should be started in kerberos mode.
+Below is an example Flume agent configuration that’s waiting for a
connection
+from MyApp:</p>
+<div class="highlight-properties"><div class="highlight"><pre><span
class="na">a1.channels</span> <span class="o">=</span> <span class="s">c1</span>
+<span class="na">a1.sources</span> <span class="o">=</span> <span
class="s">r1</span>
+<span class="na">a1.sinks</span> <span class="o">=</span> <span
class="s">k1</span>
+
+<span class="na">a1.channels.c1.type</span> <span class="o">=</span> <span
class="s">memory</span>
+
+<span class="na">a1.sources.r1.channels</span> <span class="o">=</span> <span
class="s">c1</span>
+<span class="na">a1.sources.r1.type</span> <span class="o">=</span> <span
class="s">thrift</span>
+<span class="na">a1.sources.r1.bind</span> <span class="o">=</span> <span
class="s">0.0.0.0</span>
+<span class="na">a1.sources.r1.port</span> <span class="o">=</span> <span
class="s">41414</span>
+<span class="na">a1.sources.r1.kerberos</span> <span class="o">=</span> <span
class="s">true</span>
+<span class="na">a1.sources.r1.agent-principal</span> <span class="o">=</span>
<span class="s">flume/[email protected]</span>
+<span class="na">a1.sources.r1.agent-keytab</span> <span class="o">=</span>
<span class="s">/tmp/flume.keytab</span>
+
+
+<span class="na">a1.sinks.k1.channel</span> <span class="o">=</span> <span
class="s">c1</span>
+<span class="na">a1.sinks.k1.type</span> <span class="o">=</span> <span
class="s">logger</span>
+</pre></div>
+</div>
+</div>
<div class="section" id="failover-client">
<h5>Failover Client<a class="headerlink" href="#failover-client"
title="Permalink to this headline">¶</a></h5>
<p>This class wraps the default Avro RPC client to provide failover handling
@@ -456,16 +562,16 @@ sources, sinks, and channels are allowed
is a special embedded source and events should be send to the source
via the put, putAll methods on the EmbeddedAgent object. Only File Channel
and Memory Channel are allowed as channels while Avro Sink is the only
-supported sink.</p>
+supported sink. Interceptors are also supported by the embedded agent.</p>
<p>Note: The embedded agent has a dependency on hadoop-core.jar.</p>
<p>Configuration of an Embedded Agent is similar to configuration of a
full Agent. The following is an exhaustive list of configration options:</p>
<p>Required properties are in <strong>bold</strong>.</p>
<table border="1" class="docutils">
<colgroup>
-<col width="11%" />
-<col width="9%" />
-<col width="80%" />
+<col width="20%" />
+<col width="15%" />
+<col width="65%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Property Name</th>
@@ -480,11 +586,13 @@ full Agent. The following is an exhausti
</tr>
<tr class="row-odd"><td><strong>channel.type</strong></td>
<td>–</td>
-<td>Either <tt class="docutils literal"><span class="pre">memory</span></tt>
or <tt class="docutils literal"><span class="pre">file</span></tt> which
correspond to MemoryChannel and FileChannel respectively.</td>
+<td>Either <tt class="docutils literal"><span class="pre">memory</span></tt>
or <tt class="docutils literal"><span class="pre">file</span></tt> which
correspond
+to MemoryChannel and FileChannel respectively.</td>
</tr>
<tr class="row-even"><td>channel.*</td>
<td>–</td>
-<td>Configuration options for the channel type requested, see MemoryChannel or
FileChannel user guide for an exhaustive list.</td>
+<td>Configuration options for the channel type requested,
+see MemoryChannel or FileChannel user guide for an exhaustive list.</td>
</tr>
<tr class="row-odd"><td><strong>sinks</strong></td>
<td>–</td>
@@ -492,19 +600,34 @@ full Agent. The following is an exhausti
</tr>
<tr class="row-even"><td><strong>sink.type</strong></td>
<td>–</td>
-<td>Property name must match a name in the list of sinks. Value must be <tt
class="docutils literal"><span class="pre">avro</span></tt></td>
+<td>Property name must match a name in the list of sinks.
+Value must be <tt class="docutils literal"><span
class="pre">avro</span></tt></td>
</tr>
<tr class="row-odd"><td>sink.*</td>
<td>–</td>
-<td>Configuration options for the sink. See AvroSink user guide for an
exhaustive list, however note AvroSink requires at least hostname and port.</td>
+<td>Configuration options for the sink.
+See AvroSink user guide for an exhaustive list,
+however note AvroSink requires at least hostname and port.</td>
</tr>
<tr class="row-even"><td><strong>processor.type</strong></td>
<td>–</td>
-<td>Either <tt class="docutils literal"><span class="pre">failover</span></tt>
or <tt class="docutils literal"><span class="pre">load_balance</span></tt>
which correspond to FailoverSinksProcessor and LoadBalancingSinkProcessor
respectively.</td>
+<td>Either <tt class="docutils literal"><span class="pre">failover</span></tt>
or <tt class="docutils literal"><span class="pre">load_balance</span></tt>
which correspond
+to FailoverSinksProcessor and LoadBalancingSinkProcessor respectively.</td>
</tr>
<tr class="row-odd"><td>processor.*</td>
<td>–</td>
-<td>Configuration options for the sink processor selected. See
FailoverSinksProcessor and LoadBalancingSinkProcessor user guide for an
exhaustive list.</td>
+<td>Configuration options for the sink processor selected.
+See FailoverSinksProcessor and LoadBalancingSinkProcessor
+user guide for an exhaustive list.</td>
+</tr>
+<tr class="row-even"><td>source.interceptors</td>
+<td>–</td>
+<td>Space-separated list of interceptors</td>
+</tr>
+<tr class="row-odd"><td>source.interceptors.*</td>
+<td>–</td>
+<td>Configuration options for individual interceptors
+specified in the source.interceptors property</td>
</tr>
</tbody>
</table>
@@ -520,6 +643,10 @@ full Agent. The following is an exhausti
<span class="n">properties</span><span class="o">.</span><span
class="na">put</span><span class="o">(</span><span
class="s">"sink2.hostname"</span><span class="o">,</span> <span
class="s">"collector2.apache.org"</span><span class="o">);</span>
<span class="n">properties</span><span class="o">.</span><span
class="na">put</span><span class="o">(</span><span
class="s">"sink2.port"</span><span class="o">,</span> <span
class="s">"5565"</span><span class="o">);</span>
<span class="n">properties</span><span class="o">.</span><span
class="na">put</span><span class="o">(</span><span
class="s">"processor.type"</span><span class="o">,</span> <span
class="s">"load_balance"</span><span class="o">);</span>
+<span class="n">properties</span><span class="o">.</span><span
class="na">put</span><span class="o">(</span><span
class="s">"source.interceptors"</span><span class="o">,</span> <span
class="s">"i1"</span><span class="o">);</span>
+<span class="n">properties</span><span class="o">.</span><span
class="na">put</span><span class="o">(</span><span
class="s">"source.interceptors.i1.type"</span><span
class="o">,</span> <span class="s">"static"</span><span
class="o">);</span>
+<span class="n">properties</span><span class="o">.</span><span
class="na">put</span><span class="o">(</span><span
class="s">"source.interceptors.i1.key"</span><span class="o">,</span>
<span class="s">"key1"</span><span class="o">);</span>
+<span class="n">properties</span><span class="o">.</span><span
class="na">put</span><span class="o">(</span><span
class="s">"source.interceptors.i1.value"</span><span
class="o">,</span> <span class="s">"value1"</span><span
class="o">);</span>
<span class="n">EmbeddedAgent</span> <span class="n">agent</span> <span
class="o">=</span> <span class="k">new</span> <span
class="n">EmbeddedAgent</span><span class="o">(</span><span
class="s">"myagent"</span><span class="o">);</span>
@@ -806,7 +933,7 @@ mechanism that captures the new data and
<h3><a href="index.html">This Page</a></h3>
<ul>
-<li><a class="reference internal" href="#">Flume 1.5.2 Developer Guide</a><ul>
+<li><a class="reference internal" href="#">Flume 1.6.0 Developer Guide</a><ul>
<li><a class="reference internal" href="#introduction">Introduction</a><ul>
<li><a class="reference internal" href="#overview">Overview</a></li>
<li><a class="reference internal" href="#architecture">Architecture</a><ul>
@@ -824,6 +951,7 @@ mechanism that captures the new data and
<li><a class="reference internal" href="#client-sdk">Client SDK</a></li>
<li><a class="reference internal" href="#rpc-client-interface">RPC client
interface</a></li>
<li><a class="reference internal" href="#rpc-clients-avro-and-thrift">RPC
clients - Avro and Thrift</a></li>
+<li><a class="reference internal" href="#secure-rpc-client-thrift">Secure RPC
client - Thrift</a></li>
<li><a class="reference internal" href="#failover-client">Failover
Client</a></li>
<li><a class="reference internal"
href="#loadbalancing-rpc-client">LoadBalancing RPC client</a></li>
</ul>
