Author: buildbot
Date: Wed Jul 29 20:33:50 2015
New Revision: 960078
Log:
Staging update by buildbot for flume
Modified:
websites/staging/flume/trunk/content/ (props changed)
websites/staging/flume/trunk/content/.doctrees/FlumeDeveloperGuide.doctree
websites/staging/flume/trunk/content/.doctrees/FlumeUserGuide.doctree
websites/staging/flume/trunk/content/.doctrees/documentation.doctree
websites/staging/flume/trunk/content/.doctrees/download.doctree
websites/staging/flume/trunk/content/.doctrees/environment.pickle
websites/staging/flume/trunk/content/.doctrees/getinvolved.doctree
websites/staging/flume/trunk/content/.doctrees/index.doctree
websites/staging/flume/trunk/content/.doctrees/license.doctree
websites/staging/flume/trunk/content/.doctrees/mailinglists.doctree
websites/staging/flume/trunk/content/.doctrees/releases/1.0.0.doctree
websites/staging/flume/trunk/content/.doctrees/releases/1.1.0.doctree
websites/staging/flume/trunk/content/.doctrees/releases/1.2.0.doctree
websites/staging/flume/trunk/content/.doctrees/releases/1.3.0.doctree
websites/staging/flume/trunk/content/.doctrees/releases/1.3.1.doctree
websites/staging/flume/trunk/content/.doctrees/releases/1.4.0.doctree
websites/staging/flume/trunk/content/.doctrees/releases/1.5.0.1.doctree
websites/staging/flume/trunk/content/.doctrees/releases/1.5.0.doctree
websites/staging/flume/trunk/content/.doctrees/releases/1.5.2.doctree
websites/staging/flume/trunk/content/.doctrees/releases/1.6.0.doctree
websites/staging/flume/trunk/content/.doctrees/releases/index.doctree
websites/staging/flume/trunk/content/.doctrees/source.doctree
websites/staging/flume/trunk/content/.doctrees/team.doctree
websites/staging/flume/trunk/content/FlumeDeveloperGuide.html
websites/staging/flume/trunk/content/FlumeUserGuide.html
websites/staging/flume/trunk/content/_sources/FlumeDeveloperGuide.txt
websites/staging/flume/trunk/content/_sources/FlumeUserGuide.txt
websites/staging/flume/trunk/content/documentation.html
websites/staging/flume/trunk/content/project-reports.html
websites/staging/flume/trunk/content/releases/index.html
websites/staging/flume/trunk/content/searchindex.js
Propchange: websites/staging/flume/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Wed Jul 29 20:33:50 2015
@@ -1 +1 @@
-1683005
+1693341
Modified:
websites/staging/flume/trunk/content/.doctrees/FlumeDeveloperGuide.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/FlumeUserGuide.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/documentation.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/download.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/environment.pickle
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/getinvolved.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/index.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/license.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/mailinglists.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.0.0.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.1.0.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.2.0.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.3.0.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.3.1.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.4.0.doctree
==============================================================================
Binary files - no diff available.
Modified:
websites/staging/flume/trunk/content/.doctrees/releases/1.5.0.1.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.5.0.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.5.2.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.6.0.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/releases/index.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/source.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/team.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/FlumeDeveloperGuide.html
==============================================================================
--- websites/staging/flume/trunk/content/FlumeDeveloperGuide.html (original)
+++ websites/staging/flume/trunk/content/FlumeDeveloperGuide.html Wed Jul 29
20:33:50 2015
@@ -7,7 +7,7 @@
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
- <title>Flume 1.6.0 Developer Guide — Apache Flume</title>
+ <title>Flume 1.7.0-SNAPSHOT Developer Guide — Apache Flume</title>
<link rel="stylesheet" href="_static/flume.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
@@ -27,7 +27,7 @@
<link rel="top" title="Apache Flume" href="index.html" />
<link rel="up" title="Documentation" href="documentation.html" />
<link rel="next" title="Releases" href="releases/index.html" />
- <link rel="prev" title="Flume 1.6.0 User Guide" href="FlumeUserGuide.html"
/>
+ <link rel="prev" title="Flume 1.7.0-SNAPSHOT User Guide"
href="FlumeUserGuide.html" />
</head>
<body>
<div class="header">
@@ -59,8 +59,8 @@
<div class="bodywrapper">
<div class="body">
- <div class="section" id="flume-1-6-0-developer-guide">
-<h1>Flume 1.6.0 Developer Guide<a class="headerlink"
href="#flume-1-6-0-developer-guide" title="Permalink to this
headline">¶</a></h1>
+ <div class="section" id="flume-1-7-0-snapshot-developer-guide">
+<h1>Flume 1.7.0-SNAPSHOT Developer Guide<a class="headerlink"
href="#flume-1-7-0-snapshot-developer-guide" title="Permalink to this
headline">¶</a></h1>
<div class="section" id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to
this headline">¶</a></h2>
<div class="section" id="overview">
@@ -677,9 +677,10 @@ Flume <tt class="docutils literal"><span
<img alt="Transaction sequence diagram" src="_images/DevGuide_image01.png" />
</div>
<p>A <tt class="docutils literal"><span class="pre">Transaction</span></tt> is
implemented within a <tt class="docutils literal"><span
class="pre">Channel</span></tt> implementation. Each
-<tt class="docutils literal"><span class="pre">Source</span></tt> and <tt
class="docutils literal"><span class="pre">Sink</span></tt> that is connected
to <tt class="docutils literal"><span class="pre">Channel</span></tt> must
obtain a
-<tt class="docutils literal"><span class="pre">Transaction</span></tt> object.
The <tt class="docutils literal"><span class="pre">Source</span></tt>s actually
use a <tt class="docutils literal"><span class="pre">ChannelSelector</span></tt>
-interface to encapsulate the <tt class="docutils literal"><span
class="pre">Transaction</span></tt>. The operation to stage an
+<tt class="docutils literal"><span class="pre">Source</span></tt> and <tt
class="docutils literal"><span class="pre">Sink</span></tt> that is connected
to a <tt class="docutils literal"><span class="pre">Channel</span></tt> must
obtain a
+<tt class="docutils literal"><span class="pre">Transaction</span></tt> object.
The <tt class="docutils literal"><span class="pre">Source</span></tt>s use a
<tt class="docutils literal"><span class="pre">ChannelProcessor</span></tt>
+to manage the <tt class="docutils literal"><span
class="pre">Transaction</span></tt>s, the <tt class="docutils literal"><span
class="pre">Sink</span></tt>s manage them explicitly via
+their configured <tt class="docutils literal"><span
class="pre">Channel</span></tt>. The operation to stage an
<tt class="docutils literal"><span class="pre">Event</span></tt> (put it into
a <tt class="docutils literal"><span class="pre">Channel</span></tt>) or
extract an <tt class="docutils literal"><span class="pre">Event</span></tt>
(take it out of a
<tt class="docutils literal"><span class="pre">Channel</span></tt>) is done
inside an active <tt class="docutils literal"><span
class="pre">Transaction</span></tt>. For example:</p>
<div class="highlight-java"><div class="highlight"><pre><span
class="n">Channel</span> <span class="n">ch</span> <span class="o">=</span>
<span class="k">new</span> <span class="n">MemoryChannel</span><span
class="o">();</span>
@@ -786,8 +787,6 @@ processing its own configuration setting
<span class="k">if</span> <span class="o">(</span><span
class="n">t</span> <span class="k">instanceof</span> <span
class="n">Error</span><span class="o">)</span> <span class="o">{</span>
<span class="k">throw</span> <span class="o">(</span><span
class="n">Error</span><span class="o">)</span><span class="n">t</span><span
class="o">;</span>
<span class="o">}</span>
- <span class="o">}</span> <span class="k">finally</span> <span
class="o">{</span>
- <span class="n">txn</span><span class="o">.</span><span
class="na">close</span><span class="o">();</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">status</span><span
class="o">;</span>
<span class="o">}</span>
@@ -798,13 +797,12 @@ processing its own configuration setting
<div class="section" id="source">
<h4>Source<a class="headerlink" href="#source" title="Permalink to this
headline">¶</a></h4>
<p>The purpose of a <tt class="docutils literal"><span
class="pre">Source</span></tt> is to receive data from an external client and
store
-it into the <tt class="docutils literal"><span
class="pre">Channel</span></tt>. A <tt class="docutils literal"><span
class="pre">Source</span></tt> can get an instance of its own
-<tt class="docutils literal"><span class="pre">ChannelProcessor</span></tt> to
process an <tt class="docutils literal"><span class="pre">Event</span></tt>.
The <tt class="docutils literal"><span class="pre">ChannelProcessor</span></tt>
in turn
-can get an instance of its own <tt class="docutils literal"><span
class="pre">ChannelSelector</span></tt> that’s used to get the
-<tt class="docutils literal"><span class="pre">Channel</span></tt>s associated
with the <tt class="docutils literal"><span class="pre">Source</span></tt>, as
configured in the Flume
-properties file. A <tt class="docutils literal"><span
class="pre">Transaction</span></tt> can then be retrieved from each associated
-<tt class="docutils literal"><span class="pre">Channel</span></tt> so that the
<tt class="docutils literal"><span class="pre">Source</span></tt> can place <tt
class="docutils literal"><span class="pre">Event</span></tt>s into the <tt
class="docutils literal"><span class="pre">Channel</span></tt>
-reliably, within a <tt class="docutils literal"><span
class="pre">Transaction</span></tt>.</p>
+it into the configured <tt class="docutils literal"><span
class="pre">Channel</span></tt>s. A <tt class="docutils literal"><span
class="pre">Source</span></tt> can get an instance of its own
+<tt class="docutils literal"><span class="pre">ChannelProcessor</span></tt> to
process an <tt class="docutils literal"><span class="pre">Event</span></tt>,
commited within a <tt class="docutils literal"><span
class="pre">Channel</span></tt>
+local transaction, in serial. In the case of an exception, required
+<tt class="docutils literal"><span class="pre">Channel</span></tt>s will
propagate the exception, all <tt class="docutils literal"><span
class="pre">Channel</span></tt>s will rollback their
+transaction, but events processed previously on other <tt class="docutils
literal"><span class="pre">Channel</span></tt>s will remain
+committed.</p>
<p>Similar to the <tt class="docutils literal"><span
class="pre">SinkRunner.PollingRunner</span></tt> <tt class="docutils
literal"><span class="pre">Runnable</span></tt>, thereâs
a <tt class="docutils literal"><span class="pre">PollingRunner</span></tt> <tt
class="docutils literal"><span class="pre">Runnable</span></tt> that executes
on a thread created when the
Flume framework calls <tt class="docutils literal"><span
class="pre">PollableSourceRunner.start()</span></tt>. Each configured
@@ -849,24 +847,17 @@ mechanism that captures the new data and
<span class="kd">public</span> <span class="n">Status</span> <span
class="nf">process</span><span class="o">()</span> <span
class="kd">throws</span> <span class="n">EventDeliveryException</span> <span
class="o">{</span>
<span class="n">Status</span> <span class="n">status</span> <span
class="o">=</span> <span class="kc">null</span><span class="o">;</span>
- <span class="c1">// Start transaction</span>
- <span class="n">Channel</span> <span class="n">ch</span> <span
class="o">=</span> <span class="n">getChannel</span><span class="o">();</span>
- <span class="n">Transaction</span> <span class="n">txn</span> <span
class="o">=</span> <span class="n">ch</span><span class="o">.</span><span
class="na">getTransaction</span><span class="o">();</span>
- <span class="n">txn</span><span class="o">.</span><span
class="na">begin</span><span class="o">();</span>
<span class="k">try</span> <span class="o">{</span>
- <span class="c1">// This try clause includes whatever Channel operations
you want to do</span>
+ <span class="c1">// This try clause includes whatever Channel/Event
operations you want to do</span>
<span class="c1">// Receive new data</span>
<span class="n">Event</span> <span class="n">e</span> <span
class="o">=</span> <span class="n">getSomeData</span><span class="o">();</span>
<span class="c1">// Store the Event into this Source's associated
Channel(s)</span>
- <span class="n">getChannelProcessor</span><span
class="o">().</span><span class="na">processEvent</span><span
class="o">(</span><span class="n">e</span><span class="o">)</span>
+ <span class="n">getChannelProcessor</span><span
class="o">().</span><span class="na">processEvent</span><span
class="o">(</span><span class="n">e</span><span class="o">);</span>
- <span class="n">txn</span><span class="o">.</span><span
class="na">commit</span><span class="o">();</span>
<span class="n">status</span> <span class="o">=</span> <span
class="n">Status</span><span class="o">.</span><span
class="na">READY</span><span class="o">;</span>
<span class="o">}</span> <span class="k">catch</span> <span
class="o">(</span><span class="n">Throwable</span> <span
class="n">t</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">txn</span><span class="o">.</span><span
class="na">rollback</span><span class="o">();</span>
-
<span class="c1">// Log exception, handle individual exceptions as
needed</span>
<span class="n">status</span> <span class="o">=</span> <span
class="n">Status</span><span class="o">.</span><span
class="na">BACKOFF</span><span class="o">;</span>
@@ -933,7 +924,7 @@ mechanism that captures the new data and
<h3><a href="index.html">This Page</a></h3>
<ul>
-<li><a class="reference internal" href="#">Flume 1.6.0 Developer Guide</a><ul>
+<li><a class="reference internal" href="#">Flume 1.7.0-SNAPSHOT Developer
Guide</a><ul>
<li><a class="reference internal" href="#introduction">Introduction</a><ul>
<li><a class="reference internal" href="#overview">Overview</a></li>
<li><a class="reference internal" href="#architecture">Architecture</a><ul>
Modified: websites/staging/flume/trunk/content/FlumeUserGuide.html
==============================================================================
--- websites/staging/flume/trunk/content/FlumeUserGuide.html (original)
+++ websites/staging/flume/trunk/content/FlumeUserGuide.html Wed Jul 29
20:33:50 2015
@@ -7,7 +7,7 @@
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
- <title>Flume 1.6.0 User Guide — Apache Flume</title>
+ <title>Flume 1.7.0-SNAPSHOT User Guide — Apache Flume</title>
<link rel="stylesheet" href="_static/flume.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
@@ -26,7 +26,7 @@
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="top" title="Apache Flume" href="index.html" />
<link rel="up" title="Documentation" href="documentation.html" />
- <link rel="next" title="Flume 1.6.0 Developer Guide"
href="FlumeDeveloperGuide.html" />
+ <link rel="next" title="Flume 1.7.0-SNAPSHOT Developer Guide"
href="FlumeDeveloperGuide.html" />
<link rel="prev" title="Documentation" href="documentation.html" />
</head>
<body>
@@ -59,8 +59,8 @@
<div class="bodywrapper">
<div class="body">
- <div class="section" id="flume-1-6-0-user-guide">
-<h1>Flume 1.6.0 User Guide<a class="headerlink" href="#flume-1-6-0-user-guide"
title="Permalink to this headline">¶</a></h1>
+ <div class="section" id="flume-1-7-0-snapshot-user-guide">
+<h1>Flume 1.7.0-SNAPSHOT User Guide<a class="headerlink"
href="#flume-1-7-0-snapshot-user-guide" title="Permalink to this
headline">¶</a></h1>
<div class="section" id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to
this headline">¶</a></h2>
<div class="section" id="overview">
@@ -1504,6 +1504,19 @@ indicates that they are part of the same
<td>Maximum time (in ms) before a batch will be written to Channel
The batch will be written whenever the first of size and time will be
reached.</td>
</tr>
+<tr class="row-odd"><td>backoffSleepIncrement</td>
+<td>1000</td>
+<td>Initial and incremental wait time that is triggered when a Kafka Topic
appears to be empty.
+Wait period will reduce aggressive pinging of an empty Kafka Topic. One
second is ideal for
+ingestion use cases but a lower value may be required for low latency
operations with
+interceptors.</td>
+</tr>
+<tr class="row-even"><td>maxBackoffSleep</td>
+<td>5000</td>
+<td>Maximum wait time that is triggered when a Kafka Topic appears to be
empty. Five seconds is
+ideal for ingestion use cases but a lower value may be required for low
latency operations
+with interceptors.</td>
+</tr>
<tr class="row-odd"><td>Other Kafka Consumer Properties</td>
<td>–</td>
<td>These properties are used to configure the Kafka Consumer. Any producer
property supported
@@ -3023,8 +3036,8 @@ Required properties are in <strong>bold<
<td>The type of the Java truststore. This can be “JKS” or other
supported Java truststore type.</td>
</tr>
<tr class="row-odd"><td>exclude-protocols</td>
-<td>SSLv2Hello SSLv3</td>
-<td>Space-separated list of SSL/TLS protocols to exclude</td>
+<td>SSLv3</td>
+<td>Space-separated list of SSL/TLS protocols to exclude. SSLv3 will always be
excluded in addition to the protocols specified.</td>
</tr>
<tr class="row-even"><td>maxIoWorkers</td>
<td>2 * the number of available processors in the machine</td>
@@ -3521,9 +3534,9 @@ will read this information from the firs
<p>Required properties are in <strong>bold</strong>.</p>
<table border="1" class="docutils">
<colgroup>
-<col width="8%" />
-<col width="30%" />
-<col width="62%" />
+<col width="7%" />
+<col width="16%" />
+<col width="77%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Property Name</th>
@@ -3560,6 +3573,18 @@ will read this information from the firs
<td>org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl</td>
<td>The FQCN of a class implementing
org.apache.flume.sink.solr.morphline.MorphlineHandler</td>
</tr>
+<tr class="row-odd"><td>isProductionMode</td>
+<td>false</td>
+<td>This flag should be enabled for mission critical, large-scale online
production systems that need to make progress without downtime when
unrecoverable exceptions occur. Corrupt or malformed parser input data, parser
bugs, and errors related to unknown Solr schema fields produce unrecoverable
exceptions.</td>
+</tr>
+<tr class="row-even"><td>recoverableExceptionClasses</td>
+<td>org.apache.solr.client.solrj.SolrServerException</td>
+<td>Comma separated list of recoverable exceptions that tend to be transient,
in which case the corresponding task can be retried. Examples include network
connection errors, timeouts, etc. When the production mode flag is set to true,
the recoverable exceptions configured using this parameter will not be ignored
and hence will lead to retries.</td>
+</tr>
+<tr class="row-odd"><td>isIgnoringRecoverableExceptions</td>
+<td>false</td>
+<td>This flag should be enabled, if an unrecoverable exception is accidentally
misclassified as recoverable. This enables the sink to make progress and avoid
retrying an event forever.</td>
+</tr>
</tbody>
</table>
<p>Example for agent named a1:</p>
@@ -5320,7 +5345,7 @@ polling rather than terminating.</p>
<h2>Log4J Appender<a class="headerlink" href="#log4j-appender"
title="Permalink to this headline">¶</a></h2>
<p>Appends Log4j events to a flume agent’s avro source. A client using
this
appender must have the flume-ng-sdk in the classpath (eg,
-flume-ng-sdk-1.6.0.jar).
+flume-ng-sdk-1.7.0-SNAPSHOT.jar).
Required properties are in <strong>bold</strong>.</p>
<table border="1" class="docutils">
<colgroup>
@@ -5400,7 +5425,7 @@ then the schema will be included as a Fl
<h2>Load Balancing Log4J Appender<a class="headerlink"
href="#load-balancing-log4j-appender" title="Permalink to this
headline">¶</a></h2>
<p>Appends Log4j events to a list of flume agent’s avro source. A client
using this
appender must have the flume-ng-sdk in the classpath (eg,
-flume-ng-sdk-1.6.0.jar). This appender supports a round-robin and random
+flume-ng-sdk-1.7.0-SNAPSHOT.jar). This appender supports a round-robin and
random
scheme for performing the load balancing. It also supports a configurable
backoff
timeout so that down agents are removed temporarily from the set of hosts
Required properties are in <strong>bold</strong>.</p>
@@ -6159,7 +6184,7 @@ can be leveraged to move the Flume agent
<h3><a href="index.html">This Page</a></h3>
<ul>
-<li><a class="reference internal" href="#">Flume 1.6.0 User Guide</a><ul>
+<li><a class="reference internal" href="#">Flume 1.7.0-SNAPSHOT User
Guide</a><ul>
<li><a class="reference internal" href="#introduction">Introduction</a><ul>
<li><a class="reference internal" href="#overview">Overview</a></li>
<li><a class="reference internal" href="#system-requirements">System
Requirements</a></li>
Modified: websites/staging/flume/trunk/content/_sources/FlumeDeveloperGuide.txt
==============================================================================
--- websites/staging/flume/trunk/content/_sources/FlumeDeveloperGuide.txt
(original)
+++ websites/staging/flume/trunk/content/_sources/FlumeDeveloperGuide.txt Wed
Jul 29 20:33:50 2015
@@ -15,7 +15,7 @@
======================================
-Flume 1.6.0 Developer Guide
+Flume 1.7.0-SNAPSHOT Developer Guide
======================================
Introduction
@@ -644,9 +644,10 @@ Flume ``Transaction``.
:alt: Transaction sequence diagram
A ``Transaction`` is implemented within a ``Channel`` implementation. Each
-``Source`` and ``Sink`` that is connected to ``Channel`` must obtain a
-``Transaction`` object. The ``Source``\ s actually use a ``ChannelSelector``
-interface to encapsulate the ``Transaction``. The operation to stage an
+``Source`` and ``Sink`` that is connected to a ``Channel`` must obtain a
+``Transaction`` object. The ``Source``\ s use a ``ChannelProcessor``
+to manage the ``Transaction``\ s, the ``Sink``\ s manage them explicitly via
+their configured ``Channel``. The operation to stage an
``Event`` (put it into a ``Channel``) or extract an ``Event`` (take it out of a
``Channel``) is done inside an active ``Transaction``. For example:
@@ -759,8 +760,6 @@ processing its own configuration setting
if (t instanceof Error) {
throw (Error)t;
}
- } finally {
- txn.close();
}
return status;
}
@@ -770,13 +769,12 @@ Source
~~~~~~
The purpose of a ``Source`` is to receive data from an external client and
store
-it into the ``Channel``. A ``Source`` can get an instance of its own
-``ChannelProcessor`` to process an ``Event``. The ``ChannelProcessor`` in turn
-can get an instance of its own ``ChannelSelector`` that's used to get the
-``Channel``\ s associated with the ``Source``, as configured in the Flume
-properties file. A ``Transaction`` can then be retrieved from each associated
-``Channel`` so that the ``Source`` can place ``Event``\ s into the ``Channel``
-reliably, within a ``Transaction``.
+it into the configured ``Channel``\ s. A ``Source`` can get an instance of its
own
+``ChannelProcessor`` to process an ``Event``, commited within a ``Channel``
+local transaction, in serial. In the case of an exception, required
+``Channel``\ s will propagate the exception, all ``Channel``\ s will rollback
their
+transaction, but events processed previously on other ``Channel``\ s will
remain
+committed.
Similar to the ``SinkRunner.PollingRunner`` ``Runnable``, thereâs
a ``PollingRunner`` ``Runnable`` that executes on a thread created when the
@@ -826,24 +824,17 @@ mechanism that captures the new data and
public Status process() throws EventDeliveryException {
Status status = null;
- // Start transaction
- Channel ch = getChannel();
- Transaction txn = ch.getTransaction();
- txn.begin();
try {
- // This try clause includes whatever Channel operations you want to do
+ // This try clause includes whatever Channel/Event operations you want
to do
// Receive new data
Event e = getSomeData();
// Store the Event into this Source's associated Channel(s)
- getChannelProcessor().processEvent(e)
+ getChannelProcessor().processEvent(e);
- txn.commit();
status = Status.READY;
} catch (Throwable t) {
- txn.rollback();
-
// Log exception, handle individual exceptions as needed
status = Status.BACKOFF;
Modified: websites/staging/flume/trunk/content/_sources/FlumeUserGuide.txt
==============================================================================
--- websites/staging/flume/trunk/content/_sources/FlumeUserGuide.txt (original)
+++ websites/staging/flume/trunk/content/_sources/FlumeUserGuide.txt Wed Jul 29
20:33:50 2015
@@ -15,7 +15,7 @@
======================================
-Flume 1.6.0 User Guide
+Flume 1.7.0-SNAPSHOT User Guide
======================================
Introduction
@@ -1152,6 +1152,13 @@ Property Name Default
batchSize 1000 Maximum number of messages
written to Channel in one batch
batchDurationMillis 1000 Maximum time (in ms) before a
batch will be written to Channel
The batch will be written
whenever the first of size and time will be reached.
+backoffSleepIncrement 1000 Initial and incremental wait
time that is triggered when a Kafka Topic appears to be empty.
+ Wait period will reduce
aggressive pinging of an empty Kafka Topic. One second is ideal for
+ ingestion use cases but a lower
value may be required for low latency operations with
+ interceptors.
+maxBackoffSleep 5000 Maximum wait time that is
triggered when a Kafka Topic appears to be empty. Five seconds is
+ ideal for ingestion use cases
but a lower value may be required for low latency operations
+ with interceptors.
Other Kafka Consumer Properties -- These properties are used to
configure the Kafka Consumer. Any producer property supported
by Kafka can be used. The only
requirement is to prepend the property name with the prefix ``kafka.``.
For example:
kafka.consumer.timeout.ms
@@ -1952,7 +1959,7 @@ trust-all-certs false
truststore --
The path to a custom Java truststore file. Flume uses the certificate
authority information in this file to determine whether the remote Avro
Source's SSL authentication credentials should be trusted. If not specified,
the default Java JSSE certificate authority files (typically "jssecacerts" or
"cacerts" in the Oracle JRE) will be used.
truststore-password --
The password for the specified truststore.
truststore-type JKS
The type of the Java truststore. This can be "JKS" or other supported Java
truststore type.
-exclude-protocols SSLv2Hello SSLv3
Space-separated list of SSL/TLS protocols to exclude
+exclude-protocols SSLv3
Space-separated list of SSL/TLS protocols to exclude. SSLv3 will always be
excluded in addition to the protocols specified.
maxIoWorkers 2 * the number of available processors in the
machine The maximum number of I/O worker threads. This is configured on the
NettyAvroRpcClient NioClientSocketChannelFactory.
==========================
=====================================================
===========================================================================================
@@ -2241,17 +2248,20 @@ The type is the FQCN: org.apache.flume.s
Required properties are in **bold**.
-===================
=======================================================================
========================
-Property Name Default
Description
-===================
=======================================================================
========================
-**channel** --
-**type** --
The component type name, needs to be
``org.apache.flume.sink.solr.morphline.MorphlineSolrSink``
-**morphlineFile** --
The relative or absolute path on the local file system to the
morphline configuration file. Example: ``/etc/flume-ng/conf/morphline.conf``
-morphlineId null
Optional name used to identify a morphline if there are multiple
morphlines in a morphline config file
-batchSize 1000
The maximum number of events to take per flume transaction.
-batchDurationMillis 1000
The maximum duration per flume transaction (ms). The transaction
commits after this duration or when batchSize is exceeded, whichever comes
first.
-handlerClass org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl
The FQCN of a class implementing
org.apache.flume.sink.solr.morphline.MorphlineHandler
-===================
=======================================================================
========================
+===============================
=======================================================================
========================
+Property Name Default
Description
+===============================
=======================================================================
========================
+**channel** --
+**type** --
The component type name, needs to be
``org.apache.flume.sink.solr.morphline.MorphlineSolrSink``
+**morphlineFile** --
The relative or absolute path on the local file
system to the morphline configuration file. Example:
``/etc/flume-ng/conf/morphline.conf``
+morphlineId null
Optional name used to identify a morphline if there
are multiple morphlines in a morphline config file
+batchSize 1000
The maximum number of events to take per flume
transaction.
+batchDurationMillis 1000
The maximum duration per flume transaction (ms).
The transaction commits after this duration or when batchSize is exceeded,
whichever comes first.
+handlerClass
org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl The
FQCN of a class implementing
org.apache.flume.sink.solr.morphline.MorphlineHandler
+isProductionMode false
This flag should be enabled for mission critical,
large-scale online production systems that need to make progress without
downtime when unrecoverable exceptions occur. Corrupt or malformed parser input
data, parser bugs, and errors related to unknown Solr schema fields produce
unrecoverable exceptions.
+recoverableExceptionClasses
org.apache.solr.client.solrj.SolrServerException Comma
separated list of recoverable exceptions that tend to be transient, in which
case the corresponding task can be retried. Examples include network connection
errors, timeouts, etc. When the production mode flag is set to true, the
recoverable exceptions configured using this parameter will not be ignored and
hence will lead to retries.
+isIgnoringRecoverableExceptions false
This flag should be enabled, if an unrecoverable
exception is accidentally misclassified as recoverable. This enables the sink
to make progress and avoid retrying an event forever.
+===============================
=======================================================================
========================
Example for agent named a1:
@@ -3420,7 +3430,7 @@ Log4J Appender
Appends Log4j events to a flume agent's avro source. A client using this
appender must have the flume-ng-sdk in the classpath (eg,
-flume-ng-sdk-1.6.0.jar).
+flume-ng-sdk-1.7.0-SNAPSHOT.jar).
Required properties are in **bold**.
===================== =======
==================================================================================
@@ -3484,7 +3494,7 @@ Load Balancing Log4J Appender
Appends Log4j events to a list of flume agent's avro source. A client using
this
appender must have the flume-ng-sdk in the classpath (eg,
-flume-ng-sdk-1.6.0.jar). This appender supports a round-robin and random
+flume-ng-sdk-1.7.0-SNAPSHOT.jar). This appender supports a round-robin and
random
scheme for performing the load balancing. It also supports a configurable
backoff
timeout so that down agents are removed temporarily from the set of hosts
Required properties are in **bold**.
Modified: websites/staging/flume/trunk/content/documentation.html
==============================================================================
--- websites/staging/flume/trunk/content/documentation.html (original)
+++ websites/staging/flume/trunk/content/documentation.html Wed Jul 29 20:33:50
2015
@@ -25,7 +25,7 @@
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="top" title="Apache Flume" href="index.html" />
- <link rel="next" title="Flume 1.6.0 User Guide" href="FlumeUserGuide.html"
/>
+ <link rel="next" title="Flume 1.7.0-SNAPSHOT User Guide"
href="FlumeUserGuide.html" />
<link rel="prev" title="Download" href="download.html" />
</head>
<body>
Modified: websites/staging/flume/trunk/content/project-reports.html
==============================================================================
--- websites/staging/flume/trunk/content/project-reports.html (original)
+++ websites/staging/flume/trunk/content/project-reports.html Wed Jul 29
20:33:50 2015
@@ -1,5 +1,5 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";>
-<!-- Generated by Apache Maven Doxia at Jun 1, 2015 -->
+<!-- Generated by Apache Maven Doxia at Jul 29, 2015 -->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -12,7 +12,7 @@
<script type="text/javascript"
src="./js/apache-maven-fluido.min.js"></script>
- <meta name="Date-Revision-yyyymmdd" content="20150601" />
+ <meta name="Date-Revision-yyyymmdd" content="20150729" />
<meta http-equiv="Content-Language" content="en" />
</head>
@@ -27,7 +27,7 @@
- <li id="publishDate" class="pull-right">Last Published:
2015-06-01</li>
+ <li id="publishDate" class="pull-right">Last Published:
2015-07-29</li>
</ul>
</div>
Modified: websites/staging/flume/trunk/content/releases/index.html
==============================================================================
--- websites/staging/flume/trunk/content/releases/index.html (original)
+++ websites/staging/flume/trunk/content/releases/index.html Wed Jul 29
20:33:50 2015
@@ -26,7 +26,7 @@
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="top" title="Apache Flume" href="../index.html" />
<link rel="next" title="Version 1.6.0" href="1.6.0.html" />
- <link rel="prev" title="Flume 1.6.0 Developer Guide"
href="../FlumeDeveloperGuide.html" />
+ <link rel="prev" title="Flume 1.7.0-SNAPSHOT Developer Guide"
href="../FlumeDeveloperGuide.html" />
</head>
<body>
<div class="header">
