Author: buildbot
Date: Thu Oct 15 02:03:35 2015
New Revision: 968967
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/docs/configuration/resources.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Thu Oct 15 02:03:35 2015
@@ -1 +1 @@
-1706153
+1708722
Modified:
websites/staging/slider/trunk/content/docs/configuration/resources.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/configuration/resources.html
(original)
+++ websites/staging/slider/trunk/content/docs/configuration/resources.html Thu
Oct 15 02:03:35 2015
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-resource-specification">Apache Slider Resource
Specification</h1>
+<h1 id="apache-slider-resource-specification">Apache Slider Resource
Specification<a class="headerlink" href="#apache-slider-resource-specification"
title="Permanent link">¶</a></h1>
<ul>
<li><a href="#core">Core Properties</a></li>
<li><a href="#failurepolicy">Container Failure Policy</a></li>
@@ -203,7 +214,7 @@ This includes:
these logs to be copied to HDFS and remotely retrieved, even while the
application is running</p>
<p>As such, it is the core file used by Slider to configure and manage
the application.</p>
-<h2 id="core-properties"><a name="core"></a>Core Properties</h2>
+<h2 id="core-properties"><a name="core"></a>Core Properties<a
class="headerlink" href="#core-properties" title="Permanent
link">¶</a></h2>
<p>An example resource requirement for an application that has two components
"master" and "worker" is as follows. Slider will automatically add the
requirements for the AppMaster for the application. This component is named
"slider-appmaster".</p>
<p>Some parameters that can be specified for a component instance include:</p>
<table>
@@ -234,7 +245,7 @@ the application.</p>
</table>
-<h3 id="component-instance-count-yarncomponentinstances">Component instance
count: <code>yarn.component.instances</code></h3>
+<h3 id="component-instance-count-yarncomponentinstances">Component instance
count: <code>yarn.component.instances</code><a class="headerlink"
href="#component-instance-count-yarncomponentinstances" title="Permanent
link">¶</a></h3>
<p>The property <code>yarn.component.instances</code> is one of the most
foundational one in slider
âit declares how many instances of a component to instantiate on the
cluster.</p>
<p>If the value is set to "0", no instances of a component will be created. If
set
@@ -242,7 +253,7 @@ to a larger number, more instances will
of the application, component-by-component.</p>
<p>The number of instances of each component is application-specific; there
are no recommended
values.</p>
-<h3 id="container-resource-requirements-yarnmemory-and-yarnvcores">Container
resource requirements: <code>yarn.memory</code> and
<code>yarn.vcores</code></h3>
+<h3 id="container-resource-requirements-yarnmemory-and-yarnvcores">Container
resource requirements: <code>yarn.memory</code> and <code>yarn.vcores</code><a
class="headerlink"
href="#container-resource-requirements-yarnmemory-and-yarnvcores"
title="Permanent link">¶</a></h3>
<p>These two properties define how much memory and CPU capacity each
YARN container of this component requires. YARN will queue
container requests until enough capacity exists within the cluster
@@ -278,14 +289,14 @@ time. </p>
has been granted to the container, it will simply be scheduled with less CPU
time. The penalty
for using more CPU than requested is therefore less destructive than
attempting to
use more memory than requested/granted.</p>
-<h4 id="relationship-between-yarnmemory-and-jvm-heap-size">Relationship
between <code>yarn.memory</code> and JVM Heap Size</h4>
+<h4 id="relationship-between-yarnmemory-and-jvm-heap-size">Relationship
between <code>yarn.memory</code> and JVM Heap Size<a class="headerlink"
href="#relationship-between-yarnmemory-and-jvm-heap-size" title="Permanent
link">¶</a></h4>
<p>Java applications deployed by Slider usually have a JVM heap size property
which needs
to be defined as part of the application configuration.</p>
<p>The value of <code>yarn.memory</code> MUST be bigger than the heap size
allocated to any JVM, as a JVM
uses a lot more memory than simply the heap alone. We have found that asking
for at least 50%
more appears to work, though some experimentation will be needed. </p>
<p>Slider does not attempt to derive a heap size for any component from the
YARN allocation.</p>
-<h3 id="component-instance-count-yarnrolepriority">Component instance count:
<code>yarn.role.priority</code></h3>
+<h3 id="component-instance-count-yarnrolepriority">Component instance count:
<code>yarn.role.priority</code><a class="headerlink"
href="#component-instance-count-yarnrolepriority" title="Permanent
link">¶</a></h3>
<p>The property <code>yarn.role.priority</code> has two purposes within Slider
1. It provides a unique index of individual component types. That is, it is not
the name of a component which Slider uses to index components, it is it's
priority
@@ -299,7 +310,7 @@ request (defined by <code>yarn.component
each component's container defined by <code>yarn.memory</code> and
<code>yarn.vcores</code>. It will request
the specified number of components âand keep those requests outstanding
until they are
satisfied.</p>
-<h3 id="example">Example</h3>
+<h3 id="example">Example<a class="headerlink" href="#example" title="Permanent
link">¶</a></h3>
<div class="codehilite"><pre><span class="p">{</span>
"<span class="n">schema</span>" <span class="p">:</span>
"<span class="n">http</span><span class="p">:</span><span
class="o">//</span><span class="n">example</span><span class="p">.</span><span
class="n">org</span><span class="o">/</span><span
class="n">specification</span><span class="o">/</span><span
class="n">v2</span><span class="p">.</span>0<span
class="p">.</span>0"<span class="p">,</span>
"<span class="n">metadata</span>" <span class="p">:</span> <span
class="p">{</span>
@@ -327,14 +338,14 @@ satisfied.</p>
</pre></div>
-<h2 id="the-slider-appmaster-component"><a name="slider-appmaster"></a>The
<code>slider-appmaster</code> component</h2>
+<h2 id="the-slider-appmaster-component"><a name="slider-appmaster"></a>The
<code>slider-appmaster</code> component<a class="headerlink"
href="#the-slider-appmaster-component" title="Permanent link">¶</a></h2>
<p>The examples here all have a component <code>slider-appmaster</code>. This
defines the settings of
the application master itself: the memory and CPU it requires, optionally a
label (see
<a href="#labels">"Labels"</a>). The <code>yarn.role.priority</code> value is
ignored: the priority is always "0";
and the instance count, <code>yarn.component.instances</code> is implicitly
set to "1". </p>
<p>The entry exists primarily to allow applications to configure the amount of
RAM the AM should
request.</p>
-<h2 id="container-failure-policy"><a name="failurepolicy"></a>Container
Failure Policy</h2>
+<h2 id="container-failure-policy"><a name="failurepolicy"></a>Container
Failure Policy<a class="headerlink" href="#container-failure-policy"
title="Permanent link">¶</a></h2>
<p>YARN containers hosting component instances may fail. This can happen
because of</p>
<ol>
<li>A problem in the configuration of the instance.</li>
@@ -348,8 +359,8 @@ and another running program.</li>
<p>Slider reacts to a failed container by requesting a new container from YARN,
preferably on a host that has already hosted an instance of that role. Once
the container is allocated, slider will redeploy an instance of the component.
-As it may time for YARN to have the resources to allocate the container, it
-may take some time for the replacement to be instantiated.</p>
+As it may take time for YARN to have the resources to allocate the container,
+replacements are not immediately instantiated.</p>
<p>Slider tracks failures in an attempt to differentiate problems in the
application package or its configuration from those of the underlying servers.
If a a component fails <em>too many times</em> then slider considers the
application
@@ -359,13 +370,13 @@ itself as failing, and halts.</p>
1. The duration of a failure window, a time period in which failures are
counted.
This duration can span days.
1. The maximum number of failures of any component in this time period.</p>
-<h3 id="failure-threshold-for-a-component">Failure threshold for a
component</h3>
+<h3 id="failure-threshold-for-a-component">Failure threshold for a component<a
class="headerlink" href="#failure-threshold-for-a-component" title="Permanent
link">¶</a></h3>
<p>The number of times a component may fail within a failure window is
defined by the property <code>yarn.container.failure.threshold</code></p>
<p>If set to "0" there are no limits on
the number of times containers may fail.</p>
<p>The failure thresholds for individual components can be set
independently</p>
-<h3 id="failure-window">Failure window</h3>
+<h3 id="failure-window">Failure window<a class="headerlink"
href="#failure-window" title="Permanent link">¶</a></h3>
<p>The failure window can be set by minutes, days and hours. These must be set
in the <code>global</code> options, as they apply to slider only.</p>
<div class="codehilite"><pre><span class="n">yarn</span><span
class="p">.</span><span class="n">container</span><span class="p">.</span><span
class="n">failure</span><span class="p">.</span><span
class="n">window</span><span class="p">.</span><span class="n">days</span>
@@ -388,7 +399,7 @@ restarted.</p>
<p>The default value is <code>yarn.container.failure.window.hours=6</code>;
when changing
the window size, the hour value must be explicitly set, even if to zero, to
change this.</p>
-<h3 id="recommended-values">Recommended values</h3>
+<h3 id="recommended-values">Recommended values<a class="headerlink"
href="#recommended-values" title="Permanent link">¶</a></h3>
<p>We recommend having a duration of a few hours for the window, and a
large failure limit proportional to the the number of instances of that
component </p>
@@ -397,7 +408,7 @@ component </p>
trying to reinstantiate all the components. Yet, if a component does fail
repeatedly, eventually slider will conclude that there is a problem and fail
with the exit code 73, <code>EXIT_DEPLOYMENT_FAILED</code>. </p>
-<h3 id="example_1">Example</h3>
+<h3 id="example_1">Example<a class="headerlink" href="#example_1"
title="Permanent link">¶</a></h3>
<p>Here is a <code>resource.json</code> file for an HBase cluster:</p>
<div class="codehilite"><pre><span class="p">{</span>
"<span class="n">schema</span>" <span class="p">:</span>
"<span class="n">http</span><span class="p">:</span><span
class="o">//</span><span class="n">example</span><span class="p">.</span><span
class="n">org</span><span class="o">/</span><span
class="n">specification</span><span class="o">/</span><span
class="n">v2</span><span class="p">.</span>0<span
class="p">.</span>0"<span class="p">,</span>
@@ -448,10 +459,10 @@ which are frequently failing due to conf
<p>In a production application, large failure thresholds and/or shorter windows
ensures that the application is resilient to transient failures of the
underlying
YARN cluster and hardware.</p>
-<h2 id="placement-policies-and-escalation"><a name="placement"></a>Placement
Policies and escalation</h2>
+<h2 id="placement-policies-and-escalation"><a name="placement"></a>Placement
Policies and escalation<a class="headerlink"
href="#placement-policies-and-escalation" title="Permanent link">¶</a></h2>
<p>Slider can be configured with different options for
<strong>placement</strong> âthe
policies by which it chooses where to ask YARN for nodes.</p>
-<h3 id="placement-policy">Placement Policy</h3>
+<h3 id="placement-policy">Placement Policy<a class="headerlink"
href="#placement-policy" title="Permanent link">¶</a></h3>
<p>The "placement policy" of a component is the set of rules by which Slider
makes
a decision on where to request instances of that component from YARN.</p>
<table>
@@ -499,7 +510,7 @@ set in the property <code>"yarn.componen
<p>If the component were configured to request an explicit port for its REST
endpoint,
then the same URL would reach it whenever this application were deployed
âprovided the host was available and the port not already in use.</p>
-<h4 id="notes">Notes</h4>
+<h4 id="notes">Notes<a class="headerlink" href="#notes" title="Permanent
link">¶</a></h4>
<ol>
<li>
<p>There's no support for <strong>anti-affinity</strong> âi.e. to mandate
that component
@@ -517,7 +528,7 @@ there.</p>
cannot be specified. The Application Master is deployed wherever YARN sees
fit.</p>
</li>
</ol>
-<h3 id="node-failure-threshold-yarnnodefailurethreshold">Node Failure
Threshold, <code>yarn.node.failure.threshold</code></h3>
+<h3 id="node-failure-threshold-yarnnodefailurethreshold">Node Failure
Threshold, <code>yarn.node.failure.threshold</code><a class="headerlink"
href="#node-failure-threshold-yarnnodefailurethreshold" title="Permanent
link">¶</a></h3>
<p>The configuration property <code>yarn.node.failure.threshold</code> defines
how "unreliable"
a node must be before it is skipped for placement requests.</p>
<ol>
@@ -526,7 +537,7 @@ a node must be before it is skipped for
<li>It is reset at the same time as the container failure counters; that is, at
the interval defined by the <code>yarn.container.failure.window</code>
properties</li>
</ol>
-<h3 id="escalation-yarnplacementescalateseconds">Escalation:
<code>yarn.placement.escalate.seconds</code></h3>
+<h3 id="escalation-yarnplacementescalateseconds">Escalation:
<code>yarn.placement.escalate.seconds</code><a class="headerlink"
href="#escalation-yarnplacementescalateseconds" title="Permanent
link">¶</a></h3>
<p>For any component whose placement policy is not "any", Slider saves to HDFS
a record the nodes on which instances were running. When starting a cluster,
it uses
this history to identify hosts on which to request instances. </p>
@@ -570,7 +581,7 @@ we would recommend for an escalation tim
</pre></div>
-<p>This declares that the <code>HBASE_MASTER</code> placement should be
escalated after one second,
+<p>This declares that the <code>HBASE_MASTER</code> placement should be
escalated after ten seconds,
but that that <code>HBASE_REGIONSERVER</code> instances should have an
escalation timeout of 600
seconds âten minutes. These values were chosen as an HBase Master can be
allocated
anywhere in the cluster, but a region server is significantly faster if
restarted
@@ -578,10 +589,10 @@ on the same node on which it previously
have replicated all data elsewhere, it will have been scattered across the
cluster
âresulting in remote access for most of the data, at least until a full
compaction
has taken place.</p>
-<h4 id="notes_1">Notes</h4>
+<h4 id="notes_1">Notes<a class="headerlink" href="#notes_1" title="Permanent
link">¶</a></h4>
<ol>
<li>
-<p>Escalation goes directory from "specific node" to "anywhere in cluster"; it
does
+<p>Escalation goes directly from "specific node" to "anywhere in cluster"; it
does
not have any intermediate "same-rack" policy.</p>
</li>
<li>
@@ -607,8 +618,8 @@ is not available or lacks capacity, the
<p>For more details, see: <a href="/design/rolehistory.html">Role
History</a></p>
</li>
</ol>
-<h2 id="using-labels"><a name="labels"></a>Using Labels</h2>
-<p>The <code>resources.json</code> file include specifications the labels to
be used when allocating containers for the components. The details of the YARN
Label feature can be found at <a
href="https://issues.apache.org/jira/browse/YARN-796";>YARN-796</a>.</p>
+<h2 id="using-labels"><a name="labels"></a>Using Labels<a class="headerlink"
href="#using-labels" title="Permanent link">¶</a></h2>
+<p>The <code>resources.json</code> file includes specifications of the labels
to be used when allocating containers for the components. The details of the
YARN Label feature can be found at <a
href="https://issues.apache.org/jira/browse/YARN-796";>YARN-796</a>.</p>
<p>In summary: </p>
<ul>
<li>Nodes can be assigned one or more labels</li>
@@ -619,7 +630,7 @@ is not available or lacks capacity, the
<p>This way, you can guarantee that a certain set of nodes are reserved for an
application or for a component within an application.</p>
<p>Label expression is specified through property
<code>yarn.label.expression</code>. When no label expression is specified then
it is assumed that only non-labeled nodes are used when allocating containers
for component instances.</p>
<p>If a label expression is specified for the <code>slider-appmaster</code>
component then it also becomes the default label expression for all component.
</p>
-<h4 id="example_2">Example</h4>
+<h4 id="example_2">Example<a class="headerlink" href="#example_2"
title="Permanent link">¶</a></h4>
<p>Here is a <code>resource.json</code> file for an HBase cluster which uses
labels.</p>
<p>The label for the application master is <code>hbase1</code> and the label
expression for the HBASE_MASTER components is <code>hbase1_master</code>.
<code>HBASE_REGIONSERVER</code> instances will automatically use label
<code>hbase1</code>.</p>
@@ -657,7 +668,7 @@ is not available or lacks capacity, the
<li>Perform refresh queue (<code>yarn -refreshqueue</code>) </li>
<li>Create the Slider application against the above queue using parameter
--queue while creating the application</li>
</ol>
-<h3 id="notes_2">Notes</h3>
+<h3 id="notes_2">Notes<a class="headerlink" href="#notes_2" title="Permanent
link">¶</a></h3>
<ol>
<li>
<p>If a label is defined in the <code>global</code> section, it will also
apply to all components which do
@@ -681,7 +692,7 @@ any global- or appmaster- spettings, set
the application instance will not reach its requested size.</p>
</li>
</ol>
-<h2 id="log-aggregation"><a name="logagg"></a>Log Aggregation</h2>
+<h2 id="log-aggregation"><a name="logagg"></a>Log Aggregation<a
class="headerlink" href="#log-aggregation" title="Permanent
link">¶</a></h2>
<p>Log aggregation at regular intervals for long running services (LRS) needs
to be enabled at the YARN level before
any application can exploit this functionality. To enable set
<code>yarn.log-aggregation-enable</code> to true and the
interval property to a positive value of 3600 (in secs) or more. If set to a
positive value less than 3600 (1 hour)
@@ -727,7 +738,7 @@ expressions. These properties need to be
<p>The details of the YARN Log Aggregation feature can be found at <a
href="https://issues.apache.org/jira/browse/YARN-2468";>YARN-2468</a>.</p>
-<h2 id="putting-it-all-together">Putting it all together</h2>
+<h2 id="putting-it-all-together">Putting it all together<a class="headerlink"
href="#putting-it-all-together" title="Permanent link">¶</a></h2>
<p>Here is an example of a definition of an HBase cluster.</p>
<div class="codehilite"><pre><span class="p">{</span>
"<span class="n">schema</span>"<span class="p">:</span>
"<span class="n">http</span><span class="p">:</span><span
class="o">//</span><span class="n">example</span><span class="p">.</span><span
class="n">org</span><span class="o">/</span><span
class="n">specification</span><span class="o">/</span><span
class="n">v2</span><span class="p">.</span>0<span
class="p">.</span>0"<span class="p">,</span>
