Author: buildbot
Date: Thu Oct 9 00:28:02 2014
New Revision: 925106
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/docs/manpage.html
websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Thu Oct 9 00:28:02 2014
@@ -1 +1 @@
-1629803
+1630262
Modified: websites/staging/slider/trunk/content/docs/manpage.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/manpage.html (original)
+++ websites/staging/slider/trunk/content/docs/manpage.html Thu Oct 9 00:28:02
2014
@@ -178,14 +178,14 @@ Latest release: <strong>0.40-incubating<
<h2 id="synopsis">SYNOPSIS</h2>
<p>Slider enables applications to be dynamically created on a YARN-managed
datacenter.
The program can be used to create, pause, and shutdown the application. It can
also be used to list current active
-and existing but not running "frozen" application instances.</p>
+and existing but not running "stopped" application instances.</p>
<h2 id="concepts">CONCEPTS</h2>
<ol>
<li>
<p>A <em>Slider application</em> is an application packaged to be deployed by
Slider. It consists of one or more distributed <em>components</em> </p>
</li>
<li>
-<p>A <em>Slider application instance</em> is a slider application configured
to be deployable on a specific YARN cluster, with a specific configuration. An
instance can be <em>live</em> -actually running- or <em>frozen</em>. When
frozen all its configuration details and instance-specific data are preserved
on HDFS.</p>
+<p>A <em>Slider application instance</em> is a slider application configured
to be deployable on a specific YARN cluster, with a specific configuration. An
instance can be <em>live</em> -actually running- or <em>stopped</em>. When
stopped all its configuration details and instance-specific data are preserved
on HDFS.</p>
</li>
<li>
<p>An <em>image</em> is a <em>tar.gz</em> file containing binaries used to
create the application. 1. Images are kept in the HDFS filesystem and
identified by their path names; filesystem permissions can be used to share
images amongst users.</p>
@@ -212,13 +212,13 @@ and existing but not running "frozen" ap
<p>A user can create an application instance.</p>
</li>
<li>
-<p>A live instances can be <em>frozen</em>, saving its final state to its
application instance state directory. All running components are shut down.</p>
+<p>A live instances can be <em>stopped</em>, saving its final state to its
application instance state directory. All running components are shut down.</p>
</li>
<li>
-<p>A frozen instance can be <em>thawed</em> -a its components started on or
near the servers where they were previously running.</p>
+<p>A stopped instance can be <em>started</em> -a its components started on or
near the servers where they were previously running.</p>
</li>
<li>
-<p>A frozen instance can be <em>destroyed</em>. </p>
+<p>A stopped instance can be <em>destroyed</em>. </p>
</li>
<li>
<p>Running instances can be listed. </p>
@@ -367,6 +367,24 @@ allocations, which must use the <code>--
<p>If unset, the zookeeper quorum defined in the property
<code>slider.zookeeper.quorum</code>
is used</p>
+<h5 id="-queue-queue-name"><code>--queue <queue name></code></h5>
+<p>The queue to deploy the application to. By default, YARN will pick the
queue.</p>
+<p>Example</p>
+<div class="codehilite"><pre><span class="o">--</span><span
class="n">queue</span> <span class="n">applications</span>
+</pre></div>
+
+
+<h4 id="examples"><code>Examples</code></h4>
+<p>Create an application by providing <code>template</code> and
<code>resources</code>.</p>
+<div class="codehilite"><pre><span class="n">create</span> <span
class="n">hbase1</span> <span class="o">--</span><span
class="n">template</span> <span class="o">/</span><span
class="n">usr</span><span class="o">/</span><span class="n">work</span><span
class="o">/</span><span class="n">hbase</span><span class="o">/</span><span
class="n">appConfig</span><span class="p">.</span><span class="n">json</span>
<span class="o">--</span><span class="n">resources</span> <span
class="o">/</span><span class="n">usr</span><span class="o">/</span><span
class="n">work</span><span class="o">/</span><span class="n">hbase</span><span
class="o">/</span><span class="n">resources</span><span class="p">.</span><span
class="n">json</span>
+</pre></div>
+
+
+<p>Create an application by providing <code>template</code> and
<code>resources</code> and <code>queue</code>.</p>
+<div class="codehilite"><pre><span class="n">create</span> <span
class="n">hbase1</span> <span class="o">--</span><span
class="n">template</span> <span class="o">/</span><span
class="n">usr</span><span class="o">/</span><span class="n">work</span><span
class="o">/</span><span class="n">hbase</span><span class="o">/</span><span
class="n">appConfig</span><span class="p">.</span><span class="n">json</span>
<span class="o">--</span><span class="n">resources</span> <span
class="o">/</span><span class="n">usr</span><span class="o">/</span><span
class="n">work</span><span class="o">/</span><span class="n">hbase</span><span
class="o">/</span><span class="n">resources</span><span class="p">.</span><span
class="n">json</span> <span class="o">--</span><span class="n">queue</span>
<span class="n">default</span>
+</pre></div>
+
+
<h3 id="destroy-name"><code>destroy <name></code></h3>
<p>Destroy a (stopped) applicaton instance .</p>
<p>Important: This deletes all persistent data</p>
@@ -425,17 +443,17 @@ cluster</p>
</pre></div>
-<h3 id="freeze-name-force-wait-time-message-text"><code>freeze <name>
[--force] [--wait time] [--message text]</code></h3>
-<p>(<strong>freeze</strong> has been renamed to <strong>stop</strong> in
develop branch)</p>
-<p>freeze the application instance. The running application is stopped. Its
settings are retained in HDFS.</p>
-<p>The <code>--wait</code> argument can specify a time in seconds to wait for
the application instance to be frozen.</p>
-<p>The <code>--force</code> flag causes YARN asked directly to terminate the
application instance.
-The <code>--message</code> argument supplies an optional text message to be
used in
-the request: this will appear in the application's diagnostics in the YARN RM
UI.</p>
-<p>If an unknown (or already frozen) application instance is named, no error
is returned.</p>
-<p>Examples</p>
-<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">freeze</span> <span class="n">instance1</span> <span
class="o">--</span><span class="n">wait</span> 30
-<span class="n">slider</span> <span class="n">freeze</span> <span
class="n">instance2</span> <span class="o">--</span><span
class="n">force</span> <span class="o">--</span><span class="n">message</span>
"<span class="n">maintenance</span> <span class="n">session</span>"
+<h3
id="install-package-name-name-of-the-package-package-package-file-replacepkg"><code>install-package
--name <name of the package> --package <package file>
[--replacepkg]</code></h3>
+<p>Install the application package to the default package location for the
user under ~/.slider/package/<name>. This is the location referred to by the
appConfig.json file provided in the --template parameter in the create
command.</p>
+<h5 id="-name-name-of-the-package"><code>--name <name of the
package></code></h5>
+<p>Name of the package. It may be the same as the name provided in the
metainfo.xml. Ensure that the same value is used in the default application
package location specified in the default appConfig.json file.</p>
+<h5 id="-package-package-file"><code>--package <package file></code></h5>
+<p>Location of the package on local disk.</p>
+<h5 id="-replacepkg"><code>--replacepkg</code></h5>
+<p>Optional. Whether to overwrite an already installed package.</p>
+<p>Example</p>
+<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">install</span><span class="o">-</span><span class="n">package</span>
<span class="o">--</span><span class="n">name</span> <span
class="n">HBASE</span> <span class="o">--</span><span class="n">package</span>
<span class="o">/</span><span class="n">usr</span><span class="o">/</span><span
class="n">work</span><span class="o">/</span><span
class="n">package</span><span class="o">/</span><span
class="n">hbase</span><span class="o">/</span><span
class="n">slider</span><span class="o">-</span><span
class="n">hbase</span><span class="o">-</span><span class="n">app</span><span
class="o">-</span><span class="n">package</span><span class="o">-</span>0<span
class="p">.</span>98<span class="p">.</span>4<span class="o">-</span><span
class="n">hadoop2</span><span class="p">.</span><span class="n">zip</span>
+<span class="n">slider</span> <span class="n">install</span><span
class="o">-</span><span class="n">package</span> <span class="o">--</span><span
class="n">name</span> <span class="n">HBASE</span> <span
class="o">--</span><span class="n">package</span> <span class="o">/</span><span
class="n">usr</span><span class="o">/</span><span class="n">work</span><span
class="o">/</span><span class="n">package</span><span class="o">/</span><span
class="n">hbase</span><span class="o">/</span><span
class="n">slider</span><span class="o">-</span><span
class="n">hbase</span><span class="o">-</span><span class="n">app</span><span
class="o">-</span><span class="n">package</span><span class="o">-</span>0<span
class="p">.</span>98<span class="p">.</span>4<span class="o">-</span><span
class="n">hadoop2</span><span class="p">.</span><span class="n">zip</span>
<span class="o">--</span><span class="n">replacepkg</span>
</pre></div>
@@ -490,10 +508,20 @@ is not found, the command returns the ex
<p>List all services of the service type and optionally the name. </p>
<h4
id="slider-registry-listconf-name-name-internal-servicetype-servicetype"><code>slider
registry --listconf [--name <name>] [--internal] [--servicetype
<servicetype>]</code></h4>
<p>List the configurations exported by of a named application</p>
-<h4
id="slider-registry-getconf-configuration-format-xmljsonproperties-dest-path-internal-get-the-configuration"><code>slider
registry --getconf <configuration> [--format (xml|json|properties)]
[--dest <path>] [--internal]</code> get the configuration</h4>
+<h4
id="slider-registry-getconf-configuration-format-xmljsonproperties-dest-path-internal"><code>slider
registry --getconf <configuration> [--format (xml|json|properties)]
[--dest <path>] [--internal]</code></h4>
<p>Get a named configuration in a chosen format. Default: XML</p>
<p><code>--dest <path></code> : the filename or directory to save a
configuration to.
<code>--format (xml|json|properties)</code> defines the output format</p>
+<h3 id="start-name-wait-time"><code>start <name> [--wait
time</code>]</h3>
+<p>(<strong>start</strong> used to be <strong>thaw</strong> in develop
branch)</p>
+<p>Resume a stopped application instance, recreating it from its previously
saved state. This will include a best-effort attempt to create the same number
of nodes as before, though their locations may be different.</p>
+<p>Examples:</p>
+<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">start</span> <span class="n">instance2</span>
+<span class="n">slider</span> <span class="n">start</span> <span
class="n">instance1</span> <span class="o">--</span><span class="n">wait</span>
60
+</pre></div>
+
+
+<p>If the application instance is already running, this command will not
affect it.</p>
<h3 id="status-name-out-filename"><code>status <name> [--out
<filename>]</code></h3>
<p>Get the status of the named application instance in JSON format. A filename
can be used to
specify the destination.</p>
@@ -504,16 +532,20 @@ specify the destination.</p>
</pre></div>
-<h3 id="thaw-name-wait-time"><code>thaw <name> [--wait time</code>]</h3>
-<p>(<strong>thaw</strong> has been renamed to <strong>start</strong> in
develop branch)</p>
-<p>Resume a frozen application instance, recreating it from its previously
saved state. This will include a best-effort attempt to create the same number
of nodes as before, though their locations may be different.</p>
-<p>Examples:</p>
-<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">thaw</span> <span class="n">instance2</span>
-<span class="n">slider</span> <span class="n">thaw</span> <span
class="n">instance1</span> <span class="o">--</span><span class="n">wait</span>
60
+<h3 id="stop-name-force-wait-time-message-text"><code>stop <name>
[--force] [--wait time] [--message text]</code></h3>
+<p>(<strong>stop</strong> used to be <strong>freeze</strong> in earlier
releases)</p>
+<p>stop the application instance. The running application is stopped. Its
settings are retained in HDFS.</p>
+<p>The <code>--wait</code> argument can specify a time in seconds to wait for
the application instance to be stopped.</p>
+<p>The <code>--force</code> flag causes YARN asked directly to terminate the
application instance.
+The <code>--message</code> argument supplies an optional text message to be
used in
+the request: this will appear in the application's diagnostics in the YARN RM
UI.</p>
+<p>If an unknown (or already stopped) application instance is named, no error
is returned.</p>
+<p>Examples</p>
+<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">stop</span> <span class="n">instance1</span> <span
class="o">--</span><span class="n">wait</span> 30
+<span class="n">slider</span> <span class="n">stop</span> <span
class="n">instance2</span> <span class="o">--</span><span
class="n">force</span> <span class="o">--</span><span class="n">message</span>
"<span class="n">maintenance</span> <span class="n">session</span>"
</pre></div>
-<p>If the application instance is already running, this command will not
affect it.</p>
<h3 id="version"><code>version</code></h3>
<p>The command <code>slider version</code> prints out information about the
compiled
Slider application, the version of Hadoop against which it was built -and
Modified:
websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
==============================================================================
---
websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
(original)
+++
websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
Thu Oct 9 00:28:02 2014
@@ -173,8 +173,13 @@ Latest release: <strong>0.40-incubating<
-->
<h1 id="apache-slider-resource-specification">Apache Slider Resource
Specification</h1>
+<ul>
+<li><a href="#failurepolicy">Container Failure Policy</a></li>
+<li><a href="#labels">Using Labels</a></li>
+<li><a href="#logagg">Specifying Log Aggregation</a></li>
+</ul>
<p>Resource specification is an input to Slider to specify the Yarn resource
needs for each component type that belong to the application.</p>
-<p>An example resource requirement for an application that has two components
"master" and "worker" is as follows. Slider will automatically add the
requirements for the AppMaster for the application. This compoent is named
"slider-appmaster".</p>
+<p>An example resource requirement for an application that has two components
"master" and "worker" is as follows. Slider will automatically add the
requirements for the AppMaster for the application. This component is named
"slider-appmaster".</p>
<p>Some parameters that can be specified for a component instance include:</p>
<ul>
<li><code>yarn.memory</code>: amount of memory requried for the component
instance</li>
@@ -209,7 +214,7 @@ Latest release: <strong>0.40-incubating<
</pre></div>
-<h2 id="container-failure-policy">Container Failure Policy</h2>
+<h2 id="wzxhzdk5wzxhzdk6container-failure-policy"><a
name="failurepolicy"></a>Container Failure Policy</h2>
<p>YARN containers hosting component instances may fail. This can happen
because of</p>
<ol>
<li>A problem in the configuration of the instance.</li>
@@ -304,7 +309,7 @@ with the exit code 73, <code>EXIT_DEPLOY
are requested, the failure threshold per hour is double that of the number of
masters.</p>
<p>There are ten worker components requested; the failure threshold for these
components is overridden to be fifteen. This allows all workers to fail and
-the cluster to recover âbut only anothe five failures would be tolerated
+the cluster to recover âbut only another five failures would be tolerated
for the remaining hour.</p>
<p>These failure thresholds are all heuristics. When initially configuring an
application instance, low thresholds reduce the disruption caused by components
@@ -312,6 +317,51 @@ which are frequently failing due to conf
<p>In a production application, large failure thresholds and/or shorter windows
ensures that the application is resilient to transient failures of the
underlying
YARN cluster and hardware.</p>
+<h2 id="wzxhzdk7wzxhzdk8using-labels"><a name="labels"></a>Using Labels</h2>
+<p>The resources.json file can be used to specify the labels to be used when
allocating containers for the components. The details of the YARN Label feature
can be found at <a
href="https://issues.apache.org/jira/browse/YARN-796";>YARN-796</a>.</p>
+<p>In summary:
+<em> Nodes can be assigned one or more labels
+</em> Capacity Queues can be defined with access to one or more labels
+<em> Ensure application components are associated with appropriate label
expressions
+</em> Create the application using specific queue</p>
+<p>This way, you can gurantee that a certain set of nodes are reserved for an
application or for a component within an application.</p>
+<p>Label expression is specified through property "yarn.label.expression".
When no label expression is specified then it is assummed that only non-labeled
nodes are used when allocating containers for component instances.</p>
+<p>If label expression is specified for slider-appmaster then it also becomes
the default label expression for all component. To take advantage of default
label expression leave out the property (see HBASE_REGIONSERVER in the
example). Label expression with empty string ("yarn.label.expression":"") means
nodes without labels. </p>
+<h3 id="example_1">Example</h3>
+<p>Here is a <code>resource.json</code> file for an HBase cluster which uses
labels. The label for the application instance is "hbase1" and the label
expression for the HBASE_MASTER components is "hbase1_master".
HBASE_REGIONSERVER instances will automatically use label "hbase1".
Alternatively, if you specify ("yarn.label.expression":"") for
HBASE_REGIONSERVER then the containers will only be allocated on nodes with no
labels.</p>
+<div class="codehilite"><pre><span class="p">{</span>
+ "<span class="n">schema</span>"<span class="p">:</span>
"<span class="n">http</span><span class="p">:</span><span
class="o">//</span><span class="n">example</span><span class="p">.</span><span
class="n">org</span><span class="o">/</span><span
class="n">specification</span><span class="o">/</span><span
class="n">v2</span><span class="p">.</span>0<span
class="p">.</span>0"<span class="p">,</span>
+ "<span class="n">metadata</span>"<span class="p">:</span> <span
class="p">{</span>
+ <span class="p">},</span>
+ "<span class="k">global</span>"<span class="p">:</span> <span
class="p">{</span>
+ <span class="p">},</span>
+ "<span class="n">components</span>"<span class="p">:</span>
<span class="p">{</span>
+ "<span class="n">HBASE_MASTER</span>"<span
class="p">:</span> <span class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">role</span><span class="p">.</span><span
class="n">priority</span>"<span class="p">:</span> "1"<span
class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">component</span><span class="p">.</span><span
class="n">instances</span>"<span class="p">:</span> "1"<span
class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">label</span><span class="p">.</span><span
class="n">expression</span>"<span class="p">:</span>"<span
class="n">hbase1_master</span>"
+ <span class="p">},</span>
+ "<span class="n">HBASE_REGIONSERVER</span>"<span
class="p">:</span> <span class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">role</span><span class="p">.</span><span
class="n">priority</span>"<span class="p">:</span> "1"<span
class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">component</span><span class="p">.</span><span
class="n">instances</span>"<span class="p">:</span> "1"<span
class="p">,</span>
+ <span class="p">},</span>
+ "<span class="n">slider</span><span class="o">-</span><span
class="n">appmaster</span>"<span class="p">:</span> <span
class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">label</span><span class="p">.</span><span
class="n">expression</span>"<span class="p">:</span>"<span
class="n">hbase1</span>"
+ <span class="p">}</span>
+ <span class="p">}</span>
+<span class="p">}</span>
+</pre></div>
+
+
+<p>Specifically, for the above example you will need:
+<em> Create two labels, <code>hbase1</code> and <code>hbase1_master</code>
(use yarn rmadmin commands)
+</em> Assign the labels to nodes (use yarn rmadmin commands)
+<em> Perform refresh queue (yarn -refreshqueue)
+</em> Create a queue by defining it in the capacity scheduler config
+<em> Allow the queue to access to the labels and ensure that appropriate
min/max capacity is assigned
+</em> Perform refresh queue (yarn -refreshqueue)
+* Create the Slider application against the above queue using parameter
<code>--queue</code> while creating the application</p>
+<h2 id="wzxhzdk9wzxhzdk10using-log-aggregation"><a name="logagg"></a>Using Log
Aggregation</h2>
</div>
<div id="footer">
