Author: buildbot
Date: Mon Oct 20 13:52:59 2014
New Revision: 926209
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/developing/functional_tests.html
websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Mon Oct 20 13:52:59 2014
@@ -1 +1 @@
-1632431
+1633155
Modified: websites/staging/slider/trunk/content/developing/functional_tests.html
==============================================================================
--- websites/staging/slider/trunk/content/developing/functional_tests.html
(original)
+++ websites/staging/slider/trunk/content/developing/functional_tests.html Mon
Oct 20 13:52:59 2014
@@ -183,18 +183,18 @@ Latest release: <strong>0.40-incubating<
<h1 id="functional-tests">Functional Tests</h1>
<p>The functional test suite is designed to test slider against
a live cluster. </p>
-<p>For these to work you need</p>
+<p>For these to work you MUST have </p>
<ol>
<li>A YARN Cluster -secure or insecure</li>
<li>A <code>slider-client.xml</code> file configured to interact with the
cluster</li>
<li>Agent tests and Accumulo Agent tests: nothing additional</li>
-<li>HBase provider tests: HBase <code>.tar.gz</code> uploaded to HDFS, and a
local or remote hbase conf
+<li>(deprecated) HBase provider tests: HBase aa<code>.tar.gz</code> uploaded
to HDFS, and a local or remote hbase conf
directory</li>
-<li>Accumulo provider tests: Accumulo <code>.tar.gz</code> uploaded to HDFS,
and a local or remote accumulo conf
+<li>(deprecated) Accumulo provider tests: Accumulo <code>.tar.gz</code>
uploaded to HDFS, and a local or remote accumulo conf
directory</li>
</ol>
<h2 id="configuration-of-functional-tests">Configuration of functional
tests</h2>
-<p>Maven needs to be given</p>
+<p>Maven MUST have to be given</p>
<ol>
<li>A path to the expanded test archive</li>
<li>A path to a slider configuration directory for the cluster</li>
@@ -217,10 +217,66 @@ If it is not present, the tests will fai
<h2 id="configuration-of-slider-clientxml">Configuration of
<code>slider-client.xml</code></h2>
<p>The <code>slider-client.xml</code> must have extra configuration options
for both the HBase and
Accumulo tests, as well as a common set for actually talking to a YARN
cluster.</p>
+<h3 id="how-to-pick-up-core-sitexml-hdfs-sitexml-and-yarn-sitexml-values">How
to pick up <code>core-site.xml</code>, <code>hdfs-site.xml</code> and
<code>yarn-site.xml</code> values</h3>
+<p>The <code>slider-client.xml</code> file can declare a
<code>HADOOP_CONF_DIR</code> value
+for use by both the test runner AND the command line <code>bin\slider</code>
application
+invoked.</p>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>HADOOP_CONF_DIR<span
class="nt"></name></span>
+ <span
class="nt"><value></span>/home/tester/sites/production/hadoop-conf<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
+
+
+<p>If set:</p>
+<ol>
+<li>The standard <code>-site.xml</code> files are loaded by the JUnit test
runner, to bond
+the test classes to the YARN cluster.</li>
+<li>The property is used to set the environment variable
<code>HADOOP_CONF_DIR</code>
+before the <code>bin/slider</code> or bin\slider.py` script is executed.</li>
+</ol>
+<p><strong>Note 1:</strong> a path can be set relative to
${SLIDER_CONF_DIR}</p>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>HADOOP_CONF_DIR<span
class="nt"></name></span>
+ <span class="nt"><value></span><span class="cp">${</span><span
class="n">SLIDER_CONF_DIR</span><span class="cp">}</span>/../hadoop-conf<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
+
+
+<p>or, on Windows</p>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>HADOOP_CONF_DIR<span
class="nt"></name></span>
+ <span class="nt"><value></span><span class="cp">${</span><span
class="n">SLIDER_CONF_DIR</span><span class="cp">}</span>\..\hadoop-conf<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
+
+
+<p><strong>Note 2:</strong> To test on the local cluster, use either an
absolute path,
+or refer to the environment variable <code>env.HADOOP_CONF_DIR</code>:</p>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>HADOOP_CONF_DIR<span
class="nt"></name></span>
+ <span class="nt"><value></span><span class="cp">${</span><span
class="n">env</span><span class="o">.</span><span
class="n">HADOOP_CONF_DIR</span><span class="cp">}</span><span
class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
+
+
+<p>Be aware that expansion of the <code>env.</code> environment variables and
<code>SLIDER_CONF_DIR</code>
+are only performed on the test runs. If the same
<code>slider-client.xml</code> value
+is used directly from the CLI, the environment variable
<code>HADOOP_CONF_DIR</code>
+must be set to the absolute path of the hadoop configuration directory.</p>
+<h2 id="how-to-validate-the-configuration">How to validate the
configuration</h2>
+<p>A test case executing the <code>slider diagnostics</code> command can be
used
+to print out the configuration as seen by the <code>bin/slider</code>
process</p>
+<div class="codehilite"><pre><span class="n">mvn</span> <span
class="n">integration</span><span class="o">-</span><span class="n">test</span>
<span class="o">\</span>
+ <span class="o">-</span><span class="n">Dslider</span><span
class="p">.</span><span class="n">conf</span><span class="p">.</span><span
class="n">dir</span><span class="p">=</span><span class="o">/</span><span
class="n">home</span><span class="o">/</span><span class="n">tester</span><span
class="o">/</span><span class="n">sites</span><span class="o">/</span><span
class="n">production</span><span class="o">/</span><span
class="n">slider</span> <span class="o">\</span>
+ <span class="o">-</span><span class="n">Dit</span><span
class="p">.</span><span class="n">test</span><span class="p">=</span><span
class="n">DiagnosticsCommandIT</span>
+</pre></div>
+
+
<h2 id="disabling-the-functional-tests-entirely">Disabling the functional
tests entirely</h2>
<p>All functional tests which require a live YARN cluster
-are run during the integration-test phase. They are executed with the command
-<code>mvn verify</code>.</p>
+are run during the integration-test phase. They are executed with the commands
+<code>mvn verify</code> or <code>mvn integration-test</code> .</p>
<p>If you do not wish to run the functional tests, simply use the <code>mvn
package</code> command
and only those tests which do not require a live YARN cluster will run.</p>
<h3 id="non-mandatory-options">Non-mandatory options</h3>
@@ -286,67 +342,20 @@ away</li>
<p><strong>Test setup</strong></p>
-<p>Edit config file src/test/clusters/remote/slider/slider-client.xml and
ensure that the host names are accurate for the test cluster.</p>
+<p>Edit config file
<code>src/test/clusters/remote/slider/slider-client.xml</code> or
+ your chosen equivalent and ensure that the host names are accurate for the
test cluster.</p>
<p><strong>User setup</strong></p>
-<p>Ensure that the user, running the test, is present on the cluster against
which you are running the tests. The user must be a member of the hadoop
group.</p>
-<p>E.g. adduser <strong>testuser</strong> -d /home/<strong>testuser</strong>
-G hadoop -m</p>
-<p><strong>HDFS Setup</strong></p>
-<p>Set up hdfs folders for test user</p>
-<ul>
-<li>su hdfs</li>
-<li>hdfs dfs -mkdir /user/testuser</li>
-<li>hdfs dfs -chown testuser:hdfs /user/testuser</li>
-</ul>
-<h3 id="hbase-tests">HBase Tests</h3>
-<p>The HBase tests can be enabled or disabled</p>
-<div class="codehilite"><pre><span class="nt"><property></span>
- <span class="nt"><name></span>slider.test.hbase.enabled<span
class="nt"></name></span>
- <span class="nt"><description></span>Flag to enable/disable HBase
tests<span class="nt"></description></span>
- <span class="nt"><value></span>true<span
class="nt"></value></span>
-<span class="nt"></property></span>
-</pre></div>
-
-
-<p>Mandatory test parameters must be added to
<code>slider-client.xml</code></p>
-<div class="codehilite"><pre><span class="nt"><property></span>
- <span class="nt"><name></span>slider.test.hbase.tar<span
class="nt"></name></span>
- <span class="nt"><description></span>Path to the HBase Tar file in
HDFS<span class="nt"></description></span>
- <span
class="nt"><value></span>hdfs://sandbox:8020/user/slider/hbase.tar.gz<span
class="nt"></value></span>
-<span class="nt"></property></span>
-
-<span class="nt"><property></span>
- <span class="nt"><name></span>slider.test.hbase.appconf<span
class="nt"></name></span>
- <span class="nt"><description></span>Path to the directory containing
the HBase application config<span class="nt"></description></span>
- <span class="nt"><value></span>file://<span class="cp">${</span><span
class="n">user</span><span class="o">.</span><span class="n">dir</span><span
class="cp">}</span>/src/test/clusters/sandbox/hbase<span
class="nt"></value></span>
-<span class="nt"></property></span>
-</pre></div>
-
-
-<p>Optional parameters: </p>
-<div class="codehilite"><pre> <span class="nt"><property></span>
- <span
class="nt"><name></span>slider.test.hbase.launch.wait.seconds<span
class="nt"></name></span>
- <span class="nt"><description></span>Time to wait in seconds for HBase
to start<span class="nt"></description></span>
- <span class="nt"><value></span>1800<span
class="nt"></value></span>
-<span class="nt"></property></span>
-</pre></div>
-
-
-<h4 id="accumulo-configuration-options">Accumulo configuration options</h4>
-<p>Enable/disable the tests</p>
-<div class="codehilite"><pre> <span class="nt"><property></span>
- <span class="nt"><name></span>slider.test.accumulo.enabled<span
class="nt"></name></span>
- <span class="nt"><description></span>Flag to enable/disable Accumulo
tests<span class="nt"></description></span>
- <span class="nt"><value></span>true<span
class="nt"></value></span>
- <span class="nt"></property></span>
+<p>Ensure that the user running the test, is present on the cluster against
which you are running the tests. The user must be a member of the hadoop
group.</p>
+<p>E.g:</p>
+<div class="codehilite"><pre><span class="n">adduser</span> <span
class="n">testuser</span> <span class="o">-</span><span class="n">d</span>
<span class="o">/</span><span class="n">home</span><span
class="o">/</span><span class="n">testuser</span> <span class="o">-</span><span
class="n">G</span> <span class="n">hadoop</span> <span class="o">-</span><span
class="n">m</span>
</pre></div>
-<p>Optional parameters</p>
-<div class="codehilite"><pre> <span class="nt"><property></span>
- <span
class="nt"><name></span>slider.test.accumulo.launch.wait.seconds<span
class="nt"></name></span>
- <span class="nt"><description></span>Time to wait in seconds for
Accumulo to start<span class="nt"></description></span>
- <span class="nt"><value></span>1800<span
class="nt"></value></span>
- <span class="nt"></property></span>
+<p><strong>HDFS Setup</strong></p>
+<p>Set up hdfs folders for test user</p>
+<div class="codehilite"><pre><span class="n">su</span> <span
class="n">hdfs</span>
+<span class="n">hdfs</span> <span class="n">dfs</span> <span
class="o">-</span><span class="n">mkdir</span> <span class="o">/</span><span
class="n">user</span><span class="o">/</span><span class="n">testuser</span>
+<span class="n">hdfs</span> <span class="n">dfs</span> <span
class="o">-</span><span class="n">chown</span> <span
class="n">testuser</span><span class="p">:</span><span class="n">hdfs</span>
<span class="o">/</span><span class="n">user</span><span
class="o">/</span><span class="n">testuser</span>
</pre></div>
@@ -382,6 +391,18 @@ to keep the log files around after an ap
<span
class="nt"><name></span>yarn.nodemanager.sleep-delay-before-sigkill.ms<span
class="nt"></name></span>
<span class="nt"><value></span>30000<span
class="nt"></value></span>
<span class="nt"></property></span>
+
+
+ <span class="c"><!-- registry--></span>
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>hadoop.registry.rm.enabled<span
class="nt"></name></span>
+ <span class="nt"><value></span>true<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>hadoop.registry.zk.quorum<span
class="nt"></name></span>
+ <span class="nt"><value></span><span class="cp">${</span><span
class="n">hostname</span><span class="cp">}</span>:2181<span
class="nt"></value></span>
+ <span class="nt"></property></span>
</pre></div>
@@ -410,18 +431,18 @@ the location of items in the local files
persistent HDFS cluster.</p>
</li>
<li>
-<p>Use the special property <code>slider.test.conf.dir</code> that is set to
the path
+<p>Use the special property <code>SLIDER_CONF_DIR</code> that is set to the
path
of the directory, and which can then be used to create an absolute path
from paths relative to the configuration dir:</p>
<p><property>
- <name>slider.test.hbase.appconf</name>
- <description>Path to the directory containing the HBase application
config</description>
- <value>file://${slider.test.conf.dir}/../hbase</value>
- </property></p>
+<name>slider.test.hbase.appconf</name>
+<description>Path to the directory containing the HBase application
config</description>
+<value>file://${SLIDER_CONF_DIR}/../hbase</value>
+ </property></p>
</li>
</ol>
-<p>If the actual XML file path is required, a similar property
-<code>slider.test.conf.xml</code> is set.</p>
+<p>If the actual XML file path is required, use the property
+<code>slider.test.conf.xml</code> </p>
<h2 id="parallel-execution">Parallel execution</h2>
<p>Attempts to run test cases in parallel failed -even with a configuration
to run methods in a class sequentially, but separate classes independently.</p>
@@ -537,6 +558,59 @@ as the host OS.</p>
<p>For example, to run all Agent IT tests: </p>
<div class="codehilite"><pre>mvn integration-test -Dslider.conf.dir=<span
class="cp">${</span><span class="n">your</span><span class="o">-</span><span
class="n">config</span><span class="o">-</span><span class="nb">dir</span><span
class="cp">}</span> -Dit.test=Agent\*IT
</pre></div>
+
+
+<h3 id="hbase-tests">HBase Tests</h3>
+<p>The HBase tests can be enabled or disabled</p>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.hbase.enabled<span
class="nt"></name></span>
+ <span class="nt"><description></span>Flag to enable/disable HBase
tests<span class="nt"></description></span>
+ <span class="nt"><value></span>true<span
class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
+
+
+<p>Mandatory test parameters must be added to
<code>slider-client.xml</code></p>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.hbase.tar<span
class="nt"></name></span>
+ <span class="nt"><description></span>Path to the HBase Tar file in
HDFS<span class="nt"></description></span>
+ <span
class="nt"><value></span>hdfs://sandbox:8020/user/slider/hbase.tar.gz<span
class="nt"></value></span>
+<span class="nt"></property></span>
+
+<span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.hbase.appconf<span
class="nt"></name></span>
+ <span class="nt"><description></span>Path to the directory containing
the HBase application config<span class="nt"></description></span>
+ <span class="nt"><value></span>file://<span class="cp">${</span><span
class="err">$</span><span class="p">{</span><span
class="n">SLIDER_CONF_DIR</span><span class="cp">}</span>}/../hbase<span
class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
+
+
+<p>Optional parameters: </p>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span
class="nt"><name></span>slider.test.hbase.launch.wait.seconds<span
class="nt"></name></span>
+ <span class="nt"><description></span>Time to wait in seconds for HBase
to start<span class="nt"></description></span>
+ <span class="nt"><value></span>1800<span
class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
+
+
+<h4 id="accumulo-configuration-options">Accumulo configuration options</h4>
+<p>Enable/disable the tests</p>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.accumulo.enabled<span
class="nt"></name></span>
+ <span class="nt"><description></span>Flag to enable/disable Accumulo
tests<span class="nt"></description></span>
+ <span class="nt"><value></span>true<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
+
+
+<p>Optional parameters</p>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span
class="nt"><name></span>slider.test.accumulo.launch.wait.seconds<span
class="nt"></name></span>
+ <span class="nt"><description></span>Time to wait in seconds for
Accumulo to start<span class="nt"></description></span>
+ <span class="nt"><value></span>1800<span
class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
</div>
<div id="footer">
Modified:
websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
==============================================================================
---
websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
(original)
+++
websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
Mon Oct 20 13:52:59 2014
@@ -214,7 +214,7 @@ Latest release: <strong>0.40-incubating<
</pre></div>
-<h2 id="wzxhzdk4wzxhzdk5container-failure-policy"><a
name="failurepolicy"></a>Container Failure Policy</h2>
+<h2 id="wzxhzdk5wzxhzdk6container-failure-policy"><a
name="failurepolicy"></a>Container Failure Policy</h2>
<p>YARN containers hosting component instances may fail. This can happen
because of</p>
<ol>
<li>A problem in the configuration of the instance.</li>
@@ -317,10 +317,55 @@ which are frequently failing due to conf
<p>In a production application, large failure thresholds and/or shorter windows
ensures that the application is resilient to transient failures of the
underlying
YARN cluster and hardware.</p>
-<h2 id="wzxhzdk6wzxhzdk7using-labels"><a name="labels"></a>Using Labels</h2>
+<h2 id="wzxhzdk7wzxhzdk8using-labels"><a name="labels"></a>Using Labels</h2>
<p>The resources.json file can be used to specify the labels to be used when
allocating containers for the components. The details of the YARN Label feature
can be found at <a
href="https://issues.apache.org/jira/browse/YARN-796";>YARN-796</a>.</p>
-<p>The feature is not yet committed.</p>
-<h2 id="wzxhzdk8wzxhzdk9using-log-aggregation"><a name="logagg"></a>Using Log
Aggregation</h2>
+<p>In summary: </p>
+<ul>
+<li>Nodes can be assigned one or more labels</li>
+<li>Capacity Queues can be defined with access to one or more labels</li>
+<li>Ensure application components are associated with appropriate label
expressions </li>
+<li>Create the application using specific queue</li>
+</ul>
+<p>This way, you can guarantee that a certain set of nodes are reserved for an
application or for a component within an application.</p>
+<p>Label expression is specified through property "yarn.label.expression".
When no label expression is specified then it is assumed that only non-labeled
nodes are used when allocating containers for component instances.</p>
+<p>If label expression is specified for slider-appmaster then it also becomes
the default label expression for all component. To take advantage of default
label expression leave out the property (see HBASE_REGIONSERVER in the
example). Label expression with empty string ("yarn.label.expression":"") means
nodes without labels.</p>
+<p>Example
+Here is a resource.json file for an HBase cluster which uses labels. The label
for the application instance is "hbase1" and the label expression for the
HBASE_MASTER components is "hbase1_master". HBASE_REGIONSERVER instances will
automatically use label "hbase1". Alternatively, if you specify
("yarn.label.expression":"") for HBASE_REGIONSERVER then the containers will
only be allocated on nodes with no labels.</p>
+<div class="codehilite"><pre><span class="p">{</span>
+ "<span class="n">schema</span>"<span class="p">:</span>
"<span class="n">http</span><span class="p">:</span><span
class="o">//</span><span class="n">example</span><span class="p">.</span><span
class="n">org</span><span class="o">/</span><span
class="n">specification</span><span class="o">/</span><span
class="n">v2</span><span class="p">.</span>0<span
class="p">.</span>0"<span class="p">,</span>
+ "<span class="n">metadata</span>"<span class="p">:</span> <span
class="p">{</span>
+ <span class="p">},</span>
+ "<span class="k">global</span>"<span class="p">:</span> <span
class="p">{</span>
+ <span class="p">},</span>
+ "<span class="n">components</span>"<span class="p">:</span>
<span class="p">{</span>
+ "<span class="n">HBASE_MASTER</span>"<span
class="p">:</span> <span class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">role</span><span class="p">.</span><span
class="n">priority</span>"<span class="p">:</span> "1"<span
class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">component</span><span class="p">.</span><span
class="n">instances</span>"<span class="p">:</span> "1"<span
class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">label</span><span class="p">.</span><span
class="n">expression</span>"<span class="p">:</span>"<span
class="n">hbase1_master</span>"
+ <span class="p">},</span>
+ "<span class="n">HBASE_REGIONSERVER</span>"<span
class="p">:</span> <span class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">role</span><span class="p">.</span><span
class="n">priority</span>"<span class="p">:</span> "1"<span
class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">component</span><span class="p">.</span><span
class="n">instances</span>"<span class="p">:</span> "1"<span
class="p">,</span>
+ <span class="p">},</span>
+ "<span class="n">slider</span><span class="o">-</span><span
class="n">appmaster</span>"<span class="p">:</span> <span
class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span
class="n">label</span><span class="p">.</span><span
class="n">expression</span>"<span class="p">:</span>"<span
class="n">hbase1</span>"
+ <span class="p">}</span>
+ <span class="p">}</span>
+<span class="p">}</span>
+</pre></div>
+
+
+<p>Specifically, for the above example you will need: </p>
+<ul>
+<li>Create two labels, hbase1 and hbase1_master (use yarn rmadmin commands)
</li>
+<li>Assign the labels to nodes (use yarn rmadmin commands) </li>
+<li>Perform refresh queue (yarn -refreshqueue) </li>
+<li>Create a queue by defining it in the capacity scheduler config </li>
+<li>Allow the queue to access to the labels and ensure that appropriate
min/max capacity is assigned </li>
+<li>Perform refresh queue (yarn -refreshqueue) </li>
+<li>Create the Slider application against the above queue using parameter
--queue while creating the application</li>
+</ul>
+<h2 id="wzxhzdk9wzxhzdk10using-log-aggregation"><a name="logagg"></a>Using Log
Aggregation</h2>
<p>The feature is not yet committed.</p>
</div>
