Author: buildbot
Date: Tue Sep 2 14:26:24 2014
New Revision: 921023
Log:
Staging update by buildbot for slider
Removed:
websites/staging/slider/trunk/content/docs/slider_specs/why_slider.html
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/docs/getting_started.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Tue Sep 2 14:26:24 2014
@@ -1 +1 @@
-1621871
+1622019
Modified: websites/staging/slider/trunk/content/docs/getting_started.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/getting_started.html (original)
+++ websites/staging/slider/trunk/content/docs/getting_started.html Tue Sep 2
14:26:24 2014
@@ -201,23 +201,26 @@ Latest release: <strong>0.40-incubating<
<p><a href="#installapp">Install, Configure, Start and Verify Sample
Application</a></p>
</li>
</ul>
-<h2 id="wzxhzdk19wzxhzdk20system-requirements"><a name="sysreqs"></a>System
Requirements</h2>
+<h2 id="wzxhzdk20wzxhzdk21system-requirements"><a name="sysreqs"></a>System
Requirements</h2>
<p>The Slider deployment has the following minimum system requirements:</p>
<ul>
<li>
<p>Hadoop 2.4+</p>
</li>
<li>
-<p>Required Services: HDFS, YARN, MapReduce2 and ZooKeeper</p>
+<p>Required Services: HDFS, YARN and ZooKeeper</p>
</li>
<li>
-<p>Oracle JDK 1.7 (64-bit)</p>
+<p>Oracle JDK 1.6 (64-bit)</p>
</li>
<li>
-<p>Python 2.7</p>
+<p>Python 2.6</p>
+</li>
+<li>
+<p>openssl</p>
</li>
</ul>
-<h2 id="wzxhzdk21wzxhzdk22setup-the-cluster"><a name="setup"></a>Setup the
Cluster</h2>
+<h2 id="wzxhzdk22wzxhzdk23setup-the-cluster"><a name="setup"></a>Setup the
Cluster</h2>
<p>After setting up your Hadoop cluster (using Ambari or other means) with the
services listed above, modify your YARN configuration to allow for multiple
containers on a single host. In <code>yarn-site.xml</code> make the following
modifications:</p>
@@ -237,34 +240,35 @@ containers on a single host. In <code>ya
</table>
<p>There are other options detailed in the Troubleshooting file available <a
href="/docs/troubleshooting.html">here</a>.</p>
-<h2 id="wzxhzdk23wzxhzdk24download-slider-packages"><a
name="download"></a>Download Slider Packages</h2>
+<h2 id="wzxhzdk24wzxhzdk25download-slider-packages"><a
name="download"></a>Download Slider Packages</h2>
<p>Slider releases are available at
<a
href="https://www.apache.org/dyn/closer.cgi/incubator/slider";>https://www.apache.org/dyn/closer.cgi/incubator/slider</a>.</p>
<p><em>Or, you can build it as described below.</em></p>
-<h2 id="wzxhzdk25wzxhzdk26build-slider"><a name="build"></a>Build Slider</h2>
+<h2 id="wzxhzdk26wzxhzdk27build-slider"><a name="build"></a>Build Slider</h2>
<ul>
<li>From the top level directory, execute <code>mvn clean site:site site:stage
package -DskipTests</code></li>
-<li>Use the generated compressed tar file in slider-assembly/target directory
(e.g. slider-0.30.0-all.tar.gz) for the subsequent steps</li>
+<li>Use the generated compressed tar file in slider-assembly/target directory
(e.g. slider-0.40.0-all.tar.gz or slider-0.40.0-all.zip) for the subsequent
steps</li>
+<li>If you are cloning the Slider git repo, go to <code>master</code> branch
for the latest released or <code>develop</code> for the latest under
development</li>
</ul>
-<h2 id="wzxhzdk27wzxhzdk28install-slider"><a name="install"></a>Install
Slider</h2>
+<h2 id="wzxhzdk28wzxhzdk29install-slider"><a name="install"></a>Install
Slider</h2>
<p>Slider is installed on a client machine that can access the hadoop cluster.
Follow the following steps to expand/install Slider:</p>
-<div class="codehilite"><pre>mkdir <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="o">*</span><span class="p">;</span>
+<div class="codehilite"><pre>mkdir <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>
-<span class="n">cd</span> <span class="err">$</span><span
class="p">{</span><span class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>
+cd <span class="cp">${</span><span class="n">slider</span><span
class="o">-</span><span class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>
</pre></div>
<p>Login as the "yarn" user (assuming this is a host associated with the
installed cluster). E.g., <code>su yarn</code>
<em>This assumes that all apps are being run as âyarnâ user. Any other
user can be used to run the apps - ensure that file permission is granted as
required.</em></p>
-<p>Expand the tar file: <code>tar -xvf slider-0.30.0-all.tar.gz</code></p>
-<p>Browse to the Slider directory: <code>cd slider-0.30.0/bin</code></p>
+<p>Expand the tar file: <code>tar -xvf
slider-0.40.0-incubating-all.tar.gz</code> or <code>unzip
slider-0.40.0-incubating-all.zip</code></p>
+<p>Browse to the Slider directory: <code>cd slider-0.40.0/bin</code></p>
<p><em>Add java to the PATH.</em></p>
<div class="codehilite"><pre> <span class="n">export</span> <span
class="n">PATH</span><span class="p">=</span>$<span class="n">PATH</span><span
class="p">:</span><span class="o">/</span><span class="n">usr</span><span
class="o">/</span><span class="n">jdk64</span><span class="o">/</span><span
class="n">jdk1</span><span class="p">.</span>7<span class="p">.</span>0<span
class="n">_45</span><span class="o">/</span><span class="n">bin</span>
</pre></div>
<p>(or the path to the JDK bin directory)</p>
-<p>Modify Slider configuration file
<code>${slider-install-dir}/slider-0.30.0/conf/slider-client.xml</code> to add
the following properties:</p>
+<p>Modify Slider configuration file
<code>${slider-install-dir}/slider-0.40.0/conf/slider-client.xml</code> to add
the following properties:</p>
<div class="codehilite"><pre> <span class="nt"><property></span>
<span class="nt"><name></span>yarn.application.classpath<span
class="nt"></name></span>
<span
class="nt"><value></span>/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*<span
class="nt"></value></span>
@@ -290,16 +294,28 @@ containers on a single host. In <code>ya
</pre></div>
+<p>If the target cluster is on Windows then make sure that the
<code>yarn.application.classpath</code> is set accordingly.</p>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>yarn.application.classpath<span
class="nt"></name></span>
+ <span class="nt"><value></span>
+
%HADOOP_CONF_DIR%,%HADOOP_COMMON_HOME%/share/hadoop/common/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/*,...
+ <span class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
+
+
<p>Execute:</p>
-<div class="codehilite"><pre><span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>/slider-0.30.0/bin/slider version
+<div class="codehilite"><pre><span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>/slider-0.40.0/bin/slider version
+OR
+python %slider-install-dir%/slider-0.40.0/bin/slider.py version
</pre></div>
<p>Ensure there are no errors and you can see "Compiled against Hadoop 2.4.0".
<strong>This ensures that Slider is installed correctly.</strong></p>
-<h2 id="wzxhzdk29wzxhzdk30deploy-slider-resources"><a name="deploy"></a>Deploy
Slider Resources</h2>
-<p>Ensure that all file folders are accessible to the user creating the
application instance. The example assumes "yarn" to be that user. The HDFS
folder "/slider" is a suggestion and the default application packages and
associated configurations assume this path while accessing the slider-agent
tarball.</p>
-<h3 id="create-hdfs-root-folder-for-slider">Create HDFS root folder for
Slider</h3>
-<p>Perform the following steps to create the Slider root folder with the
appropriate permissions:</p>
+<h2 id="wzxhzdk30wzxhzdk31deploy-slider-resources"><a name="deploy"></a>Deploy
Slider Resources</h2>
+<p>Ensure that all file folders are accessible to the user creating the
application instance. The example assumes "yarn" to be that user. The default
yarn user may be different e.g. hadoop.</p>
+<h3 id="ensure-hdfs-home-folder-for-the-user">Ensure HDFS home folder for the
User</h3>
+<p>Perform the following steps to create the user home folder with the
appropriate permissions:</p>
<div class="codehilite"><pre><span class="n">su</span> <span
class="n">hdfs</span>
<span class="n">hdfs</span> <span class="n">dfs</span> <span
class="o">-</span><span class="n">mkdir</span> <span class="o">/</span><span
class="n">user</span><span class="o">/</span><span class="n">yarn</span>
@@ -313,16 +329,18 @@ containers on a single host. In <code>ya
hdfs dfs -mkdir /user/yarn/agent
-hdfs dfs -copyFromLocal <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span
class="cp">}</span>/slider-0.30.0/agent/slider-agent.tar.gz /user/yarn/agent
+hdfs dfs -copyFromLocal <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span
class="cp">}</span>/slider-0.40.0/agent/slider-agent.tar.gz /user/yarn/agent
</pre></div>
-<h2 id="wzxhzdk31wzxhzdk32create-application-packages"><a
name="downsample"></a>Create Application Packages</h2>
-<p>Few pre-created application packages are available at: TBD</p>
-<p>Slider "master" branch has three sample application packages available for
use with Slider:</p>
+<h2 id="wzxhzdk32wzxhzdk33create-application-packages"><a
name="downsample"></a>Create Application Packages</h2>
+<p>Slider "master" branch has several sample application packages available
for use with Slider:</p>
<ul>
<li>
-<p>app-packages/hbase: The README.txt file describes how to create a Slider
Application Package for HBase. By default it will create a package for HBase
0.98.3 but you can create the same for other versions.</p>
+<p>app-packages/memcached-win or memcached: The README.txt describes how to
create the Slider Application package for memcached for linux or windows.
<strong>memcached</strong> app packages are a good place to <a
href="slider_specs/hello_world_slider_app.html">start</a></p>
+</li>
+<li>
+<p>app-packages/hbase or hbase-win: The README.txt file describes how to
create a Slider Application Package for HBase. By default it will create a
package for HBase 0.98.3 but you can create the same for other versions.</p>
</li>
<li>
<p>app-packages/accumulo: The README.txt file describes how to create a Slider
Application Package for Accumulo.</p>
@@ -332,7 +350,7 @@ hdfs dfs -copyFromLocal <span class="cp"
</li>
</ul>
<p>Create one or more Slider application packages and follow the steps below
to install them.</p>
-<h2
id="wzxhzdk33wzxhzdk34install-configure-start-and-verify-sample-application"><a
name="installapp"></a>Install, Configure, Start and Verify Sample
Application</h2>
+<h2
id="wzxhzdk34wzxhzdk35install-configure-start-and-verify-sample-application"><a
name="installapp"></a>Install, Configure, Start and Verify Sample
Application</h2>
<ul>
<li>
<p><a href="#load">Load Sample Application Package</a></p>
@@ -350,29 +368,33 @@ hdfs dfs -copyFromLocal <span class="cp"
<p><a href="#manage">Manage the Application Lifecycle</a></p>
</li>
</ul>
-<h3 id="wzxhzdk35wzxhzdk36load-sample-application-package"><a
name="load"></a>Load Sample Application Package</h3>
+<h3 id="wzxhzdk36wzxhzdk37load-sample-application-package"><a
name="load"></a>Load Sample Application Package</h3>
<div class="codehilite"><pre><span class="n">hdfs</span> <span
class="n">dfs</span> <span class="o">-</span><span
class="n">copyFromLocal</span> <span class="o">*</span><span
class="n">sample</span><span class="o">-</span><span
class="n">application</span><span class="o">-</span><span
class="n">package</span><span class="o">*</span> <span class="o">/</span><span
class="n">user</span><span class="o">/</span><span class="n">yarn</span>
</pre></div>
-<h3 id="wzxhzdk37wzxhzdk38create-application-specifications"><a
name="create"></a>Create Application Specifications</h3>
+<h3 id="wzxhzdk38wzxhzdk39create-application-specifications"><a
name="create"></a>Create Application Specifications</h3>
<p>Configuring a Slider application consists of two parts: the <a
href="#resspec">Resource Specification</a>,
and the *<a href="#appconfig">Application Configuration</a>.</p>
<p><strong>Note</strong>: There are sample Resource Specifications
(<strong>resources.json</strong>) and Application Configuration
-(<strong>appConfig.json</strong>) files in application packages (e.g.
/<strong>resources.json</strong> and /<strong>appConfig.json</strong>).</p>
-<h4 id="wzxhzdk39wzxhzdk40resource-specification"><a
name="resspec"></a>Resource Specification</h4>
+(<strong>appConfig.json</strong>) files in the application packages zip files
(e.g. /<strong>resources.json</strong> and
/<strong>appConfig.json</strong>).</p>
+<h4 id="wzxhzdk40wzxhzdk41resource-specification"><a
name="resspec"></a>Resource Specification</h4>
<p>Slider needs to know what components (and how many components) are in an
application package to deploy.</p>
<p>As Slider creates each instance of a component in its own YARN container,
it also needs to know what to ask YARN for in terms of <strong>memory</strong>
and <strong>CPU</strong> for those containers. </p>
<p>All this information goes into the <strong>Resources Specification</strong>
file ("Resource Spec") named <code>resources.json</code>. The Resource Spec
tells Slider how many instances of each component in the application (such as
an HBase RegionServer) to deploy and the parameters for YARN.</p>
<p>An application package should contain the default resources.json and you
can start from there. Or you can create one based on <a
href="slider_specs/resource_specification.html">Resource Specification</a>.</p>
<p>Store the Resource Spec file on your local disk (e.g.
<code>/tmp/resources.json</code>).</p>
-<h4 id="wzxhzdk41wzxhzdk42application-configuration"><a
name="appconfig"></a>Application Configuration</h4>
+<h4 id="wzxhzdk42wzxhzdk43application-configuration"><a
name="appconfig"></a>Application Configuration</h4>
<p>Alongside the Resource Spec there is the <strong>Application
Configuration</strong> file ("App Config") which includes parameters that are
specific to the application, rather than YARN. The App Config is a file that
contains all application configuration. This configuration is applied to the
default configuration provided by the application definition and then handed
off to the associated component agent.</p>
<p>For example, the heap sizes of the JVMs, The App Config defines the
configuration details <strong>specific to the application and
component</strong> instances. An application package should contain the default
resources.json and you can start from there.</p>
-<h3 id="wzxhzdk43wzxhzdk44start-the-application"><a name="start"></a>Start the
Application</h3>
+<p>Ensure the following variables are accurate:</p>
+<ul>
+<li>"application.def": "application_package.zip" (path to uploaded application
package)</li>
+</ul>
+<h3 id="wzxhzdk44wzxhzdk45start-the-application"><a name="start"></a>Start the
Application</h3>
<p>Once the steps above are completed, the application can be started through
the <strong>Slider Command Line Interface (CLI)</strong>.</p>
<p>Change directory to the "bin" directory under the slider installation</p>
-<div class="codehilite"><pre>cd <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>/slider-0.30.0/bin
+<div class="codehilite"><pre>cd <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>/slider-0.40.0/bin
</pre></div>
@@ -381,17 +403,17 @@ hdfs dfs -copyFromLocal <span class="cp"
</pre></div>
-<h3 id="wzxhzdk45wzxhzdk46verify-the-application"><a name="verify"></a>Verify
the Application</h3>
+<h3 id="wzxhzdk46wzxhzdk47verify-the-application"><a name="verify"></a>Verify
the Application</h3>
<p>The successful launch of the application can be verified via the YARN
Resource Manager Web UI. In most instances, this UI is accessible via a web
browser at port 8088 of the Resource Manager Host:</p>
<p><img alt="image alt text" src="../images/image_1_apps_rm_ui-lrg.png" /></p>
<p>The specific information for the running application is accessible via the
"ApplicationMaster" link that can be seen in the far right column of the row
associated with the running application (probably the top row):</p>
<p><img alt="image alt text" src="../images/image_0_app_master_ui_lrg.png"
/></p>
-<h3 id="wzxhzdk47wzxhzdk48obtain-client-config"><a
name="client-config"></a>Obtain client config</h3>
+<h3 id="wzxhzdk48wzxhzdk49obtain-client-config"><a
name="client-config"></a>Obtain client config</h3>
<p>An application publishes several useful details that can be used to manage
and communicate with the application instance.</p>
<p><strong>All published data</strong></p>
<p>All published data are available at the
<code>/ws/v1/slider/publisher</code> end point of the AppMaster (e.g.
http://c6403.ambari.apache.org:34701/ws/v1/slider/publisher). Its obtainable
through <code>slider-client status <app name></code> where field
<code>info.am.web.url</code> specifies the base address. The URL is also
advertised in the AppMaster tracking UI.</p>
<p><strong>Client configuration</strong>
-Client configurations are at <code>/ws/v1/slider/publisher/slider\<config
name></code> where <em>config name</em> can be </p>
+Client configurations are at <code>/ws/v1/slider/publisher/slider/<config
name></code> where <em>config name</em> can be </p>
<ul>
<li>site config file name without extension, such as <code>hbase-site</code>
(e.g.
http://c6403.ambari.apache.org:34701/ws/v1/slider/publisher/slider/hbase-site)
where the output is json formatted name-value pairs</li>
<li>or, with extension such as <code>hbase-site.xml</code>
(http://c6403.ambari.apache.org:34701/ws/v1/slider/publisher/slider/hbase-site.xml)
to get an XML formatted output that can be consumed by the clients</li>
@@ -415,14 +437,16 @@ Client configurations are at <code>/ws/v
<p><strong>Quick links</strong></p>
<p>An application may publish some quick links for the monitoring UI, JMX
endpoint, etc. as the listening ports are dynamically allocated and thus need
to be published. These information is available at
<code>/ws/v1/slider/publisher/slider/quicklinks</code>.</p>
-<h3 id="wzxhzdk49wzxhzdk50manage-the-application-lifecycle"><a
name="manage"></a>Manage the Application Lifecycle</h3>
+<h3 id="wzxhzdk50wzxhzdk51manage-the-application-lifecycle"><a
name="manage"></a>Manage the Application Lifecycle</h3>
<p>Once started, applications can be frozen/stopped, thawed/restarted, and
destroyed/removed as follows:</p>
<h4 id="frozen">Frozen:</h4>
+<p>(<em>on develop branch freeze has been renamed to stop</em>)</p>
<div class="codehilite"><pre><span class="o">./</span><span
class="n">slider</span> <span class="n">freeze</span> <span class="n">cl1</span>
</pre></div>
<h4 id="thawed">Thawed:</h4>
+<p>(<em>on develop branch thaw has been renamed to start</em>)</p>
<div class="codehilite"><pre><span class="o">./</span><span
class="n">slider</span> <span class="n">thaw</span> <span class="n">cl1</span>
</pre></div>
@@ -437,10 +461,10 @@ Client configurations are at <code>/ws/v
</pre></div>
-<h1 id="wzxhzdk51wzxhzdk52appendix-a-debugging-slider-agent"><a
name="appendixa"></a>Appendix A: Debugging Slider-Agent</h1>
+<h1 id="wzxhzdk52wzxhzdk53appendix-a-debugging-slider-agent"><a
name="appendixa"></a>Appendix A: Debugging Slider-Agent</h1>
<h2 id="create-and-deploy-slider-agent-configuration">Create and deploy Slider
Agent configuration</h2>
<p>Create an agent config file (agent.ini) based on the sample available
at:</p>
-<div class="codehilite"><pre><span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span
class="cp">}</span>/slider-0.30.0/agent/conf/agent.ini
+<div class="codehilite"><pre><span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span
class="cp">}</span>/slider-0.40.0/agent/conf/agent.ini
</pre></div>
