Author: buildbot
Date: Wed Aug 27 22:08:00 2014
New Revision: 920561
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/docs/getting_started.html
websites/staging/slider/trunk/content/docs/slider_specs/hello_world_slider_app.html
websites/staging/slider/trunk/content/docs/slider_specs/index.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Wed Aug 27 22:08:00 2014
@@ -1 +1 @@
-1620412
+1620986
Modified: websites/staging/slider/trunk/content/docs/getting_started.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/getting_started.html (original)
+++ websites/staging/slider/trunk/content/docs/getting_started.html Wed Aug 27
22:08:00 2014
@@ -201,7 +201,7 @@ Latest release: <strong>0.40-incubating<
<p><a href="#installapp">Install, Configure, Start and Verify Sample
Application</a></p>
</li>
</ul>
-<h2 id="wzxhzdk23wzxhzdk24system-requirements"><a name="sysreqs"></a>System
Requirements</h2>
+<h2 id="wzxhzdk19wzxhzdk20system-requirements"><a name="sysreqs"></a>System
Requirements</h2>
<p>The Slider deployment has the following minimum system requirements:</p>
<ul>
<li>
@@ -217,7 +217,7 @@ Latest release: <strong>0.40-incubating<
<p>Python 2.7</p>
</li>
</ul>
-<h2 id="wzxhzdk25wzxhzdk26setup-the-cluster"><a name="setup"></a>Setup the
Cluster</h2>
+<h2 id="wzxhzdk21wzxhzdk22setup-the-cluster"><a name="setup"></a>Setup the
Cluster</h2>
<p>After setting up your Hadoop cluster (using Ambari or other means) with the
services listed above, modify your YARN configuration to allow for multiple
containers on a single host. In <code>yarn-site.xml</code> make the following
modifications:</p>
@@ -237,16 +237,16 @@ containers on a single host. In <code>ya
</table>
<p>There are other options detailed in the Troubleshooting file available <a
href="/docs/troubleshooting.html">here</a>.</p>
-<h2 id="wzxhzdk27wzxhzdk28download-slider-packages"><a
name="download"></a>Download Slider Packages</h2>
+<h2 id="wzxhzdk23wzxhzdk24download-slider-packages"><a
name="download"></a>Download Slider Packages</h2>
<p>Slider releases are available at
<a
href="https://www.apache.org/dyn/closer.cgi/incubator/slider";>https://www.apache.org/dyn/closer.cgi/incubator/slider</a>.</p>
<p><em>Or, you can build it as described below.</em></p>
-<h2 id="wzxhzdk29wzxhzdk30build-slider"><a name="build"></a>Build Slider</h2>
+<h2 id="wzxhzdk25wzxhzdk26build-slider"><a name="build"></a>Build Slider</h2>
<ul>
<li>From the top level directory, execute <code>mvn clean site:site site:stage
package -DskipTests</code></li>
<li>Use the generated compressed tar file in slider-assembly/target directory
(e.g. slider-0.30.0-all.tar.gz) for the subsequent steps</li>
</ul>
-<h2 id="wzxhzdk31wzxhzdk32install-slider"><a name="install"></a>Install
Slider</h2>
+<h2 id="wzxhzdk27wzxhzdk28install-slider"><a name="install"></a>Install
Slider</h2>
<p>Slider is installed on a client machine that can access the hadoop cluster.
Follow the following steps to expand/install Slider:</p>
<div class="codehilite"><pre>mkdir <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="o">*</span><span class="p">;</span>
@@ -266,19 +266,16 @@ containers on a single host. In <code>ya
<p>(or the path to the JDK bin directory)</p>
<p>Modify Slider configuration file
<code>${slider-install-dir}/slider-0.30.0/conf/slider-client.xml</code> to add
the following properties:</p>
<div class="codehilite"><pre> <span class="nt"><property></span>
- <span class="nt"><name></span>yarn.application.classpath<span
class="nt"></name></span>
- <span
class="nt"><value></span>/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*<span
class="nt"></value></span>
+ <span class="nt"><name></span>yarn.application.classpath<span
class="nt"></name></span>
+ <span
class="nt"><value></span>/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*<span
class="nt"></value></span>
<span class="nt"></property></span>
<span class="nt"><property></span>
- <span class="nt"><name></span>slider.zookeeper.quorum<span
class="nt"></name></span>
- <span class="nt"><value></span>yourZooKeeperHost:port<span
class="nt"></value></span>
+ <span class="nt"><name></span>slider.zookeeper.quorum<span
class="nt"></name></span>
+ <span class="nt"><value></span>yourZooKeeperHost:port<span
class="nt"></value></span>
<span class="nt"></property></span>
-</pre></div>
-
-<p>In addition, specify the scheduler and HDFS addresses as follows:</p>
-<div class="codehilite"><pre><span class="nt"><property></span>
+<span class="nt"><property></span>
<span class="nt"><name></span>yarn.resourcemanager.address<span
class="nt"></name></span>
<span class="nt"><value></span>yourResourceManagerHost:8050<span
class="nt"></value></span>
<span class="nt"></property></span>
@@ -299,16 +296,12 @@ containers on a single host. In <code>ya
<p>Ensure there are no errors and you can see "Compiled against Hadoop 2.4.0".
<strong>This ensures that Slider is installed correctly.</strong></p>
-<h2 id="wzxhzdk33wzxhzdk34deploy-slider-resources"><a name="deploy"></a>Deploy
Slider Resources</h2>
+<h2 id="wzxhzdk29wzxhzdk30deploy-slider-resources"><a name="deploy"></a>Deploy
Slider Resources</h2>
<p>Ensure that all file folders are accessible to the user creating the
application instance. The example assumes "yarn" to be that user. The HDFS
folder "/slider" is a suggestion and the default application packages and
associated configurations assume this path while accessing the slider-agent
tarball.</p>
<h3 id="create-hdfs-root-folder-for-slider">Create HDFS root folder for
Slider</h3>
<p>Perform the following steps to create the Slider root folder with the
appropriate permissions:</p>
<div class="codehilite"><pre><span class="n">su</span> <span
class="n">hdfs</span>
-<span class="n">hdfs</span> <span class="n">dfs</span> <span
class="o">-</span><span class="n">mkdir</span> <span class="o">/</span><span
class="n">slider</span>
-
-<span class="n">hdfs</span> <span class="n">dfs</span> <span
class="o">-</span><span class="n">chown</span> <span class="n">yarn</span><span
class="p">:</span><span class="n">hdfs</span> <span class="o">/</span><span
class="n">slider</span>
-
<span class="n">hdfs</span> <span class="n">dfs</span> <span
class="o">-</span><span class="n">mkdir</span> <span class="o">/</span><span
class="n">user</span><span class="o">/</span><span class="n">yarn</span>
<span class="n">hdfs</span> <span class="n">dfs</span> <span
class="o">-</span><span class="n">chown</span> <span class="n">yarn</span><span
class="p">:</span><span class="n">hdfs</span> <span class="o">/</span><span
class="n">user</span><span class="o">/</span><span class="n">yarn</span>
@@ -318,13 +311,13 @@ containers on a single host. In <code>ya
<h3 id="load-slider-agent">Load Slider Agent</h3>
<div class="codehilite"><pre>su yarn
-hdfs dfs -mkdir /slider/agent
+hdfs dfs -mkdir /user/yarn/agent
-hdfs dfs -copyFromLocal <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span
class="cp">}</span>/slider-0.30.0/agent/slider-agent.tar.gz /slider/agent
+hdfs dfs -copyFromLocal <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span
class="cp">}</span>/slider-0.30.0/agent/slider-agent.tar.gz /user/yarn/agent
</pre></div>
-<h2 id="wzxhzdk35wzxhzdk36create-application-packages"><a
name="downsample"></a>Create Application Packages</h2>
+<h2 id="wzxhzdk31wzxhzdk32create-application-packages"><a
name="downsample"></a>Create Application Packages</h2>
<p>Few pre-created application packages are available at: TBD</p>
<p>Slider "master" branch has three sample application packages available for
use with Slider:</p>
<ul>
@@ -339,7 +332,7 @@ hdfs dfs -copyFromLocal <span class="cp"
</li>
</ul>
<p>Create one or more Slider application packages and follow the steps below
to install them.</p>
-<h2
id="wzxhzdk37wzxhzdk38install-configure-start-and-verify-sample-application"><a
name="installapp"></a>Install, Configure, Start and Verify Sample
Application</h2>
+<h2
id="wzxhzdk33wzxhzdk34install-configure-start-and-verify-sample-application"><a
name="installapp"></a>Install, Configure, Start and Verify Sample
Application</h2>
<ul>
<li>
<p><a href="#load">Load Sample Application Package</a></p>
@@ -357,28 +350,26 @@ hdfs dfs -copyFromLocal <span class="cp"
<p><a href="#manage">Manage the Application Lifecycle</a></p>
</li>
</ul>
-<h3 id="wzxhzdk39wzxhzdk40load-sample-application-package"><a
name="load"></a>Load Sample Application Package</h3>
-<div class="codehilite"><pre><span class="n">hdfs</span> <span
class="n">dfs</span> <span class="o">-</span><span
class="n">copyFromLocal</span> <span class="o">*</span><span
class="n">sample</span><span class="o">-</span><span
class="n">application</span><span class="o">-</span><span
class="n">package</span><span class="o">*</span> <span class="o">/</span><span
class="n">slider</span>
+<h3 id="wzxhzdk35wzxhzdk36load-sample-application-package"><a
name="load"></a>Load Sample Application Package</h3>
+<div class="codehilite"><pre><span class="n">hdfs</span> <span
class="n">dfs</span> <span class="o">-</span><span
class="n">copyFromLocal</span> <span class="o">*</span><span
class="n">sample</span><span class="o">-</span><span
class="n">application</span><span class="o">-</span><span
class="n">package</span><span class="o">*</span> <span class="o">/</span><span
class="n">user</span><span class="o">/</span><span class="n">yarn</span>
</pre></div>
-<h3 id="wzxhzdk41wzxhzdk42create-application-specifications"><a
name="create"></a>Create Application Specifications</h3>
+<h3 id="wzxhzdk37wzxhzdk38create-application-specifications"><a
name="create"></a>Create Application Specifications</h3>
<p>Configuring a Slider application consists of two parts: the <a
href="#resspec">Resource Specification</a>,
- and the *<a href="#appconfig">Application Configuration</a>. Below are
guidelines for creating these files.</p>
-<p><em>Note: There are sample Resource Specifications
(<strong>resources.json</strong>) and Application Configuration
-(<strong>appConfig.json</strong>) files in the root directory of the
-Sample Applications packages (e.g. /<strong>resources.json</strong> and
/<strong>appConfig.json</strong>). The Slider git repo also contains sample
appConfig.json and resources.json in the application packages directory (e.g.
app-packages/hbase/appConfig.json)</em></p>
-<h4 id="wzxhzdk43wzxhzdk44resource-specification"><a
name="resspec"></a>Resource Specification</h4>
-<p>Slider needs to know what components (and how many components) are in an
application package to deploy. For example, in HBase, the components are
<strong><em>master</em></strong> and <strong><em>worker</em></strong> -- the
latter hosting <strong>HBase RegionServers</strong>, and the former hosting the
<strong>HBase Master</strong>. </p>
+ and the *<a href="#appconfig">Application Configuration</a>.</p>
+<p><strong>Note</strong>: There are sample Resource Specifications
(<strong>resources.json</strong>) and Application Configuration
+(<strong>appConfig.json</strong>) files in application packages (e.g.
/<strong>resources.json</strong> and /<strong>appConfig.json</strong>).</p>
+<h4 id="wzxhzdk39wzxhzdk40resource-specification"><a
name="resspec"></a>Resource Specification</h4>
+<p>Slider needs to know what components (and how many components) are in an
application package to deploy.</p>
<p>As Slider creates each instance of a component in its own YARN container,
it also needs to know what to ask YARN for in terms of <strong>memory</strong>
and <strong>CPU</strong> for those containers. </p>
<p>All this information goes into the <strong>Resources Specification</strong>
file ("Resource Spec") named <code>resources.json</code>. The Resource Spec
tells Slider how many instances of each component in the application (such as
an HBase RegionServer) to deploy and the parameters for YARN.</p>
+<p>An application package should contain the default resources.json and you
can start from there. Or you can create one based on <a
href="slider_specs/resource_specification.html">Resource Specification</a>.</p>
<p>Store the Resource Spec file on your local disk (e.g.
<code>/tmp/resources.json</code>).</p>
-<p><a href="#appendixb">Appendix B</a> provides some sample resources.json
files.</p>
-<h4 id="wzxhzdk45wzxhzdk46application-configuration"><a
name="appconfig"></a>Application Configuration</h4>
+<h4 id="wzxhzdk41wzxhzdk42application-configuration"><a
name="appconfig"></a>Application Configuration</h4>
<p>Alongside the Resource Spec there is the <strong>Application
Configuration</strong> file ("App Config") which includes parameters that are
specific to the application, rather than YARN. The App Config is a file that
contains all application configuration. This configuration is applied to the
default configuration provided by the application definition and then handed
off to the associated component agent.</p>
-<p>For example, the heap sizes of the JVMs, The App Config defines the
configuration details <strong>specific to the application and
component</strong> instances. For HBase, this includes any values for the
<em>to-be-generated </em>hbase-site.xml file, as well as options for individual
components, such as their heap size.</p>
-<p><a href="#appendixc">Appendix C</a> provides a sample appConfig.json
file.</p>
-<h3 id="wzxhzdk47wzxhzdk48start-the-application"><a name="start"></a>Start the
Application</h3>
+<p>For example, the heap sizes of the JVMs, The App Config defines the
configuration details <strong>specific to the application and
component</strong> instances. An application package should contain the default
resources.json and you can start from there.</p>
+<h3 id="wzxhzdk43wzxhzdk44start-the-application"><a name="start"></a>Start the
Application</h3>
<p>Once the steps above are completed, the application can be started through
the <strong>Slider Command Line Interface (CLI)</strong>.</p>
<p>Change directory to the "bin" directory under the slider installation</p>
<div class="codehilite"><pre>cd <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>/slider-0.30.0/bin
@@ -386,16 +377,16 @@ Sample Applications packages (e.g. /<str
<p>Execute the following command:</p>
-<div class="codehilite"><pre><span class="o">./</span><span
class="n">slider</span> <span class="n">create</span> <span
class="n">cl1</span> <span class="o">--</span><span class="n">image</span>
<span class="n">hdfs</span><span class="p">:</span><span
class="o">//</span><span class="n">yourNameNodeHost</span><span
class="p">:</span>8020<span class="o">/</span><span
class="n">slider</span><span class="o">/</span><span
class="n">agent</span><span class="o">/</span><span
class="n">slider</span><span class="o">-</span><span
class="n">agent</span><span class="p">.</span><span class="n">tar</span><span
class="p">.</span><span class="n">gz</span> <span class="o">--</span><span
class="n">template</span> <span class="n">appConfig</span><span
class="p">.</span><span class="n">json</span> <span class="o">--</span><span
class="n">resources</span> <span class="n">resources</span><span
class="p">.</span><span class="n">json</span>
+<div class="codehilite"><pre><span class="o">./</span><span
class="n">slider</span> <span class="n">create</span> <span
class="n">cl1</span> <span class="o">--</span><span class="n">image</span>
<span class="n">hdfs</span><span class="p">:</span><span
class="o">//</span><span class="n">yourNameNodeHost</span><span
class="p">:</span>8020<span class="o">/</span><span class="n">user</span><span
class="o">/</span><span class="n">yarn</span><span class="o">/</span><span
class="n">agent</span><span class="o">/</span><span
class="n">slider</span><span class="o">-</span><span
class="n">agent</span><span class="p">.</span><span class="n">tar</span><span
class="p">.</span><span class="n">gz</span> <span class="o">--</span><span
class="n">template</span> <span class="n">appConfig</span><span
class="p">.</span><span class="n">json</span> <span class="o">--</span><span
class="n">resources</span> <span class="n">resources</span><span
class="p">.</span><span class="n">json</span>
</pre></div>
-<h3 id="wzxhzdk49wzxhzdk50verify-the-application"><a name="verify"></a>Verify
the Application</h3>
+<h3 id="wzxhzdk45wzxhzdk46verify-the-application"><a name="verify"></a>Verify
the Application</h3>
<p>The successful launch of the application can be verified via the YARN
Resource Manager Web UI. In most instances, this UI is accessible via a web
browser at port 8088 of the Resource Manager Host:</p>
<p><img alt="image alt text" src="../images/image_1_apps_rm_ui-lrg.png" /></p>
<p>The specific information for the running application is accessible via the
"ApplicationMaster" link that can be seen in the far right column of the row
associated with the running application (probably the top row):</p>
<p><img alt="image alt text" src="../images/image_0_app_master_ui_lrg.png"
/></p>
-<h3 id="wzxhzdk51wzxhzdk52obtain-client-config"><a
name="client-config"></a>Obtain client config</h3>
+<h3 id="wzxhzdk47wzxhzdk48obtain-client-config"><a
name="client-config"></a>Obtain client config</h3>
<p>An application publishes several useful details that can be used to manage
and communicate with the application instance.</p>
<p><strong>All published data</strong></p>
<p>All published data are available at the
<code>/ws/v1/slider/publisher</code> end point of the AppMaster (e.g.
http://c6403.ambari.apache.org:34701/ws/v1/slider/publisher). Its obtainable
through <code>slider-client status <app name></code> where field
<code>info.am.web.url</code> specifies the base address. The URL is also
advertised in the AppMaster tracking UI.</p>
@@ -424,7 +415,7 @@ Client configurations are at <code>/ws/v
<p><strong>Quick links</strong></p>
<p>An application may publish some quick links for the monitoring UI, JMX
endpoint, etc. as the listening ports are dynamically allocated and thus need
to be published. These information is available at
<code>/ws/v1/slider/publisher/slider/quicklinks</code>.</p>
-<h3 id="wzxhzdk53wzxhzdk54manage-the-application-lifecycle"><a
name="manage"></a>Manage the Application Lifecycle</h3>
+<h3 id="wzxhzdk49wzxhzdk50manage-the-application-lifecycle"><a
name="manage"></a>Manage the Application Lifecycle</h3>
<p>Once started, applications can be frozen/stopped, thawed/restarted, and
destroyed/removed as follows:</p>
<h4 id="frozen">Frozen:</h4>
<div class="codehilite"><pre><span class="o">./</span><span
class="n">slider</span> <span class="n">freeze</span> <span class="n">cl1</span>
@@ -446,7 +437,7 @@ Client configurations are at <code>/ws/v
</pre></div>
-<h1 id="wzxhzdk55wzxhzdk56appendix-a-debugging-slider-agent"><a
name="appendixa"></a>Appendix A: Debugging Slider-Agent</h1>
+<h1 id="wzxhzdk51wzxhzdk52appendix-a-debugging-slider-agent"><a
name="appendixa"></a>Appendix A: Debugging Slider-Agent</h1>
<h2 id="create-and-deploy-slider-agent-configuration">Create and deploy Slider
Agent configuration</h2>
<p>Create an agent config file (agent.ini) based on the sample available
at:</p>
<div class="codehilite"><pre><span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span
class="cp">}</span>/slider-0.30.0/agent/conf/agent.ini
@@ -457,152 +448,15 @@ Client configurations are at <code>/ws/v
<p><code>log_level</code> = INFO or DEBUG, to control the verbosity of log</p>
<p><code>app_log_dir</code> = the relative location of the application log
file</p>
<p><code>log_dir</code> = the relative location of the agent and command log
file</p>
-<div class="codehilite"><pre><span class="k">[server]</span>
-<span class="na">hostname</span><span class="o">=</span><span
class="s">localhost</span>
-<span class="na">port</span><span class="o">=</span><span class="s">8440</span>
-<span class="na">secured_port</span><span class="o">=</span><span
class="s">8441</span>
-<span class="na">check_path</span><span class="o">=</span><span
class="s">/ws/v1/slider/agents/</span>
-<span class="na">register_path</span><span class="o">=</span><span
class="s">/ws/v1/slider/agents/{name}/register</span>
-<span class="na">heartbeat_path</span><span class="o">=</span><span
class="s">/ws/v1/slider/agents/{name}/heartbeat</span>
-
-<span class="k">[agent]</span>
-<span class="na">app_pkg_dir</span><span class="o">=</span><span
class="s">app/definition</span>
-<span class="na">app_install_dir</span><span class="o">=</span><span
class="s">app/install</span>
-<span class="na">app_run_dir</span><span class="o">=</span><span
class="s">app/run</span>
-<span class="na">app_dbg_cmd</span><span class="o">=</span>
-<span class="na">debug_mode_enabled</span><span class="o">=</span><span
class="s">true</span>
-
-<span class="na">app_task_dir</span><span class="o">=</span><span
class="s">app/command-log</span>
-<span class="na">app_log_dir</span><span class="o">=</span><span
class="s">app/log</span>
-<span class="na">app_tmp_dir</span><span class="o">=</span><span
class="s">app/tmp</span>
-
-<span class="na">log_dir</span><span class="o">=</span><span
class="s">infra/log</span>
-<span class="na">run_dir</span><span class="o">=</span><span
class="s">infra/run</span>
-<span class="na">version_file</span><span class="o">=</span><span
class="s">infra/version</span>
-
-<span class="na">log_level</span><span class="o">=</span><span
class="s">INFO</span>
-
-<span class="k">[python]</span>
-
-<span class="k">[command]</span>
-<span class="na">max_retries</span><span class="o">=</span><span
class="s">2</span>
-<span class="na">sleep_between_retries</span><span class="o">=</span><span
class="s">1</span>
-
-<span class="k">[security]</span>
-
-<span class="k">[heartbeat]</span>
-<span class="na">state_interval</span><span class="o">=</span><span
class="s">6</span>
-<span class="na">log_lines_count</span><span class="o">=</span><span
class="s">300</span>
-</pre></div>
-
-
<p>Once created, deploy the agent.ini file to HDFS (it can be deployed in any
location accessible to application instance):</p>
<div class="codehilite"><pre><span class="n">su</span> <span
class="n">yarn</span>
-<span class="n">hdfs</span> <span class="n">dfs</span> <span
class="o">-</span><span class="n">copyFromLocal</span> <span
class="n">agent</span><span class="p">.</span><span class="n">ini</span> <span
class="o">/</span><span class="n">slider</span><span class="o">/</span><span
class="n">agent</span><span class="o">/</span><span class="n">conf</span>
+<span class="n">hdfs</span> <span class="n">dfs</span> <span
class="o">-</span><span class="n">copyFromLocal</span> <span
class="n">agent</span><span class="p">.</span><span class="n">ini</span> <span
class="o">/</span><span class="n">user</span><span class="o">/</span><span
class="n">yarn</span><span class="o">/</span><span class="n">agent</span><span
class="o">/</span><span class="n">conf</span>
</pre></div>
<p>Modify the --template json file (appConfig.json) and add the location of
the agent.ini file.</p>
-<div class="codehilite"><pre>"<span class="n">agent</span><span
class="p">.</span><span class="n">conf</span>"<span class="p">:</span>
"<span class="o">/</span><span class="n">slider</span><span
class="o">/</span><span class="n">agent</span><span class="o">/</span><span
class="n">conf</span><span class="o">/</span><span class="n">agent</span><span
class="p">.</span><span class="n">ini</span>"
-</pre></div>
-
-
-<h1 id="wzxhzdk57wzxhzdk58appendix-b-resource-template-resourcesjson"><a
name="appendixb"></a>Appendix B: Resource template (resources.json)</h1>
-<p>This sample is suitable for HBase application.</p>
-<div class="codehilite"><pre><span class="p">{</span>
- "<span class="n">schema</span>"<span class="p">:</span>
"<span class="n">http</span><span class="p">:</span><span
class="o">//</span><span class="n">example</span><span class="p">.</span><span
class="n">org</span><span class="o">/</span><span
class="n">specification</span><span class="o">/</span><span
class="n">v2</span><span class="p">.</span>0<span
class="p">.</span>0"<span class="p">,</span>
- "<span class="n">metadata</span>"<span class="p">:</span> <span
class="p">{</span>
- <span class="p">},</span>
- "<span class="k">global</span>"<span class="p">:</span> <span
class="p">{</span>
- <span class="p">},</span>
- "<span class="n">components</span>"<span class="p">:</span> <span
class="p">{</span>
- "<span class="n">HBASE_MASTER</span>"<span class="p">:</span>
<span class="p">{</span>
- "<span class="n">yarn</span><span class="p">.</span><span
class="n">role</span><span class="p">.</span><span
class="n">priority</span>"<span class="p">:</span> "1"<span
class="p">,</span>
- "<span class="n">yarn</span><span class="p">.</span><span
class="n">component</span><span class="p">.</span><span
class="n">instances</span>"<span class="p">:</span> "1"<span
class="p">,</span>
- "<span class="n">yarn</span><span class="p">.</span><span
class="n">memory</span>"<span class="p">:</span> "256"
- <span class="p">},</span>
- "<span class="n">slider</span><span class="o">-</span><span
class="n">appmaster</span>"<span class="p">:</span> <span
class="p">{</span>
- <span class="p">},</span>
- "<span class="n">HBASE_REGIONSERVER</span>"<span
class="p">:</span> <span class="p">{</span>
- "<span class="n">yarn</span><span class="p">.</span><span
class="n">role</span><span class="p">.</span><span
class="n">priority</span>"<span class="p">:</span> "2"<span
class="p">,</span>
- "<span class="n">yarn</span><span class="p">.</span><span
class="n">component</span><span class="p">.</span><span
class="n">instances</span>"<span class="p">:</span> "1"<span
class="p">,</span>
- "<span class="n">yarn</span><span class="p">.</span><span
class="n">memory</span>"<span class="p">:</span> "256"
- <span class="p">}</span>
- <span class="p">}</span>
-<span class="p">}</span>
-</pre></div>
-
-
-<h1 id="wzxhzdk59wzxhzdk60appendix-c-config-template-appcofigjson"><a
name="appendixc"></a>Appendix C: Config template (appCofig.json)</h1>
-<p>This sample is suitable for HBase application. See <a
href="/docs/slider_specs/application_instance_configuration.html">Application
Instance Configuration</a> for detailed explanation of how to create an
appConfig.json file.</p>
-<div class="codehilite"><pre>{
- "schema": "http://example.org/specification/v2.0.0",
- "metadata": {
- },
- "global": {
- "application.def": "<span class="cp">${</span><span
class="n">app</span><span class="o">.</span><span class="n">package</span><span
class="o">.</span><span class="n">name</span><span
class="cp">}</span>.zip",
- "create.default.zookeeper.node": "true",
- "config_types": "core-site,hdfs-site,hbase-site",
- "java_home": "/usr/jdk64/jdk1.7.0_45",
- "package_list": "files/hbase-<span
class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span
class="n">version</span><span class="cp">}</span>-bin.tar.gz",
- "site.global.app_user": "yarn",
- "site.global.app_log_dir": "<span class="cp">${</span><span
class="n">AGENT_LOG_ROOT</span><span class="cp">}</span>/app/log",
- "site.global.app_pid_dir": "<span class="cp">${</span><span
class="n">AGENT_WORK_ROOT</span><span class="cp">}</span>/app/run",
- "site.global.app_root": "<span class="cp">${</span><span
class="n">AGENT_WORK_ROOT</span><span
class="cp">}</span>/app/install/hbase-<span class="cp">${</span><span
class="n">hbase</span><span class="o">.</span><span
class="n">version</span><span class="cp">}</span>",
- "site.global.app_install_dir": "<span
class="cp">${</span><span class="n">AGENT_WORK_ROOT</span><span
class="cp">}</span>/app/install",
- "site.global.hbase_master_heapsize": "1024m",
- "site.global.hbase_regionserver_heapsize": "1024m",
- "site.global.hbase_instance_name": "instancename",
- "site.global.hbase_root_password": "secret",
- "site.global.user_group": "hadoop",
- "site.global.security_enabled": "false",
- "site.global.monitor_protocol": "http",
- "site.global.ganglia_server_host": "<span
class="cp">${</span><span class="n">NN_HOST</span><span
class="cp">}</span>",
- "site.global.ganglia_server_port": "8667",
- "site.global.ganglia_server_id": "Application1",
- "site.hbase-site.hbase.hstore.flush.retries.number":
"120",
- "site.hbase-site.hbase.client.keyvalue.maxsize":
"10485760",
- "site.hbase-site.hbase.hstore.compactionThreshold":
"3",
- "site.hbase-site.hbase.rootdir": "<span
class="cp">${</span><span class="n">DEFAULT_DATA_DIR</span><span
class="cp">}</span>/data",
- "site.hbase-site.hbase.stagingdir": "<span
class="cp">${</span><span class="n">DEFAULT_DATA_DIR</span><span
class="cp">}</span>/staging",
- "site.hbase-site.hbase.regionserver.handler.count":
"60",
- "site.hbase-site.hbase.regionserver.global.memstore.lowerLimit":
"0.38",
- "site.hbase-site.hbase.hregion.memstore.block.multiplier":
"2",
- "site.hbase-site.hbase.hregion.memstore.flush.size":
"134217728",
- "site.hbase-site.hbase.superuser": "yarn",
- "site.hbase-site.hbase.zookeeper.property.clientPort":
"2181",
- "site.hbase-site.hbase.regionserver.global.memstore.upperLimit":
"0.4",
- "site.hbase-site.zookeeper.session.timeout": "30000",
- "site.hbase-site.hbase.tmp.dir": "<span
class="cp">${</span><span class="n">AGENT_WORK_ROOT</span><span
class="cp">}</span>/work/app/tmp",
- "site.hbase-site.hbase.local.dir": "<span
class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span
class="n">tmp</span><span class="o">.</span><span class="n">dir</span><span
class="cp">}</span>/local",
- "site.hbase-site.hbase.hregion.max.filesize":
"10737418240",
- "site.hbase-site.hfile.block.cache.size": "0.40",
- "site.hbase-site.hbase.security.authentication":
"simple",
- "site.hbase-site.hbase.defaults.for.version.skip":
"true",
- "site.hbase-site.hbase.zookeeper.quorum": "<span
class="cp">${</span><span class="n">ZK_HOST</span><span
class="cp">}</span>",
- "site.hbase-site.zookeeper.znode.parent": "<span
class="cp">${</span><span class="n">DEF_ZK_PATH</span><span
class="cp">}</span>",
- "site.hbase-site.hbase.hstore.blockingStoreFiles":
"10",
- "site.hbase-site.hbase.hregion.majorcompaction":
"86400000",
- "site.hbase-site.hbase.security.authorization":
"false",
- "site.hbase-site.hbase.cluster.distributed": "true",
- "site.hbase-site.hbase.hregion.memstore.mslab.enabled":
"true",
- "site.hbase-site.hbase.client.scanner.caching": "100",
- "site.hbase-site.hbase.zookeeper.useMulti": "true",
- "site.hbase-site.hbase.regionserver.info.port": "0",
- "site.hbase-site.hbase.master.info.port": "<span
class="cp">${</span><span class="n">HBASE_MASTER</span><span
class="o">.</span><span class="n">ALLOCATED_PORT</span><span
class="cp">}</span>",
- "site.hbase-site.hbase.regionserver.port": "0"
- },
- "components": {
- "HBASE_MASTER": {
- },
- "slider-appmaster": {
- "jvm.heapsize": "256M"
- },
- "HBASE_REGIONSERVER": {
- }
- }
-}
+<div class="codehilite"><pre>"<span class="n">agent</span><span
class="p">.</span><span class="n">conf</span>"<span class="p">:</span>
"<span class="o">/</span><span class="n">user</span><span
class="o">/</span><span class="n">yarn</span><span class="o">/</span><span
class="n">agent</span><span class="o">/</span><span class="n">conf</span><span
class="o">/</span><span class="n">agent</span><span class="p">.</span><span
class="n">ini</span>"
</pre></div>
</div>
Modified:
websites/staging/slider/trunk/content/docs/slider_specs/hello_world_slider_app.html
==============================================================================
---
websites/staging/slider/trunk/content/docs/slider_specs/hello_world_slider_app.html
(original)
+++
websites/staging/slider/trunk/content/docs/slider_specs/hello_world_slider_app.html
Wed Aug 27 22:08:00 2014
@@ -173,12 +173,12 @@ Latest release: <strong>0.40-incubating<
-->
<h1 id="goal">Goal</h1>
-<p>Define the Slider application package for YARN hosted <a
href="https://code.google.com/p/jmemcache-daemon/";>jmemcached</a>. Jmemcached
is a functionally equivalent java implementation of memcached, a distributed
memory object caching system. </p>
+<p>Define the Slider application package for YARN hosted <a
href="https://code.google.com/p/jmemcache-daemon/";>jmemcached</a>. Jmemcached
is a functionally equivalent java implementation of memcached, a distributed
memory object caching system. The memcached daemons export the host/port they
are listening on.</p>
<h2 id="basic-version">Basic version</h2>
<p>The basic version of the app will allow creation of one or more memcached
daemons on default ports. Some memory settings may be configured. Later
examples will enhance the application package to add more capabilities.</p>
<p>The structure of an app package is discussed <a
href="/docs/slider_specs/application_package.html">here</a>.</p>
<p>In this example, the application package created looks as follows:</p>
-<div class="codehilite"><pre><span class="n">unzip</span> <span
class="o">-</span><span class="n">l</span> <span
class="s">"$@"</span> <span class="o">~/</span><span
class="n">enlistments</span><span class="o">/</span><span
class="n">ambari</span><span class="o">-</span><span
class="n">vagrant</span><span class="o">/</span><span
class="n">centos6</span><span class="mf">.4</span><span class="o">/</span><span
class="n">memcached</span><span class="o">/</span><span
class="n">jmemcached</span><span class="o">-</span><span
class="mf">1.0.0</span><span class="p">.</span><span class="n">zip</span>
+<div class="codehilite"><pre><span class="n">unzip</span> <span
class="o">-</span><span class="n">l</span> <span
class="s">"$@"</span> <span class="n">jmemcached</span><span
class="o">-</span><span class="mf">1.0.0</span><span class="p">.</span><span
class="n">zip</span>
<span class="nl">Archive:</span> <span class="o">/</span><span
class="n">jmemcached</span><span class="o">-</span><span
class="mf">1.0.0</span><span class="p">.</span><span class="n">zip</span>
<span class="n">Length</span> <span class="n">Date</span> <span
class="n">Time</span> <span class="n">Name</span>
<span class="o">--------</span> <span class="o">----</span> <span
class="o">----</span> <span class="o">----</span>
@@ -203,11 +203,18 @@ Latest release: <strong>0.40-incubating<
<span class="nt"><name></span>MEMCACHED<span
class="nt"></name></span>
<span class="nt"><comment></span>Memcache is a network accessible
key/value storage system, often used as a distributed cache.<span
class="nt"></comment></span>
<span class="nt"><version></span>1.0.0<span
class="nt"></version></span>
+ <span class="nt"><exportedConfigs></span>None<span
class="nt"></exportedConfigs></span>
<span class="nt"><components></span>
<span class="nt"><component></span>
<span class="nt"><name></span>MEMCACHED<span
class="nt"></name></span>
<span class="nt"><category></span>MASTER<span
class="nt"></category></span>
+ <span class="nt"><componentExports></span>
+ <span class="nt"><componentExport></span>
+ <span class="nt"><name></span>host_port<span
class="nt"></name></span>
+ <span class="nt"><value></span><span
class="cp">${</span><span class="n">THIS_HOST</span><span
class="cp">}</span>:<span class="cp">${</span><span class="n">site</span><span
class="o">.</span><span class="k">global</span><span class="o">.</span><span
class="n">listen_port</span><span class="cp">}</span><span
class="nt"></value></span>
+ <span class="nt"></componentExport></span>
+ <span class="nt"></componentExports></span>
<span class="nt"><commandScript></span>
<span class="nt"><script></span>scripts/memcached.py<span
class="nt"></script></span>
<span class="nt"><scriptType></span>PYTHON<span
class="nt"></scriptType></span>
@@ -277,6 +284,7 @@ Latest release: <strong>0.40-incubating<
<li>site.global.xmx_val - value of Xmx</li>
<li>site.global.xms_val - value of Xms</li>
<li>site.global.memory_val - value of memory</li>
+<li>site.global.listen_port - let Slider know that ports need to be
allocated</li>
</ul>
<p>You can add additional parameters as needed.</p>
<div class="codehilite"><pre>{
@@ -284,22 +292,22 @@ Latest release: <strong>0.40-incubating<
"metadata": {
},
"global": {
- "application.def": "/slider/jmemcached-1.0.0.zip",
+ "application.def": "jmemcached-1.0.0.zip",
"java_home": "/usr/jdk64/jdk1.7.0_45",
- "package_list": "files/jmemcached-1.0.0.tar",
- "config_types": "",
+
"site.global.app_user": "yarn",
"site.global.app_root": "<span class="cp">${</span><span
class="n">AGENT_WORK_ROOT</span><span
class="cp">}</span>/app/install/jmemcached-1.0.0",
+ "site.global.pid_file": "<span class="cp">${</span><span
class="n">AGENT_WORK_ROOT</span><span
class="cp">}</span>/app/run/component.pid",
+
"site.global.additional_cp": "/usr/lib/hadoop/lib/*",
"site.global.xmx_val": "256m",
"site.global.xms_val": "128m",
- "site.global.memory_val": "200M"
+ "site.global.memory_val": "200M",
+ "site.global.listen_port": "<span class="cp">${</span><span
class="n">MEMCACHED</span><span class="o">.</span><span
class="n">ALLOCATED_PORT</span><span class="cp">}</span>{DO_NOT_PROPAGATE}"
},
"components": {
"slider-appmaster": {
"jvm.heapsize": "256M"
- },
- "MEMCACHED": {
}
}
}
@@ -317,10 +325,13 @@ Latest release: <strong>0.40-incubating<
<span class="n">app_root</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'configurations'</span><span class="p">][</span><span
class="s">'global'</span><span class="p">][</span><span
class="s">'app_root'</span><span class="p">]</span>
<span class="n">java64_home</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'hostLevelParams'</span><span class="p">][</span><span
class="s">'java_home'</span><span class="p">]</span>
<span class="n">app_user</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'configurations'</span><span class="p">][</span><span
class="s">'global'</span><span class="p">][</span><span
class="s">'app_user'</span><span class="p">]</span>
+<span class="n">pid_file</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'configurations'</span><span class="p">][</span><span
class="s">'global'</span><span class="p">][</span><span
class="s">'pid_file'</span><span class="p">]</span>
+
<span class="n">additional_cp</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'configurations'</span><span class="p">][</span><span
class="s">'global'</span><span class="p">][</span><span
class="s">'additional_cp'</span><span class="p">]</span>
<span class="n">xmx_val</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'configurations'</span><span class="p">][</span><span
class="s">'global'</span><span class="p">][</span><span
class="s">'xmx_val'</span><span class="p">]</span>
<span class="n">xms_val</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'configurations'</span><span class="p">][</span><span
class="s">'global'</span><span class="p">][</span><span
class="s">'xms_val'</span><span class="p">]</span>
<span class="n">memory_val</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'configurations'</span><span class="p">][</span><span
class="s">'global'</span><span class="p">][</span><span
class="s">'memory_val'</span><span class="p">]</span>
+<span class="n">port</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'configurations'</span><span class="p">][</span><span
class="s">'global'</span><span class="p">][</span><span
class="s">'listen_port'</span><span class="p">]</span>
</pre></div>
@@ -340,12 +351,13 @@ Latest release: <strong>0.40-incubating<
<span class="n">import</span> <span class="n">params</span>
<span class="n">env</span><span class="p">.</span><span
class="n">set_params</span><span class="p">(</span><span
class="n">params</span><span class="p">)</span>
<span class="n">self</span><span class="p">.</span><span
class="n">configure</span><span class="p">(</span><span
class="n">env</span><span class="p">)</span>
- <span class="n">process_cmd</span> <span class="p">=</span> <span
class="n">format</span><span class="p">(</span>"<span
class="p">{</span><span class="n">java64_home</span><span
class="p">}</span><span class="o">/</span><span class="n">bin</span><span
class="o">/</span><span class="n">java</span> <span class="o">-</span><span
class="n">Xmx</span><span class="p">{</span><span class="n">xmx_val</span><span
class="p">}</span> <span class="o">-</span><span class="n">Xms</span><span
class="p">{</span><span class="n">xms_val</span><span class="p">}</span> <span
class="o">-</span><span class="n">classpath</span> <span
class="p">{</span><span class="n">app_root</span><span class="p">}</span><span
class="o">/*</span><span class="p">:{</span><span
class="n">additional_cp</span><span class="p">}</span> <span
class="n">com</span><span class="p">.</span><span
class="n">thimbleware</span><span class="p">.</span><span
class="n">jmemcached</span><span class="p">.</span><span class="n">Main
</span> <span class="o">--</span><span class="n">memory</span><span
class="p">={</span><span class="n">memory_val</span><span
class="p">}</span>"<span class="p">)</span>
+ <span class="n">process_cmd</span> <span class="p">=</span> <span
class="n">format</span><span class="p">(</span>"<span
class="p">{</span><span class="n">java64_home</span><span
class="p">}</span><span class="o">/</span><span class="n">bin</span><span
class="o">/</span><span class="n">java</span> <span class="o">-</span><span
class="n">Xmx</span><span class="p">{</span><span class="n">xmx_val</span><span
class="p">}</span> <span class="o">-</span><span class="n">Xms</span><span
class="p">{</span><span class="n">xms_val</span><span class="p">}</span> <span
class="o">-</span><span class="n">classpath</span> <span
class="p">{</span><span class="n">app_root</span><span class="p">}</span><span
class="o">/*</span><span class="p">:{</span><span
class="n">additional_cp</span><span class="p">}</span> <span
class="n">com</span><span class="p">.</span><span
class="n">thimbleware</span><span class="p">.</span><span
class="n">jmemcached</span><span class="p">.</span><span class="n">Main
</span> <span class="o">--</span><span class="n">memory</span><span
class="p">={</span><span class="n">memory_val</span><span class="p">}</span>
<span class="o">--</span><span class="n">port</span><span
class="p">={</span><span class="n">port</span><span
class="p">}</span>"<span class="p">)</span>
<span class="n">Execute</span><span class="p">(</span><span
class="n">process_cmd</span><span class="p">,</span>
<span class="n">user</span><span class="p">=</span><span
class="n">params</span><span class="p">.</span><span
class="n">app_user</span><span class="p">,</span>
<span class="n">logoutput</span><span class="p">=</span><span
class="n">False</span><span class="p">,</span>
- <span class="n">wait_for_finish</span><span class="p">=</span><span
class="n">False</span>
+ <span class="n">wait_for_finish</span><span class="p">=</span><span
class="n">False</span><span class="p">,</span>
+ <span class="n">pid_file</span><span class="p">=</span><span
class="n">params</span><span class="p">.</span><span class="n">pid_file</span>
<span class="p">)</span>
<span class="n">def</span> <span class="n">stop</span><span
class="p">(</span><span class="n">self</span><span class="p">,</span> <span
class="n">env</span><span class="p">):</span>
@@ -355,6 +367,7 @@ Latest release: <strong>0.40-incubating<
<span class="n">def</span> <span class="n">status</span><span
class="p">(</span><span class="n">self</span><span class="p">,</span> <span
class="n">env</span><span class="p">):</span>
<span class="n">import</span> <span class="n">params</span>
<span class="n">env</span><span class="p">.</span><span
class="n">set_params</span><span class="p">(</span><span
class="n">params</span><span class="p">)</span>
+ <span class="n">check_process_status</span><span class="p">(</span><span
class="n">params</span><span class="p">.</span><span
class="n">pid_file</span><span class="p">)</span>
</pre></div>
@@ -363,6 +376,7 @@ Latest release: <strong>0.40-incubating<
<li>Expand the given tarball</li>
<li>Reads the provided configuration and creates the command string</li>
<li>Executes the command to start jmemcached</li>
+<li>Start writes the PID into a file that is used to check status of the
daemon</li>
</ul>
</div>
Modified: websites/staging/slider/trunk/content/docs/slider_specs/index.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/slider_specs/index.html
(original)
+++ websites/staging/slider/trunk/content/docs/slider_specs/index.html Wed Aug
27 22:08:00 2014
@@ -193,6 +193,7 @@ Latest release: <strong>0.40-incubating<
<li>Requires apps to follow Slider guidelines and packaging ("Sliderize")</li>
</ul>
<h2 id="specifications">Specifications</h2>
+<p>Refer to <a href="hello_world_slider_app.html">Creating a Slider package
for Memcached</a> for a quick over view of how to write a Slider app. </p>
<p>The entry points to leverage Slider are:</p>
<ul>
<li><a href="application_needs.html">Application Needs</a> What it takes to be
deployable by Slider. </li>
@@ -202,7 +203,8 @@ Latest release: <strong>0.40-incubating<
<li><a href="resource_specification.html">Specification of Resources</a> How
to write a resource spec for an app?</li>
<li><a href="application_instance_configuration.html">Specifications
InstanceConfiguration</a> How to write a template config for an app?</li>
<li><a href="canonical_scenarios.html">Guidelines for Clients and Client
Applications</a></li>
-<li><a href="application_configuration.html">Specifications for
Configuration</a> Default application configuration?</li>
+<li><a href="application_configuration.html">Specifications for
Configuration</a> Default application configuration</li>
+<li><a href="specifying_exports.html">Specifying Exports</a> How to specify
exports for an application?</li>
<li><a href="/developing/index.html">Documentation for "General Developer
Guidelines"</a></li>
<li><a href="slider_specs/chaosmonkey.md">Configuring the Slider Chaos
Monkey</a></li>
</ul>
