Author: buildbot
Date: Tue Nov 25 19:04:14 2014
New Revision: 930476
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/design/architecture.html
websites/staging/slider/trunk/content/design/registry/index.html
websites/staging/slider/trunk/content/design/rolehistory.html
websites/staging/slider/trunk/content/design/specification/cli-actions.html
websites/staging/slider/trunk/content/developing/building.html
websites/staging/slider/trunk/content/developing/functional_tests.html
websites/staging/slider/trunk/content/developing/manual_testing.html
websites/staging/slider/trunk/content/developing/releasing.html
websites/staging/slider/trunk/content/developing/windows.html
websites/staging/slider/trunk/content/docs/client-configuration.html
websites/staging/slider/trunk/content/docs/configuration/specification.html
websites/staging/slider/trunk/content/docs/examples.html
websites/staging/slider/trunk/content/docs/getting_started.html
websites/staging/slider/trunk/content/docs/manpage.html
websites/staging/slider/trunk/content/docs/security.html
websites/staging/slider/trunk/content/docs/slider_specs/canonical_scenarios.html
websites/staging/slider/trunk/content/docs/slider_specs/specifying_exports.html
websites/staging/slider/trunk/content/downloads/index.html
websites/staging/slider/trunk/content/index.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Tue Nov 25 19:04:14 2014
@@ -1 +1 @@
-1641487
+1641675
Modified: websites/staging/slider/trunk/content/design/architecture.html
==============================================================================
--- websites/staging/slider/trunk/content/design/architecture.html (original)
+++ websites/staging/slider/trunk/content/design/architecture.html Tue Nov 25
19:04:14 2014
@@ -259,7 +259,7 @@ other long-lived YARN application) in a
To enable access to the database files as that submitting use, the identity of
the user is provided when the AM is created; the
AM will pass this same identity down to the created containers. This
information <em>identifies</em> the user -but does not <em>authenticate</em>
them: they are trusted to be who they claim to be.</p>
<h3 id="agent-to-application-master-secure-communication">Agent to Application
Master Secure Communication</h3>
-<p>By default, one-way SSL is leveraged to secure the communication between
Slider agents and the Application Master. However, two-way SSL can be enabled.
A more detailed discussion of the SSL implementation in Slider can be found <a
href="/design/ssl_implementation.html">here</a>.</p>
+<p>By default, one-way SSL is leveraged to secure the communication between
Slider agents and the Application Master. However, two-way SSL can be enabled.
A more detailed discussion of the SSL implementation in Slider can be found <a
href="/design/ssl_implementation.html">in the SSL documentation</a>.</p>
</div>
<div id="footer">
Modified: websites/staging/slider/trunk/content/design/registry/index.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/index.html (original)
+++ websites/staging/slider/trunk/content/design/registry/index.html Tue Nov 25
19:04:14 2014
@@ -178,6 +178,15 @@ Latest release: <strong>0.60.0-incubatin
-->
<h1 id="apache-slider-service-registry">Apache Slider Service Registry</h1>
+<h2 id="important-this-document-is-obsolete">Important: This document is
obsolete</h2>
+<p>This secion has been supplanted by the YARN Service Registry; a
cross-application
+service registry built into Apache Hadoop, with special integration with
+the YARN Resource Manager, security in a secure cluster, and other
features.</p>
+<p>This registry , the "YARN-913" was implemented and tested by the Slider
team,
+so we are confident that it works for Slider as well as other applications.
+Indeed, for the Hadoop 2.6 release, Slider was the functional test suite for
+the registry.</p>
+<h2 id="introduction">Introduction</h2>
<p>The service registry is a core part of the Slider Architecture -it is how
dynamically generated configurations are published for clients to pick up.</p>
<p>The need for a service registry goes beyond Slider, however. We effectively
Modified: websites/staging/slider/trunk/content/design/rolehistory.html
==============================================================================
--- websites/staging/slider/trunk/content/design/rolehistory.html (original)
+++ websites/staging/slider/trunk/content/design/rolehistory.html Tue Nov 25
19:04:14 2014
@@ -290,11 +290,13 @@ used in the past.</p>
</li>
<li>
<p>This history remembers when nodes were allocated. These are re-requested
-when thawing a cluster.</p>
+when starting an application instance again, and when re-requesting
+components after any node failure.</p>
</li>
<li>
<p>It must also remember when nodes were released -these are re-requested
-when returning the cluster size to a previous size during flex operations.</p>
+when expanding a set of instances of a specific component
+to a previous size during flex operations.</p>
</li>
<li>
<p>It has to track nodes for which Slider has an outstanding container request
@@ -404,7 +406,7 @@ let YARN choose.</li>
<ul>
<li>Size of the data structure is <code>O(nodes * role-instances</code>). This
could be mitigated by regular cleansing of the structure. For example, at
-thaw time (or intermittently) all unused nodes > 2 weeks old could be
dropped.</li>
+startup time (or intermittently) all unused nodes > 2 weeks old could be
dropped.</li>
<li>Locating a free node could take <code>O(nodes)</code> lookups -and if the
criteria of "newest"
is included, will take exactly <code>O(nodes)</code> lookups. As an
optimization, a list
of recently explicitly released nodes can be maintained.</li>
@@ -463,7 +465,7 @@ efficient but it does mandate that the n
<p>The two fields <code>releasing</code> and <code>requested</code> are used
to track the ongoing
-state of YARN requests; they do not need to be persisted across freeze/thaw
+state of YARN requests; they do not need to be persisted across stop/start
cycles. They may be relevant across AM restart, but without other data
structures in the AM, not enough to track what the AM was up to before
it was restarted. The strategy will be to ignore unexpected allocation
@@ -547,7 +549,7 @@ to 1 Sort + M list lookups.</p>
list of all Nodes which are available for an instance of that role,
using a comparator that places the most recently released node ahead of older
nodes.</p>
-<p>This list is not persisted -when a Slider Cluster is frozen it is moot, and
when
+<p>This list is not persisted -when a Slider Cluster is stopped it is moot,
and when
an AM is restarted this structure will be rebuilt.</p>
<ol>
<li>When a node is needed for a new request, this list is consulted first.</li>
@@ -581,12 +583,11 @@ will not be satisfied by that node.</p>
then re-requesting containers on it will amplify the damage.</p>
<h2 id="actions">Actions</h2>
<h3 id="bootstrap">Bootstrap</h3>
-<ol>
-<li>Persistent Role History file not found; empty data structures created.</li>
-</ol>
-<h3 id="thaw">Thaw</h3>
-<p>When thawing, the Role History should be loaded -if it is missing Slider
-must revert to the bootstrap actions.</p>
+<p>1Persistent Role History file not found; empty data structures created.</p>
+<h3 id="restart">Restart</h3>
+<p>When starting a cluster, the Role History should be loaded. </p>
+<p>If the history is missing <em>or cannot be loaded for any reason</em>,
+Slider must revert to the bootstrap actions.</p>
<p>If found, the Role History will contain Slider's view of the Slider
Cluster's
state at the time the history was saved, explicitly recording the last-used
time of all nodes no longer hosting a role's container. By noting which roles
@@ -626,7 +627,7 @@ after the last save is unknown.</p>
<p>After this operation, the structures are purged with all out of date
entries,
and the available node list contains a sorted list of the remainder.</p>
<h3 id="am-restart">AM Restart</h3>
-<p>1: Create the initial data structures as the thaw operation
+<p>1: Create the initial data structures as a normal start operation
2: update the structure with the list of live nodes, removing those nodes
from the list of available nodes</p>
<div class="codehilite"><pre><span class="n">now</span> <span
class="p">=</span> <span class="n">time</span><span class="p">()</span>
@@ -720,13 +721,13 @@ large cluster, as for every instance nee
nodemap is required (unless there is some clever bulk assignment list being
built
up), or a sorted version of the nodemap is maintained, with a node placed
at the front of this list whenever its is updated.</p>
-<p><strong>Thaw-time problems</strong></p>
-<p>There is also the risk that while thawing, the
<code>rolehistory.saved</code>
+<p><strong>Startup-time problems</strong></p>
+<p>There is also the risk that while starting an applicatin instance, the
<code>rolehistory.saved</code>
flag may be updated while the cluster flex is in progress, so making the saved
nodes appear out of date. Perhaps the list of recently released nodes could
-be rebuilt at thaw time.</p>
+be rebuilt at startup time.</p>
<p>The proposed <code>recentlyReleasedList</code> addresses this, though it
creates
-another data structure to maintain and rebuild at cluster thaw time
+another data structure to maintain and rebuild at startup time
from the last-used fields in the node entries.</p>
<h3 id="am-callback-oncontainersallocated">AM Callback :
onContainersAllocated</h3>
<div class="codehilite"><pre><span class="n">void</span> <span
class="n">onContainersAllocated</span><span class="p">(</span><span
class="n">List</span><span class="o"><</span><span
class="n">Container</span><span class="o">></span> <span
class="n">allocatedContainers</span><span class="p">)</span>
@@ -1026,8 +1027,7 @@ and two region servers.</p>
</pre></div>
-<p>At this point the cluster was frozen and thawed. Slider does not save the
cluster state
-at freeze time, but does as it is rebuilt.</p>
+<p>At this point the cluster was stopped and started.</p>
<p>When the cluster is restarted, every node that was active for a role at the
time the file was saved <code>1384183476028</code>
is given a last_used timestamp of that time. </p>
<p>When the history is next saved, the master has come back onto the (single)
node,
@@ -1053,11 +1053,11 @@ No region servers are yet live.</p>
</pre></div>
-<p>The <code>last_used</code> timestamps will not be changed until the cluster
is shrunk or thawed, as the <code>active</code> flag being set
+<p>The <code>last_used</code> timestamps will not be changed until the cluster
is shrunk or restarted, as the <code>active</code> flag being set
implies that the server is running both roles at the save time of
<code>1384183512217</code>.</p>
<h2 id="resolved-issues">Resolved issues</h2>
<blockquote>
-<p>How best to distinguish at thaw time from nodes used just before thawing
+<p>How best to distinguish at startup time from nodes used recently
from nodes used some period before? Should the RoleHistory simply forget
about nodes which are older than some threshold when reading in the
history?</p>
</blockquote>
@@ -1072,7 +1072,7 @@ that node creates the container and then
to add blacklisting too? Or actually monitor the container start time, and
if a container hasn't been there for very long, don't pick it.</p>
</blockquote>
-<p>Startup failures drop the node from the ready-to-use list; the node is no
longer
+<p>Container Startup failures drop the node from the ready-to-use list; the
node is no longer
trusted. We don't blacklist it (yet)</p>
<blockquote>
<p>Should we prioritise a node that was used for a long session ahead of
Modified:
websites/staging/slider/trunk/content/design/specification/cli-actions.html
==============================================================================
--- websites/staging/slider/trunk/content/design/specification/cli-actions.html
(original)
+++ websites/staging/slider/trunk/content/design/specification/cli-actions.html
Tue Nov 25 19:04:14 2014
@@ -346,13 +346,12 @@ the command-line supplied option will ov
</pre></div>
-<h2 id="action-thaw">Action: Thaw</h2>
-<p><strong>Note: thaw has been renamed to start in the develop
branch</strong></p>
-<div class="codehilite"><pre><span class="n">thaw</span> <span
class="o"><</span><span class="n">instancename</span><span
class="o">></span> <span class="p">[</span><span class="o">--</span><span
class="n">wait</span> <span class="o"><</span><span
class="n">timeout</span><span class="o">></span><span class="p">]</span>
+<h2 id="action-start">Action: Start</h2>
+<div class="codehilite"><pre><span class="n">start</span> <span
class="o"><</span><span class="n">instancename</span><span
class="o">></span> <span class="p">[</span><span class="o">--</span><span
class="n">wait</span> <span class="o"><</span><span
class="n">timeout</span><span class="o">></span><span class="p">]</span>
</pre></div>
-<p>Thaw takes an application instance with configuration and (possibly) data
on disk, and
+<p>Takes an application instance with configuration and (possibly) data on
disk, and
attempts to create a live application with the specified number of nodes</p>
<h4 id="preconditions_1">Preconditions</h4>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">not</span> <span class="n">valid</span><span class="o">-</span><span
class="n">instance</span><span class="o">-</span><span
class="n">name</span><span class="p">(</span><span
class="n">instancename</span><span class="p">)</span> <span class="p">:</span>
<span class="n">raise</span> <span class="n">SliderException</span><span
class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span
class="p">)</span>
@@ -377,7 +376,7 @@ attempts to create a live application wi
<h3 id="postconditions_1">Postconditions</h3>
-<p>After the thaw has been performed, there is now a queued request in YARN
+<p>After the start operation has been performed, there is now a queued request
in YARN
for the chosen (how?) queue</p>
<div class="codehilite"><pre><span class="n">YARN</span><span
class="o">'</span><span class="p">.</span><span
class="n">Queues</span><span class="o">'</span><span
class="p">[</span><span class="n">amqueue</span><span class="p">]</span> <span
class="p">=</span> <span class="n">YARN</span><span class="p">.</span><span
class="n">Queues</span><span class="p">[</span><span
class="n">amqueue</span><span class="p">]</span> <span class="o">+</span> <span
class="p">[</span><span class="n">launch</span><span
class="p">(</span>"<span class="n">slider</span>"<span
class="p">,</span> <span class="n">instancename</span><span class="p">,</span>
<span class="n">requirements</span><span class="p">,</span> <span
class="n">context</span><span class="p">)]</span>
</pre></div>
@@ -531,18 +530,17 @@ treated as a sign that the container is
either the generated command line is invalid, or the application is failing
to run/exiting on or nearly immediately.</p>
<h2 id="action-create">Action: Create</h2>
-<p>Create is simply <code>build</code> + <code>thaw</code> in sequence - the
postconditions from the first
+<p>Create is simply <code>build</code> + <code>start</code> in sequence - the
postconditions from the first
action are intended to match the preconditions of the second.</p>
-<h2 id="action-freeze">Action: Freeze</h2>
-<p><strong>Note: freeze has been renamed to stop in the develop
branch</strong></p>
-<div class="codehilite"><pre><span class="n">freeze</span> <span
class="n">instancename</span> <span class="p">[</span><span
class="o">--</span><span class="n">wait</span> <span class="n">time</span><span
class="p">]</span> <span class="p">[</span><span class="o">--</span><span
class="n">message</span> <span class="n">message</span><span class="p">]</span>
+<h2 id="action-stop">Action: Stop</h2>
+<div class="codehilite"><pre><span class="n">stop</span> <span
class="n">instancename</span> <span class="p">[</span><span
class="o">--</span><span class="n">wait</span> <span class="n">time</span><span
class="p">]</span> <span class="p">[</span><span class="o">--</span><span
class="n">message</span> <span class="n">message</span><span class="p">]</span>
</pre></div>
-<p>The <em>freeze</em> action "freezes" the cluster: all its nodes running in
the YARN
+<p>The <em>stop</em> action "stop" the cluster: all its nodes running in the
YARN
cluster are stopped, leaving all the persistent state.</p>
<p>The operation is intended to be idempotent: it is not an error if
-freeze is invoked on an already frozen cluster</p>
+<code>stop</code> is invoked on an already stopped cluster</p>
<h4 id="preconditions_4">Preconditions</h4>
<p>The cluster name is valid and it matches a known cluster </p>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">not</span> <span class="n">valid</span><span class="o">-</span><span
class="n">instance</span><span class="o">-</span><span
class="n">name</span><span class="p">(</span><span
class="n">instancename</span><span class="p">)</span> <span class="p">:</span>
<span class="n">raise</span> <span class="n">SliderException</span><span
class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span
class="p">)</span>
@@ -557,7 +555,7 @@ freeze is invoked on an already frozen c
<p>If the <code>--wait</code> argument specified a wait time, then the command
will block
until the cluster has finished or the wait time was exceeded. </p>
<p>If the <code>--message</code> argument specified a message -it must appear
in the
-YARN logs as the reason the cluster was frozen.</p>
+YARN logs as the reason the cluster was stopped.</p>
<p>The outcome should be the same:</p>
<div class="codehilite"><pre><span class="n">not</span> <span
class="n">slider</span><span class="o">-</span><span
class="n">instance</span><span class="o">-</span><span
class="n">live</span><span class="p">(</span><span class="n">YARN</span><span
class="o">'</span><span class="p">,</span> <span
class="n">instancename</span><span class="p">)</span>
</pre></div>
@@ -619,7 +617,7 @@ case the relevant requests will have bee
action. It is not possible to state whether or when the requests will be
satisfied.</p>
<h2 id="action-destroy">Action: Destroy</h2>
-<p>Idempotent operation to destroy a frozen cluster -it succeeds if the
+<p>Idempotent operation to destroy a stopped cluster -it succeeds if the
cluster has already been destroyed/is unknown, but not if it is
actually running.</p>
<h4 id="preconditions_7">Preconditions</h4>
Modified: websites/staging/slider/trunk/content/developing/building.html
==============================================================================
--- websites/staging/slider/trunk/content/developing/building.html (original)
+++ websites/staging/slider/trunk/content/developing/building.html Tue Nov 25
19:04:14 2014
@@ -185,7 +185,7 @@ Latest release: <strong>0.60.0-incubatin
of the local host working. Tests will fail without this. For maven builds to
work, remote network access is often a pre-requisite.</p>
<h3 id="java">Java</h3>
-<p>Slider is built on Java 6 or later</p>
+<p>Slider is built on Java 6 or later.</p>
<h3 id="python">Python</h3>
<p>Slider requires python 2.6</p>
<h3 id="maven">Maven</h3>
@@ -194,7 +194,7 @@ work, remote network access is often a p
</pre></div>
-<p><em>Important</em>: As of October 6, 2013, Maven 3.1 is not supported due to
+<p><em>Important</em>: As of December 2014, Maven 3.1 is not supported due to
<a
href="https://cwiki.apache.org/confluence/display/MAVEN/AetherClassNotFound";>version
issues</a>.</p>
<h3 id="protoc">Protoc</h3>
<p>You need a copy of the <code>protoc</code> compiler for protobuf
compilation</p>
@@ -626,11 +626,16 @@ used Groovy on the production source, th
a Java-only production codebase.</p>
<h2 id="maven-utils">Maven utils</h2>
<p>Here are some handy aliases to make maven easier </p>
-<div class="codehilite"><pre><span class="n">alias</span> <span
class="n">mci</span><span class="p">=</span><span class="s">'mvn clean
install -DskipTests'</span>
-<span class="n">alias</span> <span class="n">mi</span><span
class="p">=</span><span class="s">'mvn install -DskipTests'</span>
-<span class="n">alias</span> <span class="n">mvct</span><span
class="p">=</span><span class="s">'mvn clean test'</span>
-<span class="n">alias</span> <span class="n">mvnsite</span><span
class="p">=</span><span class="s">'mvn site:site
-Dmaven.javadoc.skip=true'</span>
-<span class="n">alias</span> <span class="n">mvt</span><span
class="p">=</span><span class="s">'mvn test'</span>
+<div class="codehilite"><pre><span class="n">alias</span> <span
class="n">mi</span><span class="p">=</span>"<span class="n">mvn</span>
<span class="n">install</span> <span class="o">-</span><span
class="n">DskipTests</span>"
+<span class="n">alias</span> <span class="n">mvi</span><span
class="p">=</span>"<span class="n">mvn</span> <span
class="n">install</span> <span class="o">-</span><span
class="n">DskipTests</span>"
+<span class="n">alias</span> <span class="n">mci</span><span
class="p">=</span>"<span class="n">mvn</span> <span class="n">clean</span>
<span class="n">install</span> <span class="o">-</span><span
class="n">DskipTests</span>"
+<span class="n">alias</span> <span class="n">mvt</span><span
class="p">=</span>"<span class="n">mvn</span> <span
class="n">test</span>"
+<span class="n">alias</span> <span class="n">mvct</span><span
class="p">=</span>"<span class="n">mvn</span> <span class="n">clean</span>
<span class="n">test</span>"
+<span class="n">alias</span> <span class="n">mvp</span><span
class="p">=</span>"<span class="n">mvn</span> <span
class="n">package</span> <span class="o">-</span><span
class="n">DskipTests</span>"
+<span class="n">alias</span> <span class="n">mvcp</span><span
class="p">=</span>"<span class="n">mvn</span> <span class="n">clean</span>
<span class="n">package</span> <span class="o">-</span><span
class="n">DskipTests</span>"
+<span class="n">alias</span> <span class="n">mvnsite</span><span
class="p">=</span>"<span class="n">mvn</span> <span
class="n">site</span><span class="p">:</span><span class="n">site</span> <span
class="o">-</span><span class="n">Dmaven</span><span class="p">.</span><span
class="n">javadoc</span><span class="p">.</span><span
class="n">skip</span><span class="p">=</span><span class="n">true</span>"
+<span class="n">alias</span> <span class="n">mvs</span><span
class="p">=</span>"<span class="n">mvn</span> <span
class="n">site</span><span class="p">:</span><span class="n">site</span> <span
class="o">-</span><span class="n">Dmaven</span><span class="p">.</span><span
class="n">javadoc</span><span class="p">.</span><span
class="n">skip</span><span class="p">=</span><span class="n">true</span> <span
class="o">-</span><span class="n">DskipTests</span>"
+<span class="n">alias</span> <span class="n">mvndep</span><span
class="p">=</span>"<span class="n">mvn</span> <span
class="n">dependency</span><span class="p">:</span><span class="n">tree</span>
<span class="o">-</span><span class="n">Dverbose</span>"
</pre></div>
</div>
Modified: websites/staging/slider/trunk/content/developing/functional_tests.html
==============================================================================
--- websites/staging/slider/trunk/content/developing/functional_tests.html
(original)
+++ websites/staging/slider/trunk/content/developing/functional_tests.html Tue
Nov 25 19:04:14 2014
@@ -313,15 +313,6 @@ need to be changed</p>
<span class="nt"><description></span>Size in MB to ask for
containers<span class="nt"></description></span>
<span class="nt"><value></span>192<span
class="nt"></value></span>
<span class="nt"></property></span>
-
- <span class="nt"><property></span>
- <span class="nt"><name></span>slider.test.launch.python<span
class="nt"></name></span>
- <span class="nt"><description></span>
- Flag to mandate use of the slider.py script.
- Ignored on windows âwhich always uses the python script
- <span class="nt"></description></span>
- <span class="nt"><value></span>false<span
class="nt"></value></span>
-<span class="nt"></property></span>
</pre></div>
Modified: websites/staging/slider/trunk/content/developing/manual_testing.html
==============================================================================
--- websites/staging/slider/trunk/content/developing/manual_testing.html
(original)
+++ websites/staging/slider/trunk/content/developing/manual_testing.html Tue
Nov 25 19:04:14 2014
@@ -179,7 +179,7 @@ Latest release: <strong>0.60.0-incubatin
<h1 id="manually-testing-apache-slider">Manually Testing Apache Slider</h1>
<p>Manual testing invloves using Slider package and an AppPackage to perform
basic
- cluster functionalities such as create/destroy, flex up/down, and freeze/thaw.
+ cluster functionalities such as create/destroy, flex up/down, and start/stop.
A python helper script is provided that can be used to automatically test
and app package.</p>
<h2 id="slidertesterpy"><code>SliderTester.py</code></h2>
<p>Details to be added.</p>
Modified: websites/staging/slider/trunk/content/developing/releasing.html
==============================================================================
--- websites/staging/slider/trunk/content/developing/releasing.html (original)
+++ websites/staging/slider/trunk/content/developing/releasing.html Tue Nov 25
19:04:14 2014
@@ -221,7 +221,7 @@ starting to make a release</em></p>
<p>The slider POMs offer different profiles of hadoop versions to build
against. The instructions
below do not choose one. If a profile is needed, append it to all the maven
commands.</p>
<p>e.g. {{mvn clean install -DskipTests -Prelease-2.6}}</p>
-<h2 id="wzxhzdk38wzxhzdk39phase-i"><a name="phase1"></a>Phase - I</h2>
+<h2 id="wzxhzdk40wzxhzdk41phase-i"><a name="phase1"></a>Phase - I</h2>
<p><strong>Step #1:</strong> Create a JIRA for the release, estimate 3h
(so you don't try to skip the tests)</p>
<div class="codehilite"><pre><span class="n">export</span> <span
class="n">SLIDER_RELEASE_JIRA</span><span class="p">=</span><span
class="n">SLIDER</span><span class="o">-</span>544
@@ -341,7 +341,7 @@ sprint -so wrap that up too.</p>
<p><em>At this point you are ready for Apache release from master.</em></p>
-<h2 id="wzxhzdk40wzxhzdk41phase-ii"><a name="phase2"></a>Phase - II</h2>
+<h2 id="wzxhzdk42wzxhzdk43phase-ii"><a name="phase2"></a>Phase - II</h2>
<p><strong>Step 1</strong></p>
<p>Get a clean enlistment.</p>
<div class="codehilite"><pre><span class="n">mkdir</span> <span
class="n">release</span>
@@ -513,8 +513,13 @@ Upload your PGP public key only to <code
<div class="codehilite"><pre><span class="n">svn</span> <span
class="n">co</span> <span class="n">https</span><span class="p">:</span><span
class="o">//</span><span class="n">dist</span><span class="p">.</span><span
class="n">apache</span><span class="p">.</span><span class="n">org</span><span
class="o">/</span><span class="n">repos</span><span class="o">/</span><span
class="n">dist</span><span class="o">/</span><span
class="n">release</span><span class="o">/</span><span
class="n">incubator</span><span class="o">/</span><span class="n">slider</span>
<span class="n">slider</span>
<span class="n">cd</span> <span class="n">slider</span>
-<span class="n">Edit</span> <span class="n">the</span> <span
class="n">KEYS</span> <span class="n">file</span> <span class="n">as</span>
<span class="n">needed</span><span class="p">.</span> <span
class="n">The</span> <span class="n">instructions</span> <span
class="n">are</span> <span class="n">at</span> <span class="n">the</span> <span
class="n">top</span><span class="p">.</span>
-<span class="n">mkdir</span> $<span class="n">SLIDER_RELEASE</span> $<span
class="n">SLIDER_RELEASE</span><span class="o">/</span><span
class="n">src</span>
+</pre></div>
+
+
+<p>Make sure your PGP keys is in the KEYS file. The instructions to add a key
+are at the top of the file</p>
+<p>Then...</p>
+<div class="codehilite"><pre><span class="n">mkdir</span> $<span
class="n">SLIDER_RELEASE</span> $<span class="n">SLIDER_RELEASE</span><span
class="o">/</span><span class="n">src</span>
<span class="n">cd</span> $<span class="n">SLIDER_RELEASE</span><span
class="o">/</span><span class="n">src</span>
<span class="n">scp</span> <span class="o">~/</span><span
class="n">public_html</span><span class="o">/</span>$<span
class="n">SLIDER_RELEASE</span><span class="o">-</span><span
class="n">rc0</span><span class="o">/*</span> <span class="p">.</span>
<span class="n">cd</span> <span class="p">..</span>
@@ -628,6 +633,28 @@ the actual development version.</p>
<p>Continue the release process from step 12.</p>
</li>
</ol>
+<h2 id="publishing-artifacts">Publishing artifacts</h2>
+<p>Once the vote has (successfully) completed on both slider-dev and
incubator-general, the artifacts may be published in two ways</p>
+<ol>
+<li>tarballs for download via the apache distribution system</li>
+<li>JAR files for download via the maven repository</li>
+</ol>
+<h3 id="publishing-files-for-apache-release">Publishing files for apache
release</h3>
+<p>See the <a href="http://www.apache.org/dev/release.html";>ASF release
guidelines</a></p>
+<h3 id="publishing-the-maven-artifacts">Publishing the maven artifacts</h3>
+<ul>
+<li>See <a
href="http://www.apache.org/dev/publishing-maven-artifacts.html";>ASF guide to
publishing maven artifacts</a></li>
+<li>See <a
href="http://repo.maven.apache.org/maven2/org/apache/apache/10/apache-10.pom";>Apache
Parent Pom</a> for a template POM containing the release plugins. This is not
used in the slider release
+âit was used as a source of some of the actions; the remainder being from
the hadoop project.</li>
+</ul>
+<p>Build the artifacts off the SVN commit used for the release.</p>
+<div class="codehilite"><pre> <span class="n">mvn</span> <span
class="o">-</span><span class="n">Papache</span><span class="o">-</span><span
class="n">release</span>
+</pre></div>
+
+
+<h4 id="important-app-package-versions">Important: App package versions</h4>
+<p>The (slider 0.60) app package versions mean the artifacts are not suitable
for uploading; they generate files such as
<code>slider-storm-app-package-${pkg.version}.pom</code> which don't contain a
valid version number.</p>
+<p>We plan to fix this by moving the app packages out of the slider project
itself, so drive them off other version information.</p>
</div>
<div id="footer">
Modified: websites/staging/slider/trunk/content/developing/windows.html
==============================================================================
--- websites/staging/slider/trunk/content/developing/windows.html (original)
+++ websites/staging/slider/trunk/content/developing/windows.html Tue Nov 25
19:04:14 2014
@@ -179,7 +179,7 @@ Latest release: <strong>0.60.0-incubatin
<ol>
<li>Windows server 2012</li>
<li>Create an admin account for yourself</li>
-<li>take snapshots as you go along</li>
+<li>On a virtual machine: take snapshots as you go along</li>
</ol>
<h1 id="core-installs">Core installs</h1>
<ul>
@@ -187,7 +187,6 @@ Latest release: <strong>0.60.0-incubatin
into a dir with no spaces, <code>C:\bin\openssl</code>. Add its bin dir to
your path.
(this will also need the Visual C++ 2008 Redistributables (DLL libraries)
installed.</li>
-<li></li>
</ul>
<h2 id="virtual-box">Virtual Box</h2>
<ol>
@@ -217,9 +216,7 @@ is "private", then edit the firewall to
<li>Everything Hadoop's <code>BUILDING.TXT</code> needs. This includes the
cygwin tools, protoc,
python and more. If you cannot build Hadoop from the <code>Windows SDK Command
Prompt</code>
then don't proceed.</li>
-<li>PFE32 is still one of the best plain text editors for development around,
-even just for viewing logs & test results.</li>
-<li>...JEdit is fairly nimble too, though not as lightweight.</li>
+<li>JEdit is a good text editor for developing and editing simple files.</li>
<li>Chrome can be used a browser instead of IE if you want âbut you should
still
view all web pages in IE to verify they look OK.</li>
<li><a
href="http://technet.microsoft.com/en-us/sysinternals/bb545021.aspx";>Sysinternals</a>.
Modified: websites/staging/slider/trunk/content/docs/client-configuration.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/client-configuration.html
(original)
+++ websites/staging/slider/trunk/content/docs/client-configuration.html Tue
Nov 25 19:04:14 2014
@@ -199,11 +199,12 @@ YARN resource manager ("the RM"). In a s
identity, the <em>principal</em> of both the HDFS namenode and the YARN RM
-and it may
also need some JVM options set in order for Java's Kerberos module to
correctly identify itself to these services.</p>
-<p>It cannot rely on local <code>HADOOP_PREFIX/conf/hadoop-site.xml</code> and
-<code>$YARN_PREFIX/conf/yarn-site.xml</code> files -because it is designed to
+<p>It cannot rely on local <code>"$HADOOP_CONF_DIR/hadoop-site.xml</code> and
+<code>$HADOOP_CONF_DIR/yarn-site.xml</code> files -because it is designed to
work on client machines that may not have Hadoop and YARN installed.</p>
-<p>Instead all client-side (non-JVM) options can be predefined in the
-configuration file <code>slider-client.xml</code>. </p>
+<p>If these files are present, and the <code>HADOOP_CONF_DIR</code>
environment variable is
+set to the directory that contains them âslider will pick up the settings in
+these files. If not, all settings must be placed in the file
<code>slider-client.xml</code></p>
<h2 id="setting-slider-jvm-options">Setting Slider JVM options</h2>
<p>Core JVM options can be set in the environment variable
<code>SLIDER_JVM_OPTS</code>;
if unset the <code>bin/slider</code> script will use the default values that
were
@@ -311,16 +312,6 @@ as the source of the <code>log4j.propert
<p>As well as standard YARN and Hadoop configuration options, Slider supports
a limited number of slider-specific configuration parameters.</p>
<div class="codehilite"><pre><span class="nt"><property></span>
- <span class="nt"><name></span>slider.zookeeper.quorum<span
class="nt"></name></span>
- <span class="nt"><value></span>localhost:2181,zookeeper2:4545<span
class="nt"></value></span>
-<span class="nt"></property></span>
-
-<span class="nt"><property></span>
- <span class="nt"><name></span>slider.yarn.queue<span
class="nt"></name></span>
- <span class="nt"><value></span>default<span
class="nt"></value></span>
-<span class="nt"></property></span>
-
-<span class="nt"><property></span>
<span class="nt"><name></span>slider.yarn.queue<span
class="nt"></name></span>
<span class="nt"><value></span>default<span
class="nt"></value></span>
<span class="nt"></property></span>
@@ -371,7 +362,7 @@ containers created in the Slider cluster
<p>This identifies the priority within the queue. The lower the value, the
higher the
priority</p>
<p>Default value: <code>1</code>.</p>
-<div class="codehilite"><pre><span class="n">bin</span><span
class="o">/</span><span class="n">slider</span> <span class="n">thaw</span>
<span class="n">cl1</span> <span class="o">-</span><span class="n">D</span>
<span class="n">slider</span><span class="p">.</span><span
class="n">yarn</span><span class="p">.</span><span class="n">queue</span><span
class="p">.</span><span class="n">priority</span><span class="p">=</span>5
+<div class="codehilite"><pre><span class="n">bin</span><span
class="o">/</span><span class="n">slider</span> <span class="n">start</span>
<span class="n">cl1</span> <span class="o">-</span><span class="n">D</span>
<span class="n">slider</span><span class="p">.</span><span
class="n">yarn</span><span class="p">.</span><span class="n">queue</span><span
class="p">.</span><span class="n">priority</span><span class="p">=</span>5
</pre></div>
@@ -399,25 +390,56 @@ that contains the application data <code
<p>Slider uses the (evolving) YARN service registry to register application
instances, to locate them, and as a means to publish the configuration
information
needed to connect to the slider-deployed applications.</p>
-<h3
id="sliderzookeeperquorum-the-zookeeper-quorum"><code>slider.zookeeper.quorum</code>
- the zookeeper quorum.</h3>
+<p>These values should be set in the <code>hadoop-site.xml</code> or
<code>yarn-site.xml</code> files
+for a cluster, for use by all applications. In particular, if the YARN
+Resource Manager is set to automatically create the paths in the registry for
+users âa required feature in secure clustersâ then it must share the same
settings.</p>
+<p>Consult the YARN registry documentation for more information.</p>
+<h3
id="hadoopregistryzkquorum-the-zookeeper-quorum"><code>hadoop.registry.zk.quorum</code>
- the zookeeper quorum.</h3>
<p>This defines the zookeeper quorum for this YARN cluster. </p>
<p>It is used to locate the service registry, enable running instances to
publish
information about their application, and for clients to query this. </p>
<p>It is also used as the default zookeeper binding for any application that
uses zookeeper in its configuration -the value set when the application is
defined will be copied into the instance definition file.</p>
-<h3
id="sliderregistrypath-the-zookeeper-path-for-the-service-registry"><code>"slider.registry.path"</code>
- the zookeeper path for the service registry</h3>
+<p>Example:</p>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>hadoop.registry.zk.quorum<span
class="nt"></name></span>
+ <span class="nt"><value></span>zk1:2181,zookeeper2:4545<span
class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
+
+
+<h3
id="hadoopregistryzkroot-the-zookeeper-path-for-the-service-registry"><code>hadoop.registry.zk.root</code>
- the zookeeper path for the service registry</h3>
<p>This declares the the zookeeper path for the service registry. </p>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>hadoop.registry.zk.root<span
class="nt"></name></span>
+ <span class="nt"><value></span>/registry<span
class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
+
+
<h2 id="debugging-configuration-issues">Debugging configuration issues</h2>
<p>If the slider packages are set to log at debug level in the log4j
configuration
file, details on properties will be part of the copious output.</p>
<h2 id="how-client-options-are-passed-down-to-created-clusters">How client
options are passed down to created clusters.</h2>
-<p>Apart from the filesystem bindings, Client configuration options are
+<p>Most cient configuration options are
not passed down to the XML site specification of the created cluster.</p>
-<p>The sole options passed down are the HDFS bindings:
<code>fs.defaultFS</code>,
+<p>Exceptions:</p>
+<ol>
+<li>
+<p>the HDFS bindings: <code>fs.defaultFS</code>,
which is passed down both as that property and as <code>fs.default.name</code>,
-and, in a secure cluster, the security flag
(<code>slider.security.enabled</code>)
+and, </p>
+</li>
+<li>
+<p>in a secure cluster, the security flag
(<code>slider.security.enabled</code>)
and the HDFS Kerberos principal.</p>
+</li>
+<li>
+<p>The yarn registry options. </p>
+</li>
+</ol>
</div>
<div id="footer">
Modified:
websites/staging/slider/trunk/content/docs/configuration/specification.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/configuration/specification.html
(original)
+++ websites/staging/slider/trunk/content/docs/configuration/specification.html
Tue Nov 25 19:04:14 2014
@@ -209,7 +209,7 @@ aspects of the Slider application master
</ol>
<p>This information defines the <em>desired state</em> of a cluster.</p>
<p>Users may edit these sections, either via the CLI, or by directly editing
the <code>cluster.json</code> file of
- a frozen cluster.</p>
+ a stopped cluster.</p>
<ol>
<li>Status information provided by a running cluster. These include:
information about the cluster, statistics, information about reach role in
Modified: websites/staging/slider/trunk/content/docs/examples.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/examples.html (original)
+++ websites/staging/slider/trunk/content/docs/examples.html Tue Nov 25
19:04:14 2014
@@ -272,12 +272,12 @@ to keep the log bloat under control.</p>
-# <span class="n">freeze</span> <span class="n">the</span> <span
class="n">cluster</span> <span class="p">(</span><span class="o">*</span><span
class="n">freeze</span> <span class="n">has</span> <span class="n">been</span>
<span class="n">renamed</span> <span class="n">to</span> <span
class="n">stop</span> <span class="n">in</span> <span class="n">develop</span>
<span class="n">branch</span><span class="o">*</span><span class="p">)</span>
-<span class="n">slider</span> <span class="n">freeze</span> <span
class="n">cl1</span> <span class="o">\</span>
+# <span class="n">stop</span> <span class="n">a</span> <span
class="n">cluster</span>
+<span class="n">slider</span> <span class="n">stop</span> <span
class="n">cl1</span> <span class="o">\</span>
<span class="o">--</span><span class="n">manager</span> <span
class="n">master</span><span class="p">:</span>8032 <span
class="o">--</span><span class="n">filesystem</span> <span
class="n">hdfs</span><span class="p">:</span><span class="o">//</span><span
class="n">master</span><span class="p">:</span>9090
-# <span class="n">thaw</span> <span class="n">a</span> <span
class="n">cluster</span> <span class="p">(</span><span class="o">*</span><span
class="n">thaw</span> <span class="n">has</span> <span class="n">been</span>
<span class="n">renamed</span> <span class="n">to</span> <span
class="n">start</span> <span class="n">in</span> <span class="n">develop</span>
<span class="n">branch</span><span class="o">*</span><span class="p">)</span>
-<span class="n">slider</span> <span class="n">thaw</span> <span
class="n">cl1</span> <span class="o">\</span>
+# <span class="n">start</span> <span class="n">a</span> <span
class="n">cluster</span>
+<span class="n">slider</span> <span class="n">start</span> <span
class="n">cl1</span> <span class="o">\</span>
<span class="o">--</span><span class="n">manager</span> <span
class="n">master</span><span class="p">:</span>8032 <span
class="o">--</span><span class="n">filesystem</span> <span
class="n">hdfs</span><span class="p">:</span><span class="o">//</span><span
class="n">master</span><span class="p">:</span>9090
# <span class="n">destroy</span> <span class="n">the</span> <span
class="n">cluster</span>
Modified: websites/staging/slider/trunk/content/docs/getting_started.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/getting_started.html (original)
+++ websites/staging/slider/trunk/content/docs/getting_started.html Tue Nov 25
19:04:14 2014
@@ -207,7 +207,7 @@ Latest release: <strong>0.60.0-incubatin
<p><a href="#installapp">Install, Configure, Start and Verify Sample
Application</a></p>
</li>
</ul>
-<h2 id="wzxhzdk20wzxhzdk21system-requirements"><a name="sysreqs"></a>System
Requirements</h2>
+<h2 id="wzxhzdk22wzxhzdk23system-requirements"><a name="sysreqs"></a>System
Requirements</h2>
<p>The Slider deployment has the following minimum system requirements:</p>
<ul>
<li>
@@ -226,7 +226,7 @@ Latest release: <strong>0.60.0-incubatin
<p>openssl</p>
</li>
</ul>
-<h2 id="wzxhzdk22wzxhzdk23setup-the-cluster"><a name="setup"></a>Setup the
Cluster</h2>
+<h2 id="wzxhzdk24wzxhzdk25setup-the-cluster"><a name="setup"></a>Setup the
Cluster</h2>
<p>Set up your Hadoop cluster with the services listed above.</p>
<p><em>Note:</em> Ensure the debug delay config is set to a non-zero value to
allow easy debugging.
If you are using a single VM or a sandbox then you may need to modify your
YARN
@@ -248,15 +248,15 @@ In <code>yarn-site.xml</code> make the f
</table>
<p>There are other options detailed in the Troubleshooting file available <a
href="/docs/troubleshooting.html">here</a>.</p>
-<h2 id="wzxhzdk24wzxhzdk25download-slider-packages"><a
name="download"></a>Download Slider Packages</h2>
+<h2 id="wzxhzdk26wzxhzdk27download-slider-packages"><a
name="download"></a>Download Slider Packages</h2>
<p><em>You can build it as described below.</em></p>
-<h2 id="wzxhzdk26wzxhzdk27build-slider"><a name="build"></a>Build Slider</h2>
+<h2 id="wzxhzdk28wzxhzdk29build-slider"><a name="build"></a>Build Slider</h2>
<ul>
<li>From the top level directory, execute <code>mvn clean site:site site:stage
package -DskipTests</code></li>
<li>Use the generated compressed tar file in slider-assembly/target directory
(e.g. slider-0.60.0-all.tar.gz or slider-0.60.0-all.zip) for the subsequent
steps</li>
<li>If you are cloning the Slider git repo, go to <code>master</code> branch
for the latest released or <code>develop</code> for the latest under
development</li>
</ul>
-<h2 id="wzxhzdk28wzxhzdk29install-slider"><a name="install"></a>Install
Slider</h2>
+<h2 id="wzxhzdk30wzxhzdk31install-slider"><a name="install"></a>Install
Slider</h2>
<p>Slider is installed on a client machine that can access the hadoop cluster.
Follow the following steps to expand/install Slider:</p>
<div class="codehilite"><pre>mkdir <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>
@@ -270,20 +270,37 @@ cd <span class="cp">${</span><span class
<p>Expand the tar file: <code>tar -xvf
slider-0.60.0-incubating-all.tar.gz</code> or <code>unzip
slider-0.60.0-incubating-all.zip</code></p>
<h2 id="configure-slider">Configure Slider</h2>
<p>Browse to the Slider directory: <code>cd slider-0.60.0/conf</code></p>
-<p>Edit slider-env.sh and specify correct values.</p>
+<p>Edit <code>slider-env.sh</code> and specify correct values.</p>
<div class="codehilite"><pre><span class="n">export</span> <span
class="n">JAVA_HOME</span><span class="p">=</span><span class="o">/</span><span
class="n">usr</span><span class="o">/</span><span class="n">jdk64</span><span
class="o">/</span><span class="n">jdk1</span><span class="p">.</span>7<span
class="p">.</span>0<span class="n">_67</span>
<span class="n">export</span> <span class="n">HADOOP_CONF_DIR</span><span
class="p">=</span><span class="o">/</span><span class="n">etc</span><span
class="o">/</span><span class="n">hadoop</span><span class="o">/</span><span
class="n">conf</span>
</pre></div>
-<p>If you are on a node that does not have the hadoop conf folder then you can
add the relevant configurations into slider-client.xml.</p>
+<p>(You only need to set <code>JAVA_HOME</code> if it is not already set)</p>
+<p>If you are on a node that does not have the hadoop conf folder then you can
add the relevant configurations into <code>slider-client.xml</code>.</p>
+<p>You can also simply configure <code>slider-client.xml</code> with the path
to the Hadoop configuration
+directory.</p>
+<p>This can be absolute</p>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>HADOOP_CONF_DIR<span
class="nt"></name></span>
+ <span class="nt"><value></span>/etc/hadoop/conf<span
class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
+
+
+<p>or it can be relative to the property <code>SLIDER_CONF_DIR</code>, which
is set to the
+directory containing the <code>slider-client.xml</code> file.</p>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>HADOOP_CONF_DIR<span
class="nt"></name></span>
+ <span class="nt"><value></span><span class="cp">${</span><span
class="n">SLIDER_CONF_DIR</span><span class="cp">}</span>/../hadoop-conf<span
class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
+
+
+<p>If the hadoop configuration dir is configured this way, the </p>
<p>For example, if you are targeting a non-secure cluster with no HA for
NameNode or ResourceManager, modify Slider configuration file
<code>${slider-install-dir}/slider-0.60.0/conf/slider-client.xml</code> to add
the following properties:</p>
<div class="codehilite"><pre> <span class="nt"><property></span>
- <span class="nt"><name></span>yarn.application.classpath<span
class="nt"></name></span>
- <span
class="nt"><value></span>/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*<span
class="nt"></value></span>
- <span class="nt"></property></span>
- <span class="nt"><property></span>
- <span class="nt"><name></span>slider.zookeeper.quorum<span
class="nt"></name></span>
+ <span class="nt"><name></span>hadoop.registry.zk.quorum<span
class="nt"></name></span>
<span class="nt"><value></span>yourZooKeeperHost:port<span
class="nt"></value></span>
<span class="nt"></property></span>
<span class="nt"><property></span>
@@ -301,25 +318,18 @@ cd <span class="cp">${</span><span class
</pre></div>
-<p>If the target cluster is on Windows then make sure that the
<code>yarn.application.classpath</code> is set accordingly.</p>
-<div class="codehilite"><pre> <span class="nt"><property></span>
- <span class="nt"><name></span>yarn.application.classpath<span
class="nt"></name></span>
- <span class="nt"><value></span>
-
%HADOOP_CONF_DIR%,%HADOOP_COMMON_HOME%/share/hadoop/common/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/*,...
- <span class="nt"></value></span>
- <span class="nt"></property></span>
+<p>Execute:</p>
+<div class="codehilite"><pre><span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>/slider-0.60.0/bin/slider version
</pre></div>
-<p>Execute:</p>
-<div class="codehilite"><pre><span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>/slider-0.60.0/bin/slider version
-OR
-python %slider-install-dir%/slider-0.60.0/bin/slider.py version
+<p>OR</p>
+<div class="codehilite"><pre><span class="n">python</span> <span
class="c">%slider-install-dir%/slider-0.60.0/bin/slider.py version</span>
</pre></div>
<p>Ensure there are no errors and you can see "Compiled against Hadoop 2.6.0".
<strong>This ensures that Slider is installed correctly.</strong></p>
-<h2 id="wzxhzdk30wzxhzdk31deploy-slider-resources"><a name="deploy"></a>Deploy
Slider Resources</h2>
+<h2 id="wzxhzdk32wzxhzdk33deploy-slider-resources"><a name="deploy"></a>Deploy
Slider Resources</h2>
<p>Ensure that all file folders are accessible to the user creating the
application instance. The example assumes "yarn" to be that user. The default
yarn user may be different e.g. hadoop.</p>
<h3 id="ensure-hdfs-home-folder-for-the-user">Ensure HDFS home folder for the
User</h3>
<p>Perform the following steps to create the user home folder with the
appropriate permissions:</p>
@@ -331,7 +341,7 @@ python %slider-install-dir%/slider-0.60.
</pre></div>
-<h2 id="wzxhzdk32wzxhzdk33create-application-packages"><a
name="downsample"></a>Create Application Packages</h2>
+<h2 id="wzxhzdk34wzxhzdk35create-application-packages"><a
name="downsample"></a>Create Application Packages</h2>
<p>Slider "master" branch has several sample application packages available
for use with Slider:</p>
<ul>
<li>
@@ -348,7 +358,7 @@ python %slider-install-dir%/slider-0.60.
</li>
</ul>
<p>Create one or more Slider application packages and follow the steps below
to install them.</p>
-<h2
id="wzxhzdk34wzxhzdk35install-configure-start-and-verify-sample-application"><a
name="installapp"></a>Install, Configure, Start and Verify Sample
Application</h2>
+<h2
id="wzxhzdk36wzxhzdk37install-configure-start-and-verify-sample-application"><a
name="installapp"></a>Install, Configure, Start and Verify Sample
Application</h2>
<ul>
<li>
<p><a href="#load">Install Sample Application Package</a></p>
@@ -366,7 +376,7 @@ python %slider-install-dir%/slider-0.60.
<p><a href="#manage">Manage the Application Lifecycle</a></p>
</li>
</ul>
-<h3 id="wzxhzdk36wzxhzdk37install-sample-application-package"><a
name="load"></a>Install Sample Application Package</h3>
+<h3 id="wzxhzdk38wzxhzdk39install-sample-application-package"><a
name="load"></a>Install Sample Application Package</h3>
<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">install</span><span class="o">-</span><span class="n">package</span>
<span class="o">--</span><span class="n">name</span> <span
class="o">*</span><span class="n">package</span> <span
class="n">name</span><span class="o">*</span> <span class="o">--</span><span
class="n">package</span> <span class="o">*</span><span
class="n">sample</span><span class="o">-</span><span
class="n">application</span><span class="o">-</span><span
class="n">package</span><span class="o">*</span>
</pre></div>
@@ -378,25 +388,25 @@ This path is also reflected in appConfig
</pre></div>
-<h3 id="wzxhzdk38wzxhzdk39create-application-specifications"><a
name="create"></a>Create Application Specifications</h3>
+<h3 id="wzxhzdk40wzxhzdk41create-application-specifications"><a
name="create"></a>Create Application Specifications</h3>
<p>Configuring a Slider application consists of two parts: the <a
href="#resspec">Resource Specification</a>,
and the *<a href="#appconfig">Application Configuration</a>.</p>
<p><strong>Note</strong>: There are sample Resource Specifications
(<strong>resources-default.json</strong>) and Application Configuration
(<strong>appConfig-default.json</strong>) files in the application packages
zip files.</p>
-<h4 id="wzxhzdk40wzxhzdk41resource-specification"><a
name="resspec"></a>Resource Specification</h4>
+<h4 id="wzxhzdk42wzxhzdk43resource-specification"><a
name="resspec"></a>Resource Specification</h4>
<p>Slider needs to know what components (and how many components) to
deploy.</p>
<p>As Slider creates each instance of a component in its own YARN container,
it also needs to know what to ask YARN for in terms of <strong>memory</strong>
and <strong>CPU</strong> for those containers. </p>
<p>All this information goes into the <strong>Resources Specification</strong>
file ("Resource Spec") named <code>resources.json</code>. The Resource Spec
tells Slider how many instances of each component in the application (such as
an HBase RegionServer) to deploy and the parameters for YARN.</p>
<p>An application package should contain the default resources.json and you
can start from there. Or you can create one based on <a
href="slider_specs/resource_specification.html">Resource Specification</a>.</p>
<p>Store the Resource Spec file on your local disk (e.g.
<code>/tmp/resources.json</code>).</p>
-<h4 id="wzxhzdk42wzxhzdk43application-configuration"><a
name="appconfig"></a>Application Configuration</h4>
+<h4 id="wzxhzdk44wzxhzdk45application-configuration"><a
name="appconfig"></a>Application Configuration</h4>
<p>Alongside the Resource Spec there is the <strong>Application
Configuration</strong> file ("App Config") which includes parameters that are
specific to the application, rather than YARN. The App Config is a file that
contains all application configuration. This configuration is applied to the
default configuration provided by the application definition and then handed
off to the associated component agent.</p>
<p>For example, the heap sizes of the JVMs, The App Config defines the
configuration details <strong>specific to the application and
component</strong> instances. An application package should contain the default
resources.json and you can start from there.</p>
<p>Ensure the following variables are accurate:</p>
<ul>
<li>"application.def": "application_package.zip" (path to uploaded application
package)</li>
</ul>
-<h3 id="wzxhzdk44wzxhzdk45start-the-application"><a name="start"></a>Start the
Application</h3>
+<h3 id="wzxhzdk46wzxhzdk47start-the-application"><a name="start"></a>Start the
Application</h3>
<p>Once the steps above are completed, the application can be started through
the <strong>Slider Command Line Interface (CLI)</strong>.</p>
<p>Change directory to the "bin" directory under the slider installation</p>
<div class="codehilite"><pre>cd <span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span class="cp">}</span>/slider-0.60.0/bin
@@ -408,12 +418,12 @@ This path is also reflected in appConfig
</pre></div>
-<h3 id="wzxhzdk46wzxhzdk47verify-the-application"><a name="verify"></a>Verify
the Application</h3>
+<h3 id="wzxhzdk48wzxhzdk49verify-the-application"><a name="verify"></a>Verify
the Application</h3>
<p>The successful launch of the application can be verified via the YARN
Resource Manager Web UI. In most instances, this UI is accessible via a web
browser at port 8088 of the Resource Manager Host:</p>
<p><img alt="image alt text" src="../images/image_1_apps_rm_ui-lrg.png" /></p>
<p>The specific information for the running application is accessible via the
"ApplicationMaster" link that can be seen in the far right column of the row
associated with the running application (probably the top row):</p>
<p><img alt="image alt text" src="../images/image_0_app_master_ui_lrg.png"
/></p>
-<h3 id="wzxhzdk48wzxhzdk49obtain-client-config"><a
name="client-config"></a>Obtain client config</h3>
+<h3 id="wzxhzdk50wzxhzdk51obtain-client-config"><a
name="client-config"></a>Obtain client config</h3>
<p>An application publishes several useful details that can be used to manage
and communicate with the application instance.</p>
<p>You can use <code>registry</code> commands to get all published data.</p>
<p><strong>All published data</strong></p>
@@ -443,8 +453,8 @@ Client configurations are at <code>/ws/v
<p><strong>Quick links</strong></p>
<p>An application may publish some quick links for the monitoring UI, JMX
endpoint, etc. as the listening ports are dynamically allocated and thus need
to be published. These information is available at
<code>/ws/v1/slider/publisher/slider/quicklinks</code>.</p>
-<h3 id="wzxhzdk50wzxhzdk51manage-the-application-lifecycle"><a
name="manage"></a>Manage the Application Lifecycle</h3>
-<p>Once started, applications can be frozen/stopped, thawed/restarted, and
destroyed/removed as follows:</p>
+<h3 id="wzxhzdk52wzxhzdk53manage-the-application-lifecycle"><a
name="manage"></a>Manage the Application Lifecycle</h3>
+<p>Once started, applications can be stopped, (re)started, and destroyed as
follows:</p>
<h4 id="stopped">Stopped:</h4>
<div class="codehilite"><pre><span class="o">./</span><span
class="n">slider</span> <span class="n">stop</span> <span class="n">cl1</span>
</pre></div>
@@ -465,7 +475,7 @@ Client configurations are at <code>/ws/v
</pre></div>
-<h1 id="wzxhzdk52wzxhzdk53appendix-a-debugging-slider-agent"><a
name="appendixa"></a>Appendix A: Debugging Slider-Agent</h1>
+<h1 id="wzxhzdk54wzxhzdk55appendix-a-debugging-slider-agent"><a
name="appendixa"></a>Appendix A: Debugging Slider-Agent</h1>
<h2 id="create-and-deploy-slider-agent-configuration">Create and deploy Slider
Agent configuration</h2>
<p>Create an agent config file (agent.ini) based on the sample available
at:</p>
<div class="codehilite"><pre><span class="cp">${</span><span
class="n">slider</span><span class="o">-</span><span
class="n">install</span><span class="o">-</span><span
class="nb">dir</span><span
class="cp">}</span>/slider-0.60.0/agent/conf/agent.ini
Modified: websites/staging/slider/trunk/content/docs/manpage.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/manpage.html (original)
+++ websites/staging/slider/trunk/content/docs/manpage.html Tue Nov 25 19:04:14
2014
@@ -239,7 +239,8 @@ and existing but not running "stopped" a
</li>
<li>
<p>Users can flex an application instance: adding or removing components
dynamically.
-If the application instance is live, the changes will have immediate effect.
If not, the changes will be picked up when the instance is next thawed.</p>
+If the application instance is live, the changes will have immediate effect.
+If not, the changes will be picked up when the instance is next started.</p>
</li>
</ol>
<!--- =======================================================================
-->
@@ -269,7 +270,7 @@ If the application instance is live, the
<p>COMMANDS</p>
<h3 id="slider-build-name"><code>slider build <name></code></h3>
<p>Build an instance of the given name, with the specific options.</p>
-<p>It is not started; this can be done later with a <code>thaw</code>
command.</p>
+<p>It is not started; this can be done later with a <code>start</code>
command.</p>
<h3 id="slider-create-name"><code>slider create <name></code></h3>
<p>Build and run an application instance of the given name </p>
<h4 id="arguments-for-build-and-create"><strong>Arguments for
<code>build</code> and <code>create</code></strong></h4>
@@ -651,7 +652,6 @@ would be saved to the file <code>service
<p>This simplifies path creation, testing a, </p>
<h3 id="start-name-wait-milliseconds-out-filename"><code>start <name>
[--wait milliseconds] [--out <filename>]</code></h3>
-<p>(<strong>start</strong> used to be <strong>thaw</strong> in develop
branch)</p>
<p>Start a stopped application instance, recreating it from its previously
saved state.</p>
<p>After the application is launched, if an <code>--out</code> argument
specified a file,
@@ -676,7 +676,6 @@ specify the destination.</p>
<h3 id="stop-name-force-wait-time-message-text"><code>stop <name>
[--force] [--wait time] [--message text]</code></h3>
-<p>(<strong>stop</strong> used to be <strong>freeze</strong> in earlier
releases)</p>
<p>stop the application instance. The running application is stopped. Its
settings are retained in HDFS.</p>
<p>The <code>--wait</code> argument can specify a time in seconds to wait for
the application instance to be stopped.</p>
<p>The <code>--force</code> flag causes YARN asked directly to terminate the
application instance.
Modified: websites/staging/slider/trunk/content/docs/security.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/security.html (original)
+++ websites/staging/slider/trunk/content/docs/security.html Tue Nov 25
19:04:14 2014
@@ -360,7 +360,7 @@ token to talk the AM. </p>
wishes to talk with the SliderAM -a token which is only provided after
the caller authenticates itself as the user that has access rights
to the cluster</p>
-<p>To allow the client to freeze a Slider application instance while they are
unable to acquire
+<p>To allow the client to stop a Slider application instance while they are
unable to acquire
a token to authenticate with the AM, use the <code>--force</code> option.</p>
<h3 id="how-to-enable-a-secure-slider-client">How to enable a secure Slider
client</h3>
<p>Slider can be placed into secure mode by setting the Hadoop security
options:</p>
Modified:
websites/staging/slider/trunk/content/docs/slider_specs/canonical_scenarios.html
==============================================================================
---
websites/staging/slider/trunk/content/docs/slider_specs/canonical_scenarios.html
(original)
+++
websites/staging/slider/trunk/content/docs/slider_specs/canonical_scenarios.html
Tue Nov 25 19:04:14 2014
@@ -334,7 +334,7 @@ Figure 2 - Unmanaged Application Client
<p>Install the application - the installation command will take care of the
population of the application resources into the pre-determined application
resource directory in HDFS. The required configuration and binary directories
will also be created.</p>
</li>
<li>
-<p>start/thaw the application - Slider/Yarn runtime negotiates and
instantiates the number of component containers designated by the cluster
description and the components are started.</p>
+<p>start the application - Slider/Yarn runtime negotiates and instantiates the
number of component containers designated by the cluster description and the
components are started.</p>
</li>
<li>
<p>stop/freeze the application - similar to stopping, applicaiton (or a subset
of their components) can be stopped.</p>
Modified:
websites/staging/slider/trunk/content/docs/slider_specs/specifying_exports.html
==============================================================================
---
websites/staging/slider/trunk/content/docs/slider_specs/specifying_exports.html
(original)
+++
websites/staging/slider/trunk/content/docs/slider_specs/specifying_exports.html
Tue Nov 25 19:04:14 2014
@@ -191,7 +191,7 @@ Latest release: <strong>0.60.0-incubatin
<h3 id="which-component-is-responsible-for-export">Which component is
responsible for export</h3>
<p>By default an arbitrary master is chosen as the master responsible for
exporting the config. <em>What this means is that when this master is STARTED
the applied config known at that time is exported</em>. Otherwise, you can
specify which master component type should export configuration by specifying
<code><publishConfig>true</publishConfig></code> under
<code><component></code>.</p>
<h2 id="exporting-formatted-data-at-application-instance-level">Exporting
formatted data at application instance level</h2>
-<p>At application level one or more exportGroups can be defined where each
group can contain one or more exported name-value pairs. The sample export
group shown below exports two data items "app.monitor" and "nimbus.host_port"
for a Storm application instance.</p>
+<p>At application level one or more exportGroups can be defined where each
group can contain one or more exported name-value pairs. The sample export
group shown below exports two data items <code>app.monitor</code> and
<code>nimbus.host_port</code> for a Storm application instance.</p>
<div class="codehilite"><pre><span class="nt"><exportGroups></span>
<span class="nt"><exportGroup></span>
<span class="nt"><name></span>QuickLinks<span
class="nt"></name></span>
Modified: websites/staging/slider/trunk/content/downloads/index.html
==============================================================================
--- websites/staging/slider/trunk/content/downloads/index.html (original)
+++ websites/staging/slider/trunk/content/downloads/index.html Tue Nov 25
19:04:14 2014
@@ -180,30 +180,9 @@ Latest release: <strong>0.60.0-incubatin
<h1 id="apache-slider-incubating-downloads">Apache Slider (incubating)
Downloads</h1>
<p>Be sure to verify your downloads by these <a
href="https://www.apache.org/info/verification.html";
title="Verify">procedures</a> using these <a
href="https://www.apache.org/dist/incubator/slider/KEYS";
title="KEYS">KEYS</a>.</p>
<h2 id="current-release">Current Release</h2>
-<h3 id="0600-incubating">0.60.0-incubating</h3>
-<p>The most recent Apache Slider™ release is version 0.60.0-incubating.
See the <a href="/release_notes/release-0.60.0.html" title="0.60.0-incubating
Release Notes">release notes</a>.</p>
-<table class="table">
-<tr>
-<th>Release Artifact</th><th>Signature & Hashes</th></tr>
-<tr>
-<td><a
href="https://www.apache.org/dyn/closer.cgi/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.tar.gz";>apache-slider-0.60.0-incubating-source-release.tar.gz</a></td>
-<td>
-<a
href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.tar.gz.asc";>ASC</a>
-<a
href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.tar.gz.md5";>MD5</a>
-<a
href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.tar.gz.sha";>SHA</a>
-</td>
-</tr>
-<tr>
-<td><a
href="https://www.apache.org/dyn/closer.cgi/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.zip";>apache-slider-0.60.0-incubating-source-release.zip</a></td>
-<td>
-<a
href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.zip.asc";>ASC</a>
-<a
href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.zip.md5";>MD5</a>
-<a
href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.zip.sha";>SHA</a>
-</td>
-</tr>
-</table>
-
-<p>Older releases may be found in the <a
href="https://archive.apache.org/dist/incubator/slider";>archives</a>.</p>
+<p>The latest release of Apache Slider™ can be
+<a href="http://www.apache.org/dyn/closer.cgi/incubator/slider/";>downloaded
from the ASF</a></p>
+<p>Older releases may be found in the <a
href="https://archive.apache.org/dist/incubator/slider";>archives</a></p>
</div>
<div id="footer">
Modified: websites/staging/slider/trunk/content/index.html
==============================================================================
--- websites/staging/slider/trunk/content/index.html (original)
+++ websites/staging/slider/trunk/content/index.html Tue Nov 25 19:04:14 2014
@@ -181,9 +181,9 @@ Latest release: <strong>0.60.0-incubatin
<p>Apache Slider is a YARN application to deploy existing distributed
applications on YARN,
monitor them and make them larger or smaller as desired -even while
the application is running.</p>
-<p>Applications can be stopped, "frozen" and restarted, "thawed" later; the
distribution
-of the deployed application across the YARN cluster is persisted -enabling
-a best-effort placement close to the previous locations on a cluster thaw.
+<p>Applications can be stopped then started; the distribution
+of the deployed application across the YARN cluster is persisted âenabling
+a best-effort placement close to the previous locations.
Applications which remember the previous placement of data (such as HBase)
can exhibit fast start-up times from this feature.</p>
<p>YARN itself monitors the health of "YARN containers" hosting parts of
@@ -214,8 +214,8 @@ application itself</p>
<p>The Slider tool is a Java command line application.</p>
<p>The tool persists the information as JSON documents in HDFS.</p>
<p>Once the cluster has been started, the cluster can be made to grow or shrink
-using the Slider commands. The cluster can also be stopped, <em>frozen</em>
-and later resumed, <em>thawed</em>.</p>
+using the Slider commands. The cluster can also be stopped
+and later restarted.</p>
<p>Slider implements all its functionality through YARN APIs and the existing
application shell scripts. The goal of the application was to have minimal
code changes and as of this writing, it has required few changes.</p>
