Modified: websites/staging/slider/trunk/content/design/rolehistory.html
==============================================================================
--- websites/staging/slider/trunk/content/design/rolehistory.html (original)
+++ websites/staging/slider/trunk/content/design/rolehistory.html Mon Nov 2
17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,10 +196,10 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1
id="apache-slider-placement-how-slider-brings-back-nodes-in-the-same-location">Apache
Slider Placement: how Slider brings back nodes in the same location</h1>
-<h3 id="last-updated-2015-03-19">Last updated 2015-03-19</h3>
-<h2 id="changes">Changes</h2>
-<h3 id="slider-080-incubating">Slider 0.80-incubating</h3>
+<h1
id="apache-slider-placement-how-slider-brings-back-nodes-in-the-same-location">Apache
Slider Placement: how Slider brings back nodes in the same location<a
class="headerlink"
href="#apache-slider-placement-how-slider-brings-back-nodes-in-the-same-location"
title="Permanent link">¶</a></h1>
+<h3 id="last-updated-2015-03-19">Last updated 2015-03-19<a class="headerlink"
href="#last-updated-2015-03-19" title="Permanent link">¶</a></h3>
+<h2 id="changes">Changes<a class="headerlink" href="#changes" title="Permanent
link">¶</a></h2>
+<h3 id="slider-080-incubating">Slider 0.80-incubating<a class="headerlink"
href="#slider-080-incubating" title="Permanent link">¶</a></h3>
<p>A major rework of placement has taken place, <a
href="https://issues.apache.org/jira/browse/SLIDER-611";>Ãber-JIRA : placement
phase 2</a></p>
<ol>
<li>Slider manages the process of relaxing a request from a specific host to
"anywhere in the cluster".</li>
@@ -216,7 +227,7 @@ exit.</li>
</ul>
</li>
</ol>
-<h5 id="role-history-reloading-enhancements">Role History Reloading
Enhancements</h5>
+<h5 id="role-history-reloading-enhancements">Role History Reloading
Enhancements<a class="headerlink" href="#role-history-reloading-enhancements"
title="Permanent link">¶</a></h5>
<p>How persisted role history has also been improved
[SLIDER-600]((https://issues.apache.org/jira/browse/SLIDER-600)</p>
<ol>
<li>Reloading of persistent history has been made resilient to changes in the
number of roles.</li>
@@ -240,7 +251,7 @@ reloading entries there are two tactics<
<li>Delete the JSON files</li>
<li>Edit the most recent JSON file and delete any entry with the type
<code>RoleHistoryMapping</code>.</li>
</ol>
-<h4 id="placement-policies">Placement policies</h4>
+<h4 id="placement-policies">Placement policies<a class="headerlink"
href="#placement-policies" title="Permanent link">¶</a></h4>
<p><code>strict</code> placement</p>
<p>Again, "strict placement" has a different policy: once a component has been
deployed on a node,
one component request will be made against that node, even if it is considered
unreliable. No
@@ -252,7 +263,7 @@ While tracking of recent failure counts
<p>There's still no explicit support for this in YARN or slider. As noted
above, Slider does
try to spread placement when rebuilding an application, but otherwise it
accepts which
hosts YARN allocates containers on.</p>
-<h3 id="slider-070-incubating">Slider 0.70-incubating</h3>
+<h3 id="slider-070-incubating">Slider 0.70-incubating<a class="headerlink"
href="#slider-070-incubating" title="Permanent link">¶</a></h3>
<p>The Slider AM tracks the recent failure count of each role on each (known)
YARN node; when
the failure count of a role is above a role-specific threshold, no more
explicit requests are
made for that host. Exception: "strict placement" component deployments are
always requsted
@@ -261,7 +272,7 @@ against the node irrespective of their h
Slider's placement history indefinitely. The fact that a node was unreliable
two weeks ago should
not mean that it should be blacklisted this week. </p>
<p>Note that this failure history is not persisted; it is lost on AM
start/restart.</p>
-<h3 id="slider-060-incubating">Slider 0.60-incubating</h3>
+<h3 id="slider-060-incubating">Slider 0.60-incubating<a class="headerlink"
href="#slider-060-incubating" title="Permanent link">¶</a></h3>
<ol>
<li>Slider supports <code>strict placement</code> in which a request is when
made for a previously
used node and never used again.</li>
@@ -272,7 +283,7 @@ nodes. As such, it offers significant co
hardware to specific applications. Note that the HDFS filesystem is still
shared across the cluster,
so IO from other in-cluster applications will impinge on each other.</li>
</ol>
-<h2 id="introduction">Introduction</h2>
+<h2 id="introduction">Introduction<a class="headerlink" href="#introduction"
title="Permanent link">¶</a></h2>
<p>Slider needs to bring up instances of a given role on the machine(s) on
which
they last ran. It must remember after shrinking or freezing an Application
Instance which
servers were last used for a role. It must then try to use this (persisted)
data to select
@@ -289,7 +300,7 @@ tables.</p>
<p>Note that it does not need to care about the placement of other roles, such
as the HBase masters -there anti-affinity between other instances is
the key requirement.</p>
-<h3 id="terminology">Terminology</h3>
+<h3 id="terminology">Terminology<a class="headerlink" href="#terminology"
title="Permanent link">¶</a></h3>
<ul>
<li><strong>Node</strong> : A server in the YARN Physical (or potentially
virtual) cluster of servers.</li>
<li><strong>Application Instance</strong>: A deployment of a distributed
application as created and
@@ -304,7 +315,7 @@ and so retained across much of the code
manage its Application Instance.</li>
<li><strong>RM</strong> YARN Resource Manager</li>
</ul>
-<h3 id="assumptions">Assumptions</h3>
+<h3 id="assumptions">Assumptions<a class="headerlink" href="#assumptions"
title="Permanent link">¶</a></h3>
<p>Here are some assumptions in Slider's design</p>
<ol>
<li>
@@ -360,7 +371,7 @@ designing the RoleHistory datastructures
of rolling statistics on recent failures would be a first step to this </p>
</li>
</ol>
-<h3 id="the-rolehistory-datastructure">The RoleHistory Datastructure</h3>
+<h3 id="the-rolehistory-datastructure">The RoleHistory Datastructure<a
class="headerlink" href="#the-rolehistory-datastructure" title="Permanent
link">¶</a></h3>
<p>The <code>RoleHistory</code> is a datastructure which models the component
assignment, and
can persist it to and restore it from the (shared) filesystem.</p>
<ul>
@@ -409,7 +420,7 @@ outstanding request - are considered can
of the time-ordered list of nodes that last ran an instance of that
component</p>
</li>
</ul>
-<h3 id="persistence">Persistence</h3>
+<h3 id="persistence">Persistence<a class="headerlink" href="#persistence"
title="Permanent link">¶</a></h3>
<p>The state of the component is persisted to HDFS on changes âbut not on
Application Instance
termination.</p>
<ol>
@@ -433,7 +444,7 @@ might seem the approach most consistent
an easy structure to work with.</p>
<p>The initial implementation will use Apache Avro as the persistence format,
with the data saved in JSON or compressed format.</p>
-<h2 id="weaknesses-in-this-design">Weaknesses in this design</h2>
+<h2 id="weaknesses-in-this-design">Weaknesses in this design<a
class="headerlink" href="#weaknesses-in-this-design" title="Permanent
link">¶</a></h2>
<p><strong>Blacklisting</strong>: even if a node fails repeatedly, this design
will still try to re-request
instances on this node; there is no blacklisting. As a central blacklist
for YARN has been proposed, it is hoped that this issue will be addressed
centrally,
@@ -452,7 +463,7 @@ history of a node -maintaining some kind
node use and picking the heaviest used, or some other more-complex algorithm.
This may be possible, but we'd need evidence that the problem existed before
trying to address it.</p>
-<h1 id="the-nodemap">The NodeMap</h1>
+<h1 id="the-nodemap">The NodeMap<a class="headerlink" href="#the-nodemap"
title="Permanent link">¶</a></h1>
<p>A core data structure, the <code>NodeMap</code>, is a map of every known
node in the YARN cluster, tracking
how many containers are allocated to specific roles in it, and, when there
are no active instances, when it was last used. This history is used to
@@ -475,7 +486,7 @@ most containers on it. This would spread
<li>When there are no empty nodes to request containers on, a request would
let YARN choose.</li>
</ol>
-<h4 id="strengths">Strengths</h4>
+<h4 id="strengths">Strengths<a class="headerlink" href="#strengths"
title="Permanent link">¶</a></h4>
<ul>
<li>Handles the multi-container on one node problem</li>
<li>By storing details about every role, cross-role decisions could be
possible</li>
@@ -485,7 +496,7 @@ let YARN choose.</li>
<li>Easy to view and debug</li>
<li>Would support cross-role collection of node failures in future</li>
</ul>
-<h4 id="weaknesses">Weaknesses</h4>
+<h4 id="weaknesses">Weaknesses<a class="headerlink" href="#weaknesses"
title="Permanent link">¶</a></h4>
<ul>
<li>Size of the data structure is <code>O(nodes * role-instances</code>). This
could be mitigated by regular cleansing of the structure. For example, at
@@ -500,8 +511,8 @@ was satisfied on a different node, the o
Slider should query the RM (or topology scripts?) to determine if nodes are
still
parts of the YARN cluster. </li>
</ul>
-<h2 id="data-structures">Data Structures</h2>
-<h3 id="rolehistory">RoleHistory</h3>
+<h2 id="data-structures">Data Structures<a class="headerlink"
href="#data-structures" title="Permanent link">¶</a></h2>
+<h3 id="rolehistory">RoleHistory<a class="headerlink" href="#rolehistory"
title="Permanent link">¶</a></h3>
<div class="codehilite"><pre><span class="n">startTime</span><span
class="o">:</span> <span class="n">long</span>
<span class="n">saveTime</span><span class="o">:</span> <span
class="n">long</span>
<span class="n">dirty</span><span class="o">:</span> <span
class="n">boolean</span>
@@ -513,7 +524,7 @@ parts of the YARN cluster. </li>
<p>This is the aggregate data structure that is persisted to/from file</p>
-<h3 id="nodemap">NodeMap</h3>
+<h3 id="nodemap">NodeMap<a class="headerlink" href="#nodemap" title="Permanent
link">¶</a></h3>
<div class="codehilite"><pre><span class="n">clusterNodes</span><span
class="o">:</span> <span class="n">Map</span><span class="o">:</span> <span
class="n">NodeId</span> <span class="o">-></span> <span
class="n">NodeInstance</span>
<span class="n">clusterNodes</span><span class="o">():</span> <span
class="n">Iterable</span><span class="o"><</span><span
class="n">NodeInstance</span><span class="o">></span>
<span class="n">getOrCreate</span><span class="o">(</span><span
class="n">NodeId</span><span class="o">):</span> <span
class="n">NodeInstance</span>
@@ -521,7 +532,7 @@ parts of the YARN cluster. </li>
<p>Maps a YARN NodeID record to a Slider <code>NodeInstance</code>
structure</p>
-<h3 id="nodeinstance">NodeInstance</h3>
+<h3 id="nodeinstance">NodeInstance<a class="headerlink" href="#nodeinstance"
title="Permanent link">¶</a></h3>
<p>Every node in the YARN cluster is modeled as an ragged array of
<code>NodeEntry</code> instances, indexed
by role index -</p>
<div class="codehilite"><pre><span class="n">NodeEntry</span><span
class="p">[</span><span class="n">roles</span><span class="p">]</span>
@@ -535,7 +546,7 @@ by role index -</p>
<p>This could be implemented in a map or an indexed array; the array is more
efficient but it does mandate that the number of roles are bounded and
fixed.</p>
-<h3 id="nodeentry">NodeEntry</h3>
+<h3 id="nodeentry">NodeEntry<a class="headerlink" href="#nodeentry"
title="Permanent link">¶</a></h3>
<p>Records the details about all of a roles containers on a node. The
<code>active</code> field records the number of containers currently
active.</p>
<div class="codehilite"><pre><span class="n">active</span><span
class="o">:</span> <span class="n">int</span>
@@ -556,9 +567,9 @@ responses (which may come from pre-resta
unexpected container release responses as failures.</p>
<p>The <code>active</code> counter is only decremented after a container
release response
has been received.</p>
-<h3 id="rolestatus">RoleStatus</h3>
+<h3 id="rolestatus">RoleStatus<a class="headerlink" href="#rolestatus"
title="Permanent link">¶</a></h3>
<p>This is the existing
<code>org.apache.hoya.yarn.appmaster.state.RoleStatus</code> class</p>
-<h3 id="rolelist">RoleList</h3>
+<h3 id="rolelist">RoleList<a class="headerlink" href="#rolelist"
title="Permanent link">¶</a></h3>
<p>A list mapping role to int enum is needed to index NodeEntry elements in
the NodeInstance arrays. Although such an enum is already implemented in the
Slider
Providers, explicitly serializing and deserializing it would make
@@ -566,7 +577,7 @@ the persistent structure easier to parse
to changes in the number or position of roles.</p>
<p>This list could also retain information about recently used/released nodes,
so that the selection of containers to request could shortcut a search</p>
-<h3 id="containerpriority">ContainerPriority</h3>
+<h3 id="containerpriority">ContainerPriority<a class="headerlink"
href="#containerpriority" title="Permanent link">¶</a></h3>
<p>The container priority field (a 32 bit integer) is used by Slider (0.5.x)
to index the specific role in a container so as to determine which role
has been offered in a container allocation message, and which role has
@@ -584,7 +595,7 @@ The main requirement is: not have >
which places an upper bound on the size of the Application Instance.</p>
<p>The splitting and merging will be implemented in a ContainerPriority class,
for uniform access.</p>
-<h3 id="outstandingrequest">OutstandingRequest</h3>
+<h3 id="outstandingrequest">OutstandingRequest<a class="headerlink"
href="#outstandingrequest" title="Permanent link">¶</a></h3>
<p>Tracks an outstanding request. This is used to correlate an allocation
response
(whose Container Priority file is used to locate this request), with the
node and role used in the request.</p>
@@ -598,7 +609,7 @@ node and role used in the request.</p>
<p>The node identifier may be null -which indicates that a request was made
without
a specific target node</p>
-<h3 id="outstandingrequesttracker">OutstandingRequestTracker</h3>
+<h3 id="outstandingrequesttracker">OutstandingRequestTracker<a
class="headerlink" href="#outstandingrequesttracker" title="Permanent
link">¶</a></h3>
<p>Contains a map from requestID to the specific
<code>OutstandingRequest</code> made,
and generates the request ID</p>
<div class="codehilite"><pre><span class="n">nextRequestId</span><span
class="o">:</span> <span class="n">int</span>
@@ -617,7 +628,7 @@ and generates the request ID</p>
<p>The list operation can be implemented inefficiently unless it is found
to be important -if so a more complex structure will be needed.</p>
-<h3 id="availablenodes">AvailableNodes</h3>
+<h3 id="availablenodes">AvailableNodes<a class="headerlink"
href="#availablenodes" title="Permanent link">¶</a></h3>
<p>This is a field in <code>RoleHistory</code></p>
<div class="codehilite"><pre><span class="n">availableNodes</span><span
class="o">:</span> <span class="n">List</span><span class="o"><</span><span
class="n">NodeInstance</span><span class="o">>[]</span>
</pre></div>
@@ -654,7 +665,7 @@ as the target for the next resource requ
<p>Strategy #1 is simpler; Strategy #2 <em>may</em> decrease the affinity of
component placement,
as the AM will be explicitly requesting an instance on a node which it knows
is not running an instance of that role.</p>
-<h4 id="issue-what-to-do-about-failing-nodes">ISSUE What to do about failing
nodes?</h4>
+<h4 id="issue-what-to-do-about-failing-nodes">ISSUE What to do about failing
nodes?<a class="headerlink" href="#issue-what-to-do-about-failing-nodes"
title="Permanent link">¶</a></h4>
<p>Should a node whose container just failed be placed at the
top of the stack, ready for the next request? </p>
<p>If the container failed due to an unexpected crash in the application,
asking
@@ -664,10 +675,10 @@ back a new role instance on that machine
will not be satisfied by that node.</p>
<p>If there is a problem with the node, such that containers repeatedly fail
on it,
then re-requesting containers on it will amplify the damage.</p>
-<h2 id="actions">Actions</h2>
-<h3 id="bootstrap">Bootstrap</h3>
+<h2 id="actions">Actions<a class="headerlink" href="#actions" title="Permanent
link">¶</a></h2>
+<h3 id="bootstrap">Bootstrap<a class="headerlink" href="#bootstrap"
title="Permanent link">¶</a></h3>
<p>Persistent Role History file not found; empty data structures created.</p>
-<h3 id="restart">Restart</h3>
+<h3 id="restart">Restart<a class="headerlink" href="#restart" title="Permanent
link">¶</a></h3>
<p>When starting an Application Instance, the Role History should be loaded.
</p>
<p>If the history is missing <em>or cannot be loaded for any reason</em>,
Slider must revert to the bootstrap actions.</p>
@@ -710,7 +721,7 @@ The state of the Application Instance af
<p>After this operation, the structures are purged with all out of date
entries,
and the available node list contains a sorted list of the remainder.</p>
-<h3 id="am-restart">AM Restart</h3>
+<h3 id="am-restart">AM Restart<a class="headerlink" href="#am-restart"
title="Permanent link">¶</a></h3>
<p>1: Create the initial data structures as a normal start operation
2: update the structure with the list of live nodes, removing those nodes
from the list of available nodes</p>
@@ -730,7 +741,7 @@ from the list of available nodes</p>
<p>There's no need to resort the available node list âall that has happened
is that some entries have been removed</p>
-<h3 id="teardown">Teardown</h3>
+<h3 id="teardown">Teardown<a class="headerlink" href="#teardown"
title="Permanent link">¶</a></h3>
<ol>
<li>If dirty, save role history to its file.</li>
<li>Issue release requests</li>
@@ -743,7 +754,7 @@ the RoleHistory data to have been writte
should assume that the history was saved to disk at some point during the life
of the Application Instance âideally after the most recent change, and that
the information
in it is only an approximate about what the previous state of the Application
Instance was.</p>
-<h3 id="flex-requesting-a-container-in-role-role">Flex: Requesting a container
in role <code>role</code></h3>
+<h3 id="flex-requesting-a-container-in-role-role">Flex: Requesting a container
in role <code>role</code><a class="headerlink"
href="#flex-requesting-a-container-in-role-role" title="Permanent
link">¶</a></h3>
<div class="codehilite"><pre><span class="n">node</span> <span
class="p">=</span> <span class="n">availableNodes</span><span
class="p">[</span><span class="n">roleId</span><span class="p">].</span><span
class="n">pop</span><span class="p">()</span>
<span class="k">if</span> <span class="n">node</span> !<span
class="p">=</span> <span class="n">null</span> <span class="p">:</span>
<span class="n">node</span><span class="p">.</span><span
class="n">nodeEntry</span><span class="p">[</span><span
class="n">roleId</span><span class="p">].</span><span
class="n">requested</span><span class="o">++</span><span class="p">;</span>
@@ -771,7 +782,7 @@ As a result, the request will be downgra
request âan acceptable degradation on an Application Instance where all the
other entries
in the nodemap have instances of that specific node âbut not when there are
empty nodes. </p>
-<h4 id="solutions">Solutions</h4>
+<h4 id="solutions">Solutions<a class="headerlink" href="#solutions"
title="Permanent link">¶</a></h4>
<ol>
<li>
<p>Add some randomness in the search of the datastructure, rather than simply
@@ -790,7 +801,7 @@ even though it was recently requested".<
isn't currently satisfying requests.</p>
</li>
</ol>
-<h4 id="history-issues">History Issues</h4>
+<h4 id="history-issues">History Issues<a class="headerlink"
href="#history-issues" title="Permanent link">¶</a></h4>
<p>Without using that history, there is a risk that a very old assignment
is used in place of a recent one and the value of locality decreased.</p>
<p>But there are consequences:</p>
@@ -808,7 +819,7 @@ be rebuilt at startup time.</p>
<p>The proposed <code>recentlyReleasedList</code> addresses this, though it
creates
another data structure to maintain and rebuild at startup time
from the last-used fields in the node entries.</p>
-<h3 id="am-callback-oncontainersallocated">AM Callback :
onContainersAllocated</h3>
+<h3 id="am-callback-oncontainersallocated">AM Callback :
onContainersAllocated<a class="headerlink"
href="#am-callback-oncontainersallocated" title="Permanent link">¶</a></h3>
<div class="codehilite"><pre><span class="n">void</span> <span
class="n">onContainersAllocated</span><span class="p">(</span><span
class="n">List</span><span class="o"><</span><span
class="n">Container</span><span class="o">></span> <span
class="n">allocatedContainers</span><span class="p">)</span>
</pre></div>
@@ -881,7 +892,7 @@ re-inserted into the AvailableNodes list
in the total ordering of the list.</p>
</li>
</ol>
-<h3 id="nmclientasync-callback-oncontainerstarted">NMClientAsync Callback:
onContainerStarted()</h3>
+<h3 id="nmclientasync-callback-oncontainerstarted">NMClientAsync Callback:
onContainerStarted()<a class="headerlink"
href="#nmclientasync-callback-oncontainerstarted" title="Permanent
link">¶</a></h3>
<div class="codehilite"><pre><span class="n">onContainerStarted</span><span
class="p">(</span><span class="n">ContainerId</span> <span
class="n">containerId</span><span class="p">)</span>
</pre></div>
@@ -890,12 +901,12 @@ in the total ordering of the list.</p>
of starting containers, moving it into the map of live nodes; the counters
in the associated <code>RoleInstance</code> are updated accordingly; the node
entry
adjusted to indicate it has one more live node and one less starting node.</p>
-<h3 id="nmclientasync-callback-oncontainerstartfailed">NMClientAsync Callback:
onContainerStartFailed()</h3>
+<h3 id="nmclientasync-callback-oncontainerstartfailed">NMClientAsync Callback:
onContainerStartFailed()<a class="headerlink"
href="#nmclientasync-callback-oncontainerstartfailed" title="Permanent
link">¶</a></h3>
<p>The AM uses this as a signal to remove the container from the list
of starting containers âthe count of starting containers for the relevant
NodeEntry is decremented. If the node is now available for instances of this
container, it is returned to the queue of available nodes.</p>
-<h3
id="flex-releasing-a-component-instance-from-the-application-instance">Flex:
Releasing a Component instance from the Application Instance</h3>
+<h3
id="flex-releasing-a-component-instance-from-the-application-instance">Flex:
Releasing a Component instance from the Application Instance<a
class="headerlink"
href="#flex-releasing-a-component-instance-from-the-application-instance"
title="Permanent link">¶</a></h3>
<p>Simple strategy: find a node with at least one active container of that
type</p>
<div class="codehilite"><pre><span class="n">select</span> <span
class="n">a</span> <span class="n">node</span> <span class="n">N</span> <span
class="n">in</span> <span class="n">nodemap</span> <span class="n">where</span>
<span class="k">for</span> <span class="n">NodeEntry</span><span
class="p">[</span><span class="n">roleId</span><span class="p">]:</span> <span
class="n">active</span> <span class="o">></span> <span
class="n">releasing</span><span class="p">;</span>
<span class="n">nodeentry</span> <span class="p">=</span> <span
class="n">node</span><span class="p">.</span><span class="n">get</span><span
class="p">(</span><span class="n">roleId</span><span class="p">)</span>
@@ -919,7 +930,7 @@ must: be of the target role, and not alr
<li>The (existing) <code>containersBeingReleased</code> Map has the container
inserted into it</li>
</ol>
<p>After the AM processes the request, it triggers a callback</p>
-<h3 id="am-callback-oncontainerscompleted">AM callback
onContainersCompleted:</h3>
+<h3 id="am-callback-oncontainerscompleted">AM callback
onContainersCompleted:<a class="headerlink"
href="#am-callback-oncontainerscompleted" title="Permanent link">¶</a></h3>
<div class="codehilite"><pre><span class="n">void</span> <span
class="n">onContainersCompleted</span><span class="p">(</span><span
class="n">List</span><span class="o"><</span><span
class="n">ContainerStatus</span><span class="o">></span> <span
class="n">completedContainers</span><span class="p">)</span>
</pre></div>
@@ -972,7 +983,7 @@ on that node, it will have been inserted
As a result, it is highly likely that a new container will be requested on
the same node. (The only way a node the list would be newer is
be if other containers were completed in the same callback)</p>
-<h3 id="implementation-notes">Implementation Notes</h3>
+<h3 id="implementation-notes">Implementation Notes<a class="headerlink"
href="#implementation-notes" title="Permanent link">¶</a></h3>
<p>Notes made while implementing the design.</p>
<p><code>OutstandingRequestTracker</code> should also track requests made with
no target node; this makes seeing what is going on easier.
<code>ARMClientImpl</code>
@@ -999,13 +1010,13 @@ have been released? I.e. Before YARN ha
<p>RoleStats were removed âleft in app state. Although the rolestats would
belong here, leaving them where they were reduced the amount of change
in the <code>AppState</code> class, so risk of something breaking.</p>
-<h2 id="miniyarncluster-node-ids">MiniYARNCluster node IDs</h2>
+<h2 id="miniyarncluster-node-ids">MiniYARNCluster node IDs<a
class="headerlink" href="#miniyarncluster-node-ids" title="Permanent
link">¶</a></h2>
<p>Mini YARN cluster NodeIDs all share the same hostname , at least when
running
against file://; so mini tests with >1 NM don't have a 1:1 mapping of
<code>NodeId:NodeInstance</code>. What will happen is that
<code>NodeInstance getOrCreateNodeInstance(Container container) '
will always return the same (now shared)</code>NodeInstance`.</p>
-<h2 id="releasing-containers-when-shrinking-an-application-instance">Releasing
Containers when shrinking an Application Instance</h2>
+<h2 id="releasing-containers-when-shrinking-an-application-instance">Releasing
Containers when shrinking an Application Instance<a class="headerlink"
href="#releasing-containers-when-shrinking-an-application-instance"
title="Permanent link">¶</a></h2>
<p>When identifying instances to release in a bulk downscale operation, the
full
list of targets must be identified together. This is not just to eliminate
multiple scans of the data structures, but because the containers are not
@@ -1023,12 +1034,12 @@ a new datastructure is created to identi
(e.g a list of ContainerIds under each NodeEntry), or we add an index reference
in a RoleInstance that identifies the node. We already effectively have that
in the container</p>
-<h3 id="dropping-any-available-nodes-that-are-busy">dropping any available
nodes that are busy</h3>
+<h3 id="dropping-any-available-nodes-that-are-busy">dropping any available
nodes that are busy<a class="headerlink"
href="#dropping-any-available-nodes-that-are-busy" title="Permanent
link">¶</a></h3>
<p>When scanning the available list, any nodes that are no longer idle for that
role should be dropped from the list.</p>
<p>This can happen when an instance was allocated on a different node from
that requested.</p>
-<h3
id="finding-a-node-when-a-component-has-some-live-instances-and-no-available-hosts-known-to">Finding
a node when a component has some live instances and no available hosts known
to</h3>
+<h3
id="finding-a-node-when-a-component-has-some-live-instances-and-no-available-hosts-known-to">Finding
a node when a component has some live instances and no available hosts known
to<a class="headerlink"
href="#finding-a-node-when-a-component-has-some-live-instances-and-no-available-hosts-known-to"
title="Permanent link">¶</a></h3>
<p>Slider.</p>
<p>One condition found during testing is the following: </p>
<ol>
@@ -1054,7 +1065,7 @@ which are not running an instance of the
the request would have to omit those nodes for which an allocation has already
been
made off the available list and yet for which no container has yet been
granted]. </p>
-<h2 id="reworked-outstanding-request-tracker">Reworked Outstanding Request
Tracker</h2>
+<h2 id="reworked-outstanding-request-tracker">Reworked Outstanding Request
Tracker<a class="headerlink" href="#reworked-outstanding-request-tracker"
title="Permanent link">¶</a></h2>
<p>The reworked request tracker behaves as follows</p>
<ol>
<li>outstanding requests with specific placements are tracked by <code>(role,
hostname)</code></li>
@@ -1068,7 +1079,7 @@ then sorted.</li>
</ol>
<p>This strategy returns unused hosts to the list of possible hosts, while
retaining
the ordering of that list in most-recent-first.</p>
-<h3 id="weaknesses_1">Weaknesses</h3>
+<h3 id="weaknesses_1">Weaknesses<a class="headerlink" href="#weaknesses_1"
title="Permanent link">¶</a></h3>
<p>if one or more container requests cannot be satisifed, then all the hosts in
the set of outstanding requests will be retained, so all these hosts in the
will be considered unavailable for new location-specific requests.
@@ -1080,7 +1091,7 @@ same priority (i.e. same Slider Role ID)
of instances of the target role were decreated during a flex such that
the placement could now be satisfied on the target host. This is not considered
a significant problem.</p>
-<h1 id="persistence_1">Persistence</h1>
+<h1 id="persistence_1">Persistence<a class="headerlink" href="#persistence_1"
title="Permanent link">¶</a></h1>
<p>The initial implementation uses the JSON-formatted Avro format; while
significantly
less efficient than a binary format, it is human-readable</p>
<p>Here are sequence of entries from a test run on a single node YARN cluster;
running 1 HBase Master
@@ -1134,7 +1145,7 @@ No region servers are yet live.</p>
<p>The <code>last_used</code> timestamps will not be changed until the
Application Instance is shrunk or restarted, as the <code>active</code> flag
being set
implies that the server is running both roles at the save time of
<code>1384183512217</code>.</p>
-<h2 id="resolved-issues">Resolved issues</h2>
+<h2 id="resolved-issues">Resolved issues<a class="headerlink"
href="#resolved-issues" title="Permanent link">¶</a></h2>
<blockquote>
<p>How best to distinguish at startup time from nodes used recently
from nodes used some period before? Should the RoleHistory simply forget
Modified:
websites/staging/slider/trunk/content/design/specification/cli-actions.html
==============================================================================
--- websites/staging/slider/trunk/content/design/specification/cli-actions.html
(original)
+++ websites/staging/slider/trunk/content/design/specification/cli-actions.html
Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,29 +196,29 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-cli-actions">Apache Slider CLI Actions</h1>
-<h2 id="important">Important</h2>
+<h1 id="apache-slider-cli-actions">Apache Slider CLI Actions<a
class="headerlink" href="#apache-slider-cli-actions" title="Permanent
link">¶</a></h1>
+<h2 id="important">Important<a class="headerlink" href="#important"
title="Permanent link">¶</a></h2>
<ol>
<li>This document is still being updated from the original hoya design</li>
<li>The new cluster model of separated specification files for internal,
resource and application configuration
has not been incorporated.</li>
<li>What is up to date is the CLI command list and arguments</li>
</ol>
-<h2 id="client-configuration">client configuration</h2>
+<h2 id="client-configuration">client configuration<a class="headerlink"
href="#client-configuration" title="Permanent link">¶</a></h2>
<p>As well as the CLI options, the <code>conf/slider-client.xml</code> XML
file can define arguments used to communicate with the Application instance</p>
-<h4 id="fsdefaultfs"><code>fs.defaultFS</code></h4>
+<h4 id="fsdefaultfs"><code>fs.defaultFS</code><a class="headerlink"
href="#fsdefaultfs" title="Permanent link">¶</a></h4>
<p>Equivalent to setting the filesystem with <code>--filesystem</code></p>
-<h2 id="common">Common</h2>
-<h3 id="system-properties">System Properties</h3>
+<h2 id="common">Common<a class="headerlink" href="#common" title="Permanent
link">¶</a></h2>
+<h3 id="system-properties">System Properties<a class="headerlink"
href="#system-properties" title="Permanent link">¶</a></h3>
<p>Arguments of the form <code>-S key=value</code> define JVM system
properties.</p>
<p>These are supported primarily to define options needed for some Kerberos
configurations.</p>
-<h3 id="definitions">Definitions</h3>
+<h3 id="definitions">Definitions<a class="headerlink" href="#definitions"
title="Permanent link">¶</a></h3>
<p>Arguments of the form <code>-D key=value</code> define JVM system
properties.</p>
<p>These can define client options that are not set in
<code>conf/slider-client.xml</code> - or to override them.</p>
-<h3 id="cluster-names">Cluster names</h3>
+<h3 id="cluster-names">Cluster names<a class="headerlink"
href="#cluster-names" title="Permanent link">¶</a></h3>
<p>All actions that must take an instance name will fail with
<code>EXIT_UNKNOWN_INSTANCE</code>
if one is not provided.</p>
-<h2 id="action-build">Action: Build</h2>
+<h2 id="action-build">Action: Build<a class="headerlink" href="#action-build"
title="Permanent link">¶</a></h2>
<p>Builds a cluster -creates all the on-filesystem datastructures, and
generates a cluster description
that is both well-defined and deployable -<em>but does not actually start the
cluster</em></p>
<div class="codehilite"><pre><span class="n">build</span> <span
class="p">(</span><span class="n">instancename</span><span class="p">,</span>
@@ -226,7 +237,7 @@ that is both well-defined and deployable
</pre></div>
-<h4 id="preconditions">Preconditions</h4>
+<h4 id="preconditions">Preconditions<a class="headerlink"
href="#preconditions" title="Permanent link">¶</a></h4>
<p>(Note that the ordering of these preconditions is not guaranteed to remain
constant)</p>
<p>The instance name is valid</p>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">not</span> <span class="n">valid</span><span class="o">-</span><span
class="n">instance</span><span class="o">-</span><span
class="n">name</span><span class="p">(</span><span
class="n">instancename</span><span class="p">)</span> <span class="p">:</span>
<span class="n">raise</span> <span class="n">SliderException</span><span
class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span
class="p">)</span>
@@ -260,7 +271,7 @@ If it exists when another process wishes
afterwards. A process attempting to acquire the writelock must check for the
existence of this file before AND after creating the
writelock file, failing if its present. This retains a small race condition: a
second or later reader may still be reading the data
when a process successfully acquires the write lock. If this proves to be an
issue, a stricter model could be implemented, with each reading process
creating a unique named readlock- file.</p>
-<h4 id="postconditions">Postconditions</h4>
+<h4 id="postconditions">Postconditions<a class="headerlink"
href="#postconditions" title="Permanent link">¶</a></h4>
<p>All the instance directories exist</p>
<div class="codehilite"><pre><span class="n">is</span><span
class="o">-</span><span class="n">dir</span><span class="p">(</span><span
class="n">HDFS</span><span class="o">'</span><span class="p">,</span> <span
class="n">instance</span><span class="o">-</span><span
class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span
class="o">'</span><span class="p">,</span> <span
class="n">instancename</span><span class="p">))</span>
<span class="n">is</span><span class="o">-</span><span
class="n">dir</span><span class="p">(</span><span class="n">HDFS</span><span
class="o">'</span><span class="p">,</span> <span
class="n">original</span><span class="o">-</span><span
class="n">conf</span><span class="o">-</span><span class="n">path</span><span
class="p">(</span><span class="n">HDFS</span><span class="o">'</span><span
class="p">,</span> <span class="n">instancename</span><span class="p">))</span>
@@ -354,14 +365,14 @@ the command-line supplied option will ov
</pre></div>
-<h2 id="action-start">Action: Start</h2>
+<h2 id="action-start">Action: Start<a class="headerlink" href="#action-start"
title="Permanent link">¶</a></h2>
<div class="codehilite"><pre><span class="n">start</span> <span
class="o"><</span><span class="n">instancename</span><span
class="o">></span> <span class="p">[</span><span class="o">--</span><span
class="n">wait</span> <span class="o"><</span><span
class="n">timeout</span><span class="o">></span><span class="p">]</span>
</pre></div>
<p>Takes an application instance with configuration and (possibly) data on
disk, and
attempts to create a live application with the specified number of nodes</p>
-<h4 id="preconditions_1">Preconditions</h4>
+<h4 id="preconditions_1">Preconditions<a class="headerlink"
href="#preconditions_1" title="Permanent link">¶</a></h4>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">not</span> <span class="n">valid</span><span class="o">-</span><span
class="n">instance</span><span class="o">-</span><span
class="n">name</span><span class="p">(</span><span
class="n">instancename</span><span class="p">)</span> <span class="p">:</span>
<span class="n">raise</span> <span class="n">SliderException</span><span
class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span
class="p">)</span>
</pre></div>
@@ -383,7 +394,7 @@ attempts to create a live application wi
</pre></div>
-<h3 id="postconditions_1">Postconditions</h3>
+<h3 id="postconditions_1">Postconditions<a class="headerlink"
href="#postconditions_1" title="Permanent link">¶</a></h3>
<p>After the start operation has been performed, there is now a queued request
in YARN
for the chosen (how?) queue</p>
<div class="codehilite"><pre><span class="n">YARN</span><span
class="o">'</span><span class="p">.</span><span
class="n">Queues</span><span class="o">'</span><span
class="p">[</span><span class="n">amqueue</span><span class="p">]</span> <span
class="p">=</span> <span class="n">YARN</span><span class="p">.</span><span
class="n">Queues</span><span class="p">[</span><span
class="n">amqueue</span><span class="p">]</span> <span class="o">+</span> <span
class="p">[</span><span class="n">launch</span><span
class="p">(</span>"<span class="n">slider</span>"<span
class="p">,</span> <span class="n">instancename</span><span class="p">,</span>
<span class="n">requirements</span><span class="p">,</span> <span
class="n">context</span><span class="p">)]</span>
@@ -398,17 +409,17 @@ the application has failed</p>
</pre></div>
-<h2 id="outcome-am-launched-state">Outcome: AM-launched state</h2>
+<h2 id="outcome-am-launched-state">Outcome: AM-launched state<a
class="headerlink" href="#outcome-am-launched-state" title="Permanent
link">¶</a></h2>
<p>Some time after the AM was queued, if the relevant
prerequisites of the launch request are met, the AM will be deployed</p>
-<h4 id="preconditions_2">Preconditions</h4>
+<h4 id="preconditions_2">Preconditions<a class="headerlink"
href="#preconditions_2" title="Permanent link">¶</a></h4>
<ul>
<li>The resources referenced in HDFS (still) are accessible by the user</li>
<li>The requested YARN memory and core requirements could be met on the YARN
cluster and
specific YARN application queue.</li>
<li>There is sufficient capacity in the YARN cluster to create a container for
the AM.</li>
</ul>
-<h4 id="postconditions_2">Postconditions</h4>
+<h4 id="postconditions_2">Postconditions<a class="headerlink"
href="#postconditions_2" title="Permanent link">¶</a></h4>
<p>Define a YARN state at a specific time <code>t</code> as
<code>YARN(t)</code>; the fact that
an AM is launched afterwards</p>
<p>The AM is deployed if there is some time <code>t</code> after the
submission time <code>t0</code>
@@ -429,7 +440,7 @@ requests using resources as they become
<p>For tests on a dedicated YARN cluster, a few tens of seconds appear to be
enough
for the AM-launched state to be reached, a failure to occur, or to conclude
that the resource requirements are unsatisfiable.</p>
-<h2 id="outcome-am-started-state">Outcome: AM-started state</h2>
+<h2 id="outcome-am-started-state">Outcome: AM-started state<a
class="headerlink" href="#outcome-am-started-state" title="Permanent
link">¶</a></h2>
<p>A (usually short) time after the AM is launched, it should start</p>
<ul>
<li>The node hosting the container is working reliably</li>
@@ -444,13 +455,13 @@ with an error code.</li>
</ul>
<p>Node failures/command line failures are treated by YARN as an AM failure
which
will trigger a restart attempt -this may be on the same or a different
node.</p>
-<h4 id="preconditions_3">preconditions</h4>
+<h4 id="preconditions_3">preconditions<a class="headerlink"
href="#preconditions_3" title="Permanent link">¶</a></h4>
<p>The AM was launched at an earlier time, <code>t1</code></p>
<div class="codehilite"><pre><span class="n">exists</span> <span
class="n">t1</span> <span class="n">where</span> <span class="n">t1</span>
<span class="o">></span> <span class="n">t0</span> <span
class="n">and</span> <span class="n">am</span><span class="o">-</span><span
class="n">launched</span><span class="p">(</span><span
class="n">YARN</span><span class="p">(</span><span class="n">t1</span><span
class="p">)</span>
</pre></div>
-<h4 id="postconditions_3">Postconditions</h4>
+<h4 id="postconditions_3">Postconditions<a class="headerlink"
href="#postconditions_3" title="Permanent link">¶</a></h4>
<p>The application is actually started if it is listed in the YARN application
list
as being in the state <code>RUNNING</code>, an RPC port has been registered
with YARN (visible as the <code>rpcPort</code>
attribute in the YARN Application Report,and that port is servicing RPC
requests
@@ -466,7 +477,7 @@ from authenticated callers.</p>
<p>A test for accepting cluster requests is querying the cluster status
with <code>SliderClusterProtocol.getJSONClusterStatus()</code>. If this returns
a parseable cluster description, the AM considers itself live.</p>
-<h2 id="outcome-applicaton-instance-operational-state">Outcome: Applicaton
Instance operational state</h2>
+<h2 id="outcome-applicaton-instance-operational-state">Outcome: Applicaton
Instance operational state<a class="headerlink"
href="#outcome-applicaton-instance-operational-state" title="Permanent
link">¶</a></h2>
<p>Once started, Slider enters the operational state of trying to keep the
numbers
of live role instances matching the numbers specified in the cluster
specification.</p>
<p>The AM must request the a container for each desired instance of a specific
roles of the
@@ -476,7 +487,7 @@ the specific application roles on the al
cluster size is dynamically updated.</p>
<p>The AM releases containers when the cluster size is shrunk during a flex
operation,
or during teardown.</p>
-<h3 id="steady-state-condition">steady state condition</h3>
+<h3 id="steady-state-condition">steady state condition<a class="headerlink"
href="#steady-state-condition" title="Permanent link">¶</a></h3>
<p>The steady state of a Slider cluster is that the number of live instances
of a role,
plus the number of requested instances , minus the number of instances for
which release requests have been made must match that of the desired
number.</p>
@@ -499,10 +510,10 @@ history of notifications received from Y
<p>Slider does not consider it an error if the number of actual instances
remains below
the desired value (i.e. outstanding requests are not being satisfied) -this is
an operational state of the cluster that Slider cannot address.</p>
-<h3 id="cluster-startup">Cluster startup</h3>
+<h3 id="cluster-startup">Cluster startup<a class="headerlink"
href="#cluster-startup" title="Permanent link">¶</a></h3>
<p>On a healthy dedicated test cluster, the time for the requests to be
satisfied is
a few tens of seconds at most: a failure to achieve this state is a sign of a
problem.</p>
-<h3 id="node-or-process-failure">Node or process failure</h3>
+<h3 id="node-or-process-failure">Node or process failure<a class="headerlink"
href="#node-or-process-failure" title="Permanent link">¶</a></h3>
<p>After a container or node failure, a new container for a new instance of
that role
is requested.</p>
<p>The failure count is incremented -it can be accessed via the
<code>"role.failed.instances"</code>
@@ -518,7 +529,7 @@ option: <code>"slider.container.failure.
</pre></div>
-<h3 id="instance-startup-failure">Instance startup failure</h3>
+<h3 id="instance-startup-failure">Instance startup failure<a
class="headerlink" href="#instance-startup-failure" title="Permanent
link">¶</a></h3>
<p>Startup failures are measured alongside general node failures.</p>
<p>A container is deemed to have failed to start if either of the following
conditions
were met:</p>
@@ -537,10 +548,10 @@ startup failures differently from ongoin
treated as a sign that the container is failing to launch the program reliably
-
either the generated command line is invalid, or the application is failing
to run/exiting on or nearly immediately.</p>
-<h2 id="action-create">Action: Create</h2>
+<h2 id="action-create">Action: Create<a class="headerlink"
href="#action-create" title="Permanent link">¶</a></h2>
<p>Create is simply <code>build</code> + <code>start</code> in sequence - the
postconditions from the first
action are intended to match the preconditions of the second.</p>
-<h2 id="action-stop">Action: Stop</h2>
+<h2 id="action-stop">Action: Stop<a class="headerlink" href="#action-stop"
title="Permanent link">¶</a></h2>
<div class="codehilite"><pre><span class="n">stop</span> <span
class="n">instancename</span> <span class="p">[</span><span
class="o">--</span><span class="n">wait</span> <span class="n">time</span><span
class="p">]</span> <span class="p">[</span><span class="o">--</span><span
class="n">message</span> <span class="n">message</span><span class="p">]</span>
</pre></div>
@@ -549,7 +560,7 @@ action are intended to match the precond
cluster are stopped, leaving all the persistent state.</p>
<p>The operation is intended to be idempotent: it is not an error if
<code>stop</code> is invoked on an already stopped cluster</p>
-<h4 id="preconditions_4">Preconditions</h4>
+<h4 id="preconditions_4">Preconditions<a class="headerlink"
href="#preconditions_4" title="Permanent link">¶</a></h4>
<p>The cluster name is valid and it matches a known cluster </p>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">not</span> <span class="n">valid</span><span class="o">-</span><span
class="n">instance</span><span class="o">-</span><span
class="n">name</span><span class="p">(</span><span
class="n">instancename</span><span class="p">)</span> <span class="p">:</span>
<span class="n">raise</span> <span class="n">SliderException</span><span
class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span
class="p">)</span>
@@ -558,7 +569,7 @@ cluster are stopped, leaving all the per
</pre></div>
-<h4 id="postconditions_4">Postconditions</h4>
+<h4 id="postconditions_4">Postconditions<a class="headerlink"
href="#postconditions_4" title="Permanent link">¶</a></h4>
<p>If the cluster was running, an RPC call has been sent to it
<code>stopCluster(message)</code></p>
<p>If the <code>--wait</code> argument specified a wait time, then the command
will block
until the cluster has finished or the wait time was exceeded. </p>
@@ -569,7 +580,7 @@ YARN logs as the reason the cluster was
</pre></div>
-<h2 id="action-flex">Action: Flex</h2>
+<h2 id="action-flex">Action: Flex<a class="headerlink" href="#action-flex"
title="Permanent link">¶</a></h2>
<p>Flex the cluster size: add or remove roles. </p>
<div class="codehilite"><pre><span class="n">flex</span> <span
class="n">instancename</span>
<span class="n">components</span><span class="p">:</span><span
class="n">List</span><span class="p">[(</span><span
class="n">String</span><span class="p">,</span> <span class="n">int</span><span
class="p">)]</span>
@@ -581,13 +592,13 @@ YARN logs as the reason the cluster was
<li>if the cluster is running, it is given the new cluster specification,
which will change the desired steady-state of the application</li>
</ol>
-<h4 id="preconditions_5">Preconditions</h4>
+<h4 id="preconditions_5">Preconditions<a class="headerlink"
href="#preconditions_5" title="Permanent link">¶</a></h4>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">not</span> <span class="n">is</span><span class="o">-</span><span
class="n">file</span><span class="p">(</span><span class="n">HDFS</span><span
class="p">,</span> <span class="n">cluster</span><span class="o">-</span><span
class="n">json</span><span class="o">-</span><span class="n">path</span><span
class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span
class="n">instancename</span><span class="p">))</span> <span class="p">:</span>
<span class="n">raise</span> <span class="n">SliderException</span><span
class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span
class="p">)</span>
</pre></div>
-<h4 id="postconditions_5">Postconditions</h4>
+<h4 id="postconditions_5">Postconditions<a class="headerlink"
href="#postconditions_5" title="Permanent link">¶</a></h4>
<div class="codehilite"><pre><span class="n">let</span> <span
class="n">originalSpec</span> <span class="p">=</span> <span
class="n">data</span><span class="p">(</span><span class="n">HDFS</span><span
class="p">,</span> <span class="n">cluster</span><span class="o">-</span><span
class="n">json</span><span class="o">-</span><span class="n">path</span><span
class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span
class="n">instancename</span><span class="p">))</span>
<span class="n">let</span> <span class="n">updatedSpec</span> <span
class="p">=</span> <span class="n">originalspec</span> <span
class="n">where</span><span class="p">:</span>
@@ -599,7 +610,7 @@ which will change the desired steady-sta
</pre></div>
-<h4 id="am-actions-on-flex">AM actions on flex</h4>
+<h4 id="am-actions-on-flex">AM actions on flex<a class="headerlink"
href="#am-actions-on-flex" title="Permanent link">¶</a></h4>
<div class="codehilite"><pre><span class="n">boolean</span> <span
class="n">SliderAppMaster</span><span class="p">.</span><span
class="n">flexCluster</span><span class="p">(</span><span
class="n">ClusterDescription</span> <span class="n">updatedSpec</span><span
class="p">)</span>
</pre></div>
@@ -608,12 +619,12 @@ which will change the desired steady-sta
then <code>AppState</code> is updated with the new desired role counts. The
operation will
return once all requests to add or remove role instances have been queued,
and be <code>True</code> iff the desired steady state of the cluster has been
changed.</p>
-<h4 id="preconditions_6">Preconditions</h4>
+<h4 id="preconditions_6">Preconditions<a class="headerlink"
href="#preconditions_6" title="Permanent link">¶</a></h4>
<div class="codehilite"><pre> <span class="n">well</span><span
class="o">-</span><span class="n">defined</span><span class="o">-</span><span
class="n">application</span><span class="o">-</span><span
class="n">instance</span><span class="p">(</span><span
class="n">HDFS</span><span class="p">,</span> <span
class="n">updatedSpec</span><span class="p">)</span>
</pre></div>
-<h4 id="postconditions_6">Postconditions</h4>
+<h4 id="postconditions_6">Postconditions<a class="headerlink"
href="#postconditions_6" title="Permanent link">¶</a></h4>
<div class="codehilite"><pre><span class="n">forall</span> <span
class="n">role</span> <span class="n">in</span> <span
class="n">AppState</span><span class="p">.</span><span
class="n">Roles</span><span class="p">.</span><span class="n">keys</span><span
class="p">:</span>
<span class="n">AppState</span><span class="o">'</span><span
class="p">.</span><span class="n">Roles</span><span class="o">'</span><span
class="p">[</span><span class="n">role</span><span class="p">].</span><span
class="n">desiredCount</span> <span class="p">=</span> <span
class="n">updatedSpec</span><span class="p">[</span><span
class="n">roles</span><span class="p">][</span>"<span
class="n">yarn</span><span class="p">.</span><span
class="n">component</span><span class="p">.</span><span
class="n">instances</span>"<span class="p">]</span>
<span class="n">result</span> <span class="p">=</span> <span
class="n">AppState</span><span class="o">'</span> !<span class="p">=</span>
<span class="n">AppState</span>
@@ -624,35 +635,35 @@ and be <code>True</code> iff the desired
case the relevant requests will have been queued by the completion of the
action. It is not possible to state whether or when the requests will be
satisfied.</p>
-<h2 id="action-destroy">Action: Destroy</h2>
+<h2 id="action-destroy">Action: Destroy<a class="headerlink"
href="#action-destroy" title="Permanent link">¶</a></h2>
<p>Idempotent operation to destroy a stopped cluster -it succeeds if the
cluster has already been destroyed/is unknown, but not if it is
actually running.</p>
-<h4 id="preconditions_7">Preconditions</h4>
+<h4 id="preconditions_7">Preconditions<a class="headerlink"
href="#preconditions_7" title="Permanent link">¶</a></h4>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">not</span> <span class="n">valid</span><span class="o">-</span><span
class="n">instance</span><span class="o">-</span><span
class="n">name</span><span class="p">(</span><span
class="n">instancename</span><span class="p">)</span> <span class="p">:</span>
<span class="n">raise</span> <span class="n">SliderException</span><span
class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span
class="p">)</span>
<span class="k">if</span> <span class="n">slider</span><span
class="o">-</span><span class="n">instance</span><span class="o">-</span><span
class="n">live</span><span class="p">(</span><span class="n">YARN</span><span
class="p">,</span> <span class="n">instancename</span><span class="p">)</span>
<span class="p">:</span> <span class="n">raise</span> <span
class="n">SliderException</span><span class="p">(</span><span
class="n">EXIT_CLUSTER_IN_USE</span><span class="p">)</span>
</pre></div>
-<h4 id="postconditions_7">Postconditions</h4>
+<h4 id="postconditions_7">Postconditions<a class="headerlink"
href="#postconditions_7" title="Permanent link">¶</a></h4>
<p>The cluster directory and all its children do not exist</p>
<div class="codehilite"><pre><span class="n">not</span> <span
class="n">is</span><span class="o">-</span><span class="n">dir</span><span
class="p">(</span><span class="n">HDFS</span><span class="o">'</span><span
class="p">,</span> <span class="n">application</span><span
class="o">-</span><span class="n">instance</span><span class="o">-</span><span
class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span
class="o">'</span><span class="p">,</span> <span
class="n">instancename</span><span class="p">))</span>
</pre></div>
-<h2 id="action-status">Action: Status</h2>
+<h2 id="action-status">Action: Status<a class="headerlink"
href="#action-status" title="Permanent link">¶</a></h2>
<div class="codehilite"><pre><span class="n">status</span> <span
class="n">instancename</span> <span class="p">[</span><span
class="o">--</span><span class="n">out</span> <span
class="n">outfile</span><span class="p">]</span>
2
</pre></div>
-<h4 id="preconditions_8">Preconditions</h4>
+<h4 id="preconditions_8">Preconditions<a class="headerlink"
href="#preconditions_8" title="Permanent link">¶</a></h4>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">not</span> <span class="n">slider</span><span class="o">-</span><span
class="n">instance</span><span class="o">-</span><span
class="n">live</span><span class="p">(</span><span class="n">YARN</span><span
class="p">,</span> <span class="n">instancename</span><span class="p">)</span>
<span class="p">:</span> <span class="n">raise</span> <span
class="n">SliderException</span><span class="p">(</span><span
class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
</pre></div>
-<h4 id="postconditions_8">Postconditions</h4>
+<h4 id="postconditions_8">Postconditions<a class="headerlink"
href="#postconditions_8" title="Permanent link">¶</a></h4>
<p>The status of the application has been successfully queried and printed
out:</p>
<div class="codehilite"><pre><span class="n">let</span> <span
class="n">status</span> <span class="p">=</span> <span
class="n">slider</span><span class="o">-</span><span class="n">live</span><span
class="o">-</span><span class="n">instances</span><span class="p">(</span><span
class="n">YARN</span><span class="p">).</span><span
class="n">rpcPort</span><span class="p">.</span><span
class="n">getJSONClusterStatus</span><span class="p">()</span>
</pre></div>
@@ -669,18 +680,18 @@ actually running.</p>
</pre></div>
-<h2 id="action-exists">Action: Exists</h2>
+<h2 id="action-exists">Action: Exists<a class="headerlink"
href="#action-exists" title="Permanent link">¶</a></h2>
<p>This probes for a named cluster being defined or actually being in the
running
state.</p>
<p>In the running state; it is essentially the status
operation with only the exit code returned</p>
-<h4 id="preconditions_9">Preconditions</h4>
+<h4 id="preconditions_9">Preconditions<a class="headerlink"
href="#preconditions_9" title="Permanent link">¶</a></h4>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">not</span> <span class="n">is</span><span class="o">-</span><span
class="n">file</span><span class="p">(</span><span class="n">HDFS</span><span
class="p">,</span> <span class="n">application</span><span
class="o">-</span><span class="n">instance</span><span class="o">-</span><span
class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span
class="p">,</span> <span class="n">instancename</span><span class="p">))</span>
<span class="p">:</span>
<span class="n">raise</span> <span class="n">SliderException</span><span
class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span
class="p">)</span>
</pre></div>
-<h4 id="postconditions_9">Postconditions</h4>
+<h4 id="postconditions_9">Postconditions<a class="headerlink"
href="#postconditions_9" title="Permanent link">¶</a></h4>
<p>The operation succeeds if the cluster is running and the RPC call returns
the cluster
status.</p>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">live</span> <span class="n">and</span> <span class="n">not</span>
<span class="n">slider</span><span class="o">-</span><span
class="n">instance</span><span class="o">-</span><span
class="n">live</span><span class="p">(</span><span class="n">YARN</span><span
class="p">,</span> <span class="n">instancename</span><span class="p">):</span>
@@ -690,7 +701,7 @@ status.</p>
</pre></div>
-<h2 id="action-getconf">Action: getConf</h2>
+<h2 id="action-getconf">Action: getConf<a class="headerlink"
href="#action-getconf" title="Permanent link">¶</a></h2>
<p>This returns the live client configuration of the cluster -the
site-xml file.</p>
<div class="codehilite"><pre><span class="n">getconf</span> <span
class="o">--</span><span class="n">format</span> <span class="p">(</span><span
class="n">xml</span><span class="o">|</span><span
class="k">properties</span><span class="p">)</span> <span
class="o">--</span><span class="n">out</span> <span class="p">[</span><span
class="n">outfile</span><span class="p">]</span>
@@ -698,12 +709,12 @@ site-xml file.</p>
<p><em>We may want to think hard about whether this is needed</em></p>
-<h4 id="preconditions_10">Preconditions</h4>
+<h4 id="preconditions_10">Preconditions<a class="headerlink"
href="#preconditions_10" title="Permanent link">¶</a></h4>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">not</span> <span class="n">slider</span><span class="o">-</span><span
class="n">instance</span><span class="o">-</span><span
class="n">live</span><span class="p">(</span><span class="n">YARN</span><span
class="p">,</span> <span class="n">instancename</span><span class="p">)</span>
<span class="p">:</span> <span class="n">raise</span> <span
class="n">SliderException</span><span class="p">(</span><span
class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
</pre></div>
-<h4 id="postconditions_10">Postconditions</h4>
+<h4 id="postconditions_10">Postconditions<a class="headerlink"
href="#postconditions_10" title="Permanent link">¶</a></h4>
<p>The operation succeeds if the cluster status can be retrieved and saved to
the named file/printed to stdout in the format chosen</p>
<div class="codehilite"><pre><span class="n">let</span> <span
class="n">status</span> <span class="p">=</span> <span
class="n">slider</span><span class="o">-</span><span class="n">live</span><span
class="o">-</span><span class="n">instances</span><span class="p">(</span><span
class="n">YARN</span><span class="p">).</span><span
class="n">rpcPort</span><span class="p">.</span><span
class="n">getJSONClusterStatus</span><span class="p">()</span>
@@ -720,13 +731,13 @@ the named file/printed to stdout in the
</pre></div>
-<h2 id="action-list">Action: list</h2>
+<h2 id="action-list">Action: list<a class="headerlink" href="#action-list"
title="Permanent link">¶</a></h2>
<div class="codehilite"><pre><span class="n">list</span> <span
class="p">[</span><span class="n">instancename</span><span class="p">]</span>
</pre></div>
<p>Lists all clusters of a user, or only the one given</p>
-<h4 id="preconditions_11">Preconditions</h4>
+<h4 id="preconditions_11">Preconditions<a class="headerlink"
href="#preconditions_11" title="Permanent link">¶</a></h4>
<p>If a instancename is specified it must be in YARNs list of active or
completed applications
of that user:</p>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">instancename</span> !<span class="p">=</span> "" <span
class="n">and</span> <span class="p">[]</span> <span class="o">==</span> <span
class="n">yarn</span><span class="o">-</span><span
class="n">application</span><span class="o">-</span><span
class="n">instances</span><span class="p">(</span><span
class="n">YARN</span><span class="p">,</span> <span
class="n">instancename</span><span class="p">,</span> <span
class="n">user</span><span class="p">)</span>
@@ -734,7 +745,7 @@ of that user:</p>
</pre></div>
-<h4 id="postconditions_11">Postconditions</h4>
+<h4 id="postconditions_11">Postconditions<a class="headerlink"
href="#postconditions_11" title="Permanent link">¶</a></h4>
<p>If no instancename was given, all slider applications of that user are
listed,
else only the one running (or one of the finished ones)</p>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">instancename</span> <span class="o">==</span> "" <span
class="p">:</span>
@@ -746,7 +757,7 @@ else only the one running (or one of the
</pre></div>
-<h2 id="action-killcontainer">Action: killcontainer</h2>
+<h2 id="action-killcontainer">Action: killcontainer<a class="headerlink"
href="#action-killcontainer" title="Permanent link">¶</a></h2>
<p>This is an operation added for testing. It will kill a container in the
cluster
<em>without flexing the cluster size</em>. As a result, the cluster will
detect the
failure and attempt to recover from the failure by instantiating a new instance
@@ -755,7 +766,7 @@ of the cluster</p>
</pre></div>
-<h4 id="preconditions_12">Preconditions</h4>
+<h4 id="preconditions_12">Preconditions<a class="headerlink"
href="#preconditions_12" title="Permanent link">¶</a></h4>
<div class="codehilite"><pre><span class="k">if</span> <span
class="n">not</span> <span class="n">slider</span><span class="o">-</span><span
class="n">instance</span><span class="o">-</span><span
class="n">live</span><span class="p">(</span><span class="n">YARN</span><span
class="p">,</span> <span class="n">instancename</span><span class="p">)</span>
<span class="p">:</span> <span class="n">raise</span> <span
class="n">SliderException</span><span class="p">(</span><span
class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
<span class="n">exists</span> <span class="n">c</span> <span
class="n">in</span> <span class="n">slider</span><span class="o">-</span><span
class="n">app</span><span class="o">-</span><span
class="n">containers</span><span class="p">(</span><span
class="n">YARN</span><span class="p">,</span> <span
class="n">instancename</span><span class="p">,</span> <span
class="n">user</span><span class="p">)</span> <span class="n">where</span>
<span class="n">c</span><span class="p">.</span><span class="n">id</span> <span
class="o">==</span> <span class="n">container</span><span
class="o">-</span><span class="n">id</span>
@@ -765,7 +776,7 @@ of the cluster</p>
</pre></div>
-<h4 id="postconditions_12">Postconditions</h4>
+<h4 id="postconditions_12">Postconditions<a class="headerlink"
href="#postconditions_12" title="Permanent link">¶</a></h4>
<p>The container is not in the list of containers in the cluster</p>
<div class="codehilite"><pre><span class="n">not</span> <span
class="n">exists</span> <span class="n">c</span> <span class="n">in</span>
<span class="n">containers</span><span class="p">(</span><span
class="n">YARN</span><span class="p">)</span> <span class="n">where</span>
<span class="n">c</span><span class="p">.</span><span class="n">id</span> <span
class="o">==</span> <span class="n">container</span><span
class="o">-</span><span class="n">id</span>
</pre></div>
Modified: websites/staging/slider/trunk/content/design/specification/index.html
==============================================================================
--- websites/staging/slider/trunk/content/design/specification/index.html
(original)
+++ websites/staging/slider/trunk/content/design/specification/index.html Mon
Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="specification-of-apache-slider-behaviour">Specification of Apache
Slider behaviour</h1>
+<h1 id="specification-of-apache-slider-behaviour">Specification of Apache
Slider behaviour<a class="headerlink"
href="#specification-of-apache-slider-behaviour" title="Permanent
link">¶</a></h1>
<p>This is a a "more rigorous" definition of the behavior of Slider in terms
of its state and its command-line operations -by defining a 'formal' model
of HDFS, YARN and Slider's internal state, then describing the operations
Modified:
websites/staging/slider/trunk/content/design/specification/slider-model.html
==============================================================================
---
websites/staging/slider/trunk/content/design/specification/slider-model.html
(original)
+++
websites/staging/slider/trunk/content/design/specification/slider-model.html
Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,9 +196,9 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="formal-apache-slider-model">Formal Apache Slider Model</h1>
+<h1 id="formal-apache-slider-model">Formal Apache Slider Model<a
class="headerlink" href="#formal-apache-slider-model" title="Permanent
link">¶</a></h1>
<p>This is the model of Slider and YARN for the rest of the specification.</p>
-<h2 id="file-system">File System</h2>
+<h2 id="file-system">File System<a class="headerlink" href="#file-system"
title="Permanent link">¶</a></h2>
<p>A File System <code>HDFS</code> represents a Hadoop FileSystem -either HDFS
or another File
System which spans the cluster. There are also other filesystems that
can act as sources of data that is then copied into HDFS. These will be marked
@@ -201,7 +212,7 @@ Two key references are</p>
<li><a
href="https://github.com/steveloughran/hadoop-trunk/blob/stevel/HADOOP-9361-filesystem-contract/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md";>The
model of the filesystem</a></li>
</ol>
<p>The model and its predicates and invariants will be used in these
specifications.</p>
-<h2 id="yarn">YARN</h2>
+<h2 id="yarn">YARN<a class="headerlink" href="#yarn" title="Permanent
link">¶</a></h2>
<p>From the perspective of YARN application, The YARN runtime is a state,
<code>YARN</code>,
comprised of: <code>(Apps, Queues, Nodes)</code></p>
<div class="codehilite"><pre><span class="n">Apps</span><span
class="o">:</span> <span class="n">Map</span><span class="o">[</span><span
class="n">AppId</span><span class="o">,</span> <span
class="n">ApplicationReport</span><span class="o">]</span>
@@ -265,7 +276,7 @@ all nodes</p>
</pre></div>
-<h3 id="operations-predicates-used-the-specifications">Operations &
predicates used the specifications</h3>
+<h3 id="operations-predicates-used-the-specifications">Operations &
predicates used the specifications<a class="headerlink"
href="#operations-predicates-used-the-specifications" title="Permanent
link">¶</a></h3>
<div class="codehilite"><pre><span class="n">def</span> <span
class="n">applications</span><span class="p">(</span><span
class="n">YARN</span><span class="p">,</span> <span class="n">type</span><span
class="p">)</span> <span class="p">=</span>
<span class="p">[</span> <span class="n">app</span><span
class="p">.</span><span class="n">report</span> <span class="k">for</span>
<span class="n">app</span> <span class="n">in</span> <span
class="n">YARN</span><span class="p">.</span><span class="n">Apps</span><span
class="p">.</span><span class="n">values</span> <span class="n">where</span>
<span class="n">app</span><span class="p">.</span><span
class="n">report</span><span class="p">.</span><span class="n">Type</span>
<span class="o">==</span> <span class="n">type</span><span class="p">]</span>
@@ -274,7 +285,7 @@ all nodes</p>
</pre></div>
-<h2 id="usergroupinformation">UserGroupInformation</h2>
+<h2 id="usergroupinformation">UserGroupInformation<a class="headerlink"
href="#usergroupinformation" title="Permanent link">¶</a></h2>
<p>Applications are launched and executed on hosts computers: either client
machines
or nodes in the cluster, these have their own state which may need modeling</p>
<div class="codehilite"><pre><span class="n">HostState</span><span
class="o">:</span> <span class="n">Map</span><span class="o">[</span><span
class="n">String</span><span class="o">,</span> <span
class="n">String</span><span class="o">]</span>
@@ -290,8 +301,8 @@ access to the filesystem and to parts of
If it did they would be used throughout the specification to bind to a YARN or
HDFS instance.</p>
<p><code>UserGroupInformation.getCurrentUser(): UserGroupInformation</code></p>
<p>Returns the current user information. This information is immutable and
fixed for the duration of the process.</p>
-<h2 id="slider-model">Slider Model</h2>
-<h3 id="cluster-name">Cluster name</h3>
+<h2 id="slider-model">Slider Model<a class="headerlink" href="#slider-model"
title="Permanent link">¶</a></h2>
+<h3 id="cluster-name">Cluster name<a class="headerlink" href="#cluster-name"
title="Permanent link">¶</a></h3>
<p>A valid cluster name is a name of length > 1 which follows the internet
hostname scheme of letter followed by letter or digit</p>
<div class="codehilite"><pre><span class="n">def</span> <span
class="n">valid</span><span class="o">-</span><span
class="n">cluster</span><span class="o">-</span><span
class="n">name</span><span class="p">(</span><span class="n">c</span><span
class="p">)</span> <span class="p">=</span>
<span class="n">len</span><span class="p">(</span><span
class="n">c</span><span class="p">)</span><span class="o">></span> 0
@@ -300,7 +311,7 @@ If it did they would be used throughout
</pre></div>
-<h3 id="persistent-cluster-state">Persistent Cluster State</h3>
+<h3 id="persistent-cluster-state">Persistent Cluster State<a
class="headerlink" href="#persistent-cluster-state" title="Permanent
link">¶</a></h3>
<p>A Slider cluster's persistent state is stored in a path</p>
<div class="codehilite"><pre><span class="n">def</span> <span
class="n">cluster</span><span class="o">-</span><span
class="n">path</span><span class="p">(</span><span class="n">FS</span><span
class="p">,</span> <span class="n">clustername</span><span class="p">)</span>
<span class="p">=</span> <span class="n">user</span><span
class="o">-</span><span class="n">home</span><span class="p">(</span><span
class="n">FS</span><span class="p">)</span> <span class="o">+</span> <span
class="p">[</span>"<span class="n">clusters</span>"<span
class="p">,</span> <span class="n">clustername</span><span class="p">]</span>
<span class="n">def</span> <span class="n">cluster</span><span
class="o">-</span><span class="n">json</span><span class="o">-</span><span
class="n">path</span><span class="p">(</span><span class="n">FS</span><span
class="p">,</span> <span class="n">clustername</span><span class="p">)</span>
<span class="p">=</span> <span class="n">cluster</span><span
class="o">-</span><span class="n">path</span><span class="p">(</span><span
class="n">FS</span><span class="p">,</span> <span
class="n">clustername</span><span class="p">)</span> <span class="o">+</span>
<span class="p">[</span>"<span class="n">cluster</span><span
class="p">.</span><span class="n">json</span>"<span class="p">]</span>
@@ -346,7 +357,7 @@ specific instance bindings and saved int
</pre></div>
-<h3
id="invariant-there-must-never-be-more-than-one-running-instance-of-a-named-slider-cluster">Invariant:
there must never be more than one running instance of a named Slider
cluster</h3>
+<h3
id="invariant-there-must-never-be-more-than-one-running-instance-of-a-named-slider-cluster">Invariant:
there must never be more than one running instance of a named Slider cluster<a
class="headerlink"
href="#invariant-there-must-never-be-more-than-one-running-instance-of-a-named-slider-cluster"
title="Permanent link">¶</a></h3>
<p>There must never be more than one instance of the same Slider cluster
running:</p>
<div class="codehilite"><pre><span class="n">forall</span> <span
class="n">a</span> <span class="n">in</span> <span class="n">user</span><span
class="o">-</span><span class="n">applications</span><span
class="p">(</span><span class="n">YARN</span><span class="p">,</span>
"<span class="n">slider</span>"<span class="p">,</span> <span
class="n">user</span><span class="p">):</span>
<span class="n">len</span><span class="p">(</span><span
class="n">slider</span><span class="o">-</span><span class="n">app</span><span
class="o">-</span><span class="n">running</span><span class="o">-</span><span
class="n">instances</span><span class="p">(</span><span
class="n">YARN</span><span class="p">,</span> <span class="n">a</span><span
class="p">.</span><span class="n">Name</span><span class="p">,</span> <span
class="n">user</span><span class="p">))</span> <span class="o"><</span><span
class="p">=</span> 1
@@ -355,7 +366,7 @@ specific instance bindings and saved int
<p>There may be multiple instances in a finished state, and one running
instance alongside multiple finished instances -the applications
that work with Slider MUST select a running cluster ahead of any terminated
clusters.</p>
-<h3 id="containers-of-an-application">Containers of an application</h3>
+<h3 id="containers-of-an-application">Containers of an application<a
class="headerlink" href="#containers-of-an-application" title="Permanent
link">¶</a></h3>
<p>The containers of a slider application are the set of containers of that
application</p>
<div class="codehilite"><pre><span class="n">def</span> <span
class="n">slider</span><span class="o">-</span><span class="n">app</span><span
class="o">-</span><span class="n">containers</span><span
class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span
class="n">clustername</span><span class="p">,</span> <span
class="n">user</span><span class="p">)</span> <span class="p">=</span>
<span class="n">app</span><span class="o">-</span><span
class="n">containers</span><span class="p">(</span><span
class="n">YARN</span><span class="p">,</span> <span class="n">appid</span>
<span class="n">where</span>
@@ -363,7 +374,7 @@ that work with Slider MUST select a runn
</pre></div>
-<h3 id="rpc-access-to-a-slider-cluster">RPC Access to a slider cluster</h3>
+<h3 id="rpc-access-to-a-slider-cluster">RPC Access to a slider cluster<a
class="headerlink" href="#rpc-access-to-a-slider-cluster" title="Permanent
link">¶</a></h3>
<p>An application is accepting RPC requests for a given protocol if there is a
port binding
defined and it is possible to authenticate a connection using the specified
protocol</p>
<div class="codehilite"><pre> <span class="n">def</span> <span
class="n">rpc</span><span class="o">-</span><span
class="n">connection</span><span class="p">(</span><span
class="n">appReport</span><span class="p">,</span> <span
class="n">protocol</span><span class="p">)</span> <span class="p">=</span>
@@ -375,9 +386,9 @@ that work with Slider MUST select a runn
<p>Being able to open an RPC port is the strongest definition of liveness
possible
to make: if the AM responds to RPC operations, it is doing useful work.</p>
-<h3 id="valid-cluster-description">Valid Cluster Description</h3>
+<h3 id="valid-cluster-description">Valid Cluster Description<a
class="headerlink" href="#valid-cluster-description" title="Permanent
link">¶</a></h3>
<p>The <code>cluster.json</code> file of a cluster configures Slider to deploy
the application. </p>
-<h4
id="well-defined-clustercluster-description">well-defined-cluster(cluster-description)</h4>
+<h4
id="well-defined-clustercluster-description">well-defined-cluster(cluster-description)<a
class="headerlink" href="#well-defined-clustercluster-description"
title="Permanent link">¶</a></h4>
<p>A Cluster Description is well-defined if it is valid JSON and required
properties are present</p>
<p><strong>OBSOLETE</strong></p>
<p>Irrespective of specific details for deploying the Slider AM or any
provider-specific role instances,
@@ -396,7 +407,7 @@ is well-defined if</p>
<code>org.apache.slider.api.ClusterDescription</code> </p>
<p>Currently Slider ignores unknown elements during parsing. This may be
changed.</p>
<p>The test for this state does not refer to the cluster filesystem</p>
-<h4 id="deployable-clusterfs-cluster-description">deployable-cluster(FS,
cluster-description)</h4>
+<h4 id="deployable-clusterfs-cluster-description">deployable-cluster(FS,
cluster-description)<a class="headerlink"
href="#deployable-clusterfs-cluster-description" title="Permanent
link">¶</a></h4>
<p>A Cluster Description defines a deployable cluster if it is well-defined
cluster and the contents contain valid information to deploy a cluster</p>
<p>This defines how a cluster description is valid in the extends the valid
configuration with </p>
<ul>
@@ -434,7 +445,7 @@ and cast to <code>SliderProviderFactory<
</li>
</ul>
-<h4
id="valid-for-providercluster-description-provider">valid-for-provider(cluster-description,
provider)</h4>
+<h4
id="valid-for-providercluster-description-provider">valid-for-provider(cluster-description,
provider)<a class="headerlink"
href="#valid-for-providercluster-description-provider" title="Permanent
link">¶</a></h4>
<p>A provider considers a specification valid if its own validation logic is
satisfied. This normally
consists of rules about the number of instances of different roles; it may
include other logic.</p>
</div>
