Author: buildbot
Date: Tue May 19 15:33:19 2015
New Revision: 951892
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/docs/slider_specs/application_pkg_upgrade.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Tue May 19 15:33:19 2015
@@ -1 +1 @@
-1680194
+1680318
Modified:
websites/staging/slider/trunk/content/docs/slider_specs/application_pkg_upgrade.html
==============================================================================
---
websites/staging/slider/trunk/content/docs/slider_specs/application_pkg_upgrade.html
(original)
+++
websites/staging/slider/trunk/content/docs/slider_specs/application_pkg_upgrade.html
Tue May 19 15:33:19 2015
@@ -192,25 +192,43 @@ to <code>1.1.0</code>, then sufficient <
at the time of the upgrade. Application owners also need to ensure that their
application is compatible
in a mixed version mode. So, while the <strong><em>regionservers</em></strong>
are getting upgraded, the <strong><em>master</em></strong> will
still be the older version. Hence <code>1.1.0</code>
<strong><em>regionserver</em></strong> should be backward compatible with
<code>0.98.4</code>
-<strong><em>masters</em></strong>.</p>
+<strong><em>master</em></strong>.</p>
<p>Currently, <em>Ambari Slider View</em> supports automated deployment and
management of application packages.
Automated application package upgrade will be supported soon.</p>
<h2 id="phases-of-upgradedowngrade">Phases of Upgrade/Downgrade</h2>
<h4 id="yarn-core-libraries-and-configurations-upgradedowngrade">YARN core
(libraries and configurations) upgrade/downgrade</h4>
+<blockquote>
<p>Running Slider apps will continue to run, with no downtime</p>
+</blockquote>
<h4 id="slider-client-upgradedowngrade">Slider client upgrade/downgrade</h4>
-<p>Does not affect running Slider apps at all. New version of client can
co-exist with older versions.</p>
+<blockquote>
+<p>Does not affect running Slider apps at all. New version of client can
co-exist with older versions of client.</p>
+</blockquote>
<h4
id="slider-application-master-upgradedowngrade-of-running-applications">Slider
Application Master upgrade/downgrade of running applications</h4>
+<blockquote>
<p>Applications started prior to the start of YARN core upgrade/downgrade,
will continue to run with the
-older versions of Slider and Hadoop core libraries. To upgrade running Slider
AMs the application needs
-to be stopped and restarted with the new version of the client. There is no
support for rolling upgrade
-of Slider AM.</p>
+older versions of Slider core and Hadoop libraries. There is no support for
rolling upgrade of Slider AM.
+To upgrade running Slider AMs, the application needs to be stopped and
restarted with the new version
+of the client.</p>
+</blockquote>
<h4
id="applications-deployed-by-slider-binaries-and-configurations-upgradedowngrade">Applications
deployed by Slider (binaries and configurations) upgrade/downgrade</h4>
+<blockquote>
<p>This is the what this document is primarily about.</p>
+</blockquote>
<h2 id="upgrade-slider-application-packages">Upgrade Slider Application
Packages</h2>
<p>A run-book style list of atomic steps exposed by Slider. These steps will
be automated
by Apache Ambari in a future release. It can be easily orchestrated by a shell
script
or executed manually in the correct order.</p>
+<p>Note, the following YARN property should be specified to a reasonably high
value (say 5 or 10) before performing
+application upgrade. The default value of this property in a YARN cluster is
set to 2, which is not recommended
+for long running applications (specifically when performing upgrade).</p>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span
class="nt"><name></span>yarn.resourcemanager.am.max-attempts<span
class="nt"></name></span>
+ <span class="nt"><value></span>10<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
+
+
<h3
id="an-application-owner-brings-a-new-package-and-creates-a-new-instance">An
application owner brings a new package and creates a new instance</h3>
<p>Upload a new application package
<em>slider-hbase-app-package_v1.0.zip</em></p>
<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">package</span> <span class="o">--</span><span
class="n">install</span> <span class="o">--</span><span class="n">name</span>
<span class="n">MyHBase_Facebook</span> <span class="o">--</span><span
class="n">version</span> 1<span class="p">.</span>0 <span
class="o">--</span><span class="n">package</span> <span
class="o">~/</span><span class="n">slider</span><span class="o">-</span><span
class="n">hbase</span><span class="o">-</span><span class="n">app</span><span
class="o">-</span><span class="n">package_v1</span><span
class="p">.</span>0<span class="p">.</span><span class="n">zip</span>
@@ -218,12 +236,12 @@ or executed manually in the correct orde
<p>Create an application <em>MyHBase_Facebook_Finance</em></p>
-<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">create</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">template</span> <span
class="o">~/</span><span class="n">myHBase_template_v1</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span> <span
class="o">--</span><span class="n">resources</span> <span
class="o">~/</span><span class="n">myHBase_resources_v1</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span>
+<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">create</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">template</span> <span
class="o">~/</span><span class="n">myHBase_appConfig_v1</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span> <span
class="o">--</span><span class="n">resources</span> <span
class="o">~/</span><span class="n">myHBase_resources_v1</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span>
</pre></div>
-<p>Application <em>MyHBase_Facebook_Finance</em> goes through its usual
lifecycle</p>
-<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">flex</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">component</span> <span
class="n">HBASE_MASTER</span> 5
+<p>Application <em>MyHBase_Facebook_Finance</em> goes through its usual
lifecycle, say flex <em>regionservers</em> up</p>
+<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">flex</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">component</span> <span
class="n">HBASE_REGIONSERVER</span> 5
</pre></div>
@@ -239,16 +257,16 @@ or executed manually in the correct orde
<h3
id="at-some-point-the-app-owner-has-a-newer-version-of-the-package-and-needs-to-upgrade">At
some point, the app owner has a newer version of the package and needs to
upgrade</h3>
-<p>Uploads the newer version 2.0 of the app package</p>
+<p>Upload the newer version 2.0 of the app package</p>
<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">package</span> <span class="o">--</span><span
class="n">install</span> <span class="o">--</span><span class="n">name</span>
<span class="n">MyHBase_Facebook</span> <span class="o">--</span><span
class="n">version</span> 2<span class="p">.</span>0 <span
class="o">--</span><span class="n">package</span> <span
class="o">~/</span><span class="n">slider</span><span class="o">-</span><span
class="n">hbase</span><span class="o">-</span><span class="n">app</span><span
class="o">-</span><span class="n">package_v2</span><span
class="p">.</span>0<span class="p">.</span><span class="n">zip</span>
</pre></div>
-<p>This command upgrades the internal state of Slider App Master with the new
config and resource
-specifications. After this, upgrade command needs to be issued against all
containers to upgrade
+<p>The following command upgrades the internal state of Slider App Master with
the new config and resource
+specifications. After this, upgrade command needs to be issued against all
containers, to upgrade
them to the new app version e.g. v2.0 in this case. Note, it is recommended to
not issue the
-flex command, although Slider does not explicitly block it.</p>
-<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">upgrade</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">template</span> <span
class="o">~/</span><span class="n">myHBase_template_v2</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span> <span
class="o">--</span><span class="n">resources</span> <span
class="o">~/</span><span class="n">myHBase_resources_v2</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span>
+flex command until all containers are upgraded, although Slider does not
explicitly block it.</p>
+<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">upgrade</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">template</span> <span
class="o">~/</span><span class="n">myHBase_appConfig_v2</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span> <span
class="o">--</span><span class="n">resources</span> <span
class="o">~/</span><span class="n">myHBase_resources_v2</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span>
</pre></div>
@@ -256,11 +274,11 @@ flex command, although Slider does not e
<code>--containers</code> option (with additional filter options
<code>--components</code> and/or <code>--version</code>) to find all
the containers that are running for the application. The orchestrator needs to
decide how it wants to
span/schedule the upgrade commands of every single container. This will
primarily depend on the
-type of the application. For e.g. say <em>HBase</em> might want to call
upgrade on all <em>Master</em> containers
+type of the application. For e.g. say, <em>HBase</em> might want to call
upgrade on the <em>Master</em> container(s)
first before calling upgrade on any of the <em>RegionServer</em> containers.
<em>Storm</em> might want to provide
a 5 min delay between the upgrade of the <em>Nimbus</em> and
<em>Supervisor</em> container(s).</p>
<p>Upgrade all containers with ids <code>id1, id2, .. idn</code>. It is
possible that all these n containers can
-actually be down at the same time. If this is not desired, then issue n
upgrade command with a
+actually be down at the same time. If this is not desired, then issue n
upgrade commands with a
single container id at a time.</p>
<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">upgrade</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">containers</span> <span class="n">id1</span>
<span class="n">id2</span> <span class="p">..</span> <span class="n">idn</span>
@@ -278,9 +296,12 @@ one container at a time.</p>
<h3 id="when-do-we-know-that-upgrade-is-complete">When do we know that upgrade
is complete?</h3>
-<p>The upgrade command need to be executed against all containers and upgrade
can be considered
-complete only if the command <code>slider list MyHBase_Facebook_Finance
--containers --version 1.0</code><br />
-returns no containers.</p>
+<p>The upgrade command needs to be executed against all containers. Upgrade
can be considered
+complete only if the following command returns no containers.</p>
+<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">list</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">containers</span> <span
class="o">--</span><span class="n">version</span> 1<span class="p">.</span>0
+</pre></div>
+
+
<p>The app owner can also call <code>slider list MyHBase_Facebook_Finance
--containers</code> to explicitly
check that all expected containers are running and with the desired new
version (e.g. 2.0 in
this case).</p>
@@ -295,11 +316,11 @@ specifications. It needs to be followed
were upgraded to the newer version.</p>
<p>This command although called upgrade, actually downgrades the internal
state of Slider App
Master with the old v1.0 config and resource specifications</p>
-<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">upgrade</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">template</span> <span
class="o">~/</span><span class="n">myHBase_template_v1</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span> <span
class="o">--</span><span class="n">resources</span> <span
class="o">~/</span><span class="n">myHBase_resources_v1</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span>
+<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">upgrade</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">template</span> <span
class="o">~/</span><span class="n">myHBase_appConfig_v1</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span> <span
class="o">--</span><span class="n">resources</span> <span
class="o">~/</span><span class="n">myHBase_resources_v1</span><span
class="p">.</span>0<span class="p">.</span><span class="n">json</span>
</pre></div>
-<p>Take the list of all containers returned by the list command -</p>
+<p>Gather the list of all containers returned by the <em>list</em> command
-</p>
<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">list</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">containers</span> <span
class="o">--</span><span class="n">version</span> 2<span class="p">.</span>0
</pre></div>
@@ -309,36 +330,52 @@ Master with the old v1.0 config and reso
</pre></div>
+<p>Downgrade is done when the following command returns no containers.</p>
+<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">list</span> <span class="n">MyHBase_Facebook_Finance</span> <span
class="o">--</span><span class="n">containers</span> <span
class="o">--</span><span class="n">version</span> 2<span class="p">.</span>0
+</pre></div>
+
+
<h2 id="pre-and-post-upgrade-hooks">Pre and post upgrade hooks</h2>
<h3 id="pre-upgrade-hook-optional">Pre-upgrade hook (optional)</h3>
+<blockquote>
<p>The pre-upgrade steps, if provided, will allow applications to execute
simple housekeeping tasks
before Slider actually calls stop operation in an upgrade scenario
(specifically if they need to
be performed in every single container and 1000s of them are running). An
example could be to
send a message to a queue that the current instance of memcached is going down
so that the load
balancer rules can be dynamically updated. Performing long-running tasks or
tasks that needs to
be executed by only 1 instance of a component (n instances of which are
running) should be
-performed manually before calling upgrade (or maybe before calling the upgrade
command). The
+performed manually before starting the upgrade process. The
pre-upgrade hook is not a good candidate for such operations. Additional
parameters might have
to be exposed, like timeout, which can be used to wait at most n seconds (say)
after which Slider
will call the application stop hook even if the pre-upgrade operation is not
completed.</p>
-<h3 id="post-upgrade-hook-optional-not-yet-supported">Post-upgrade hook
(optional) - not yet supported</h3>
+<p>Use <code>app-packages/hbase/package/scripts/hbase_master.py</code> as a
sample for defining the
+pre-upgrade hook. Note, the pre-upgrade hook will be triggered only if the
currently running
+application has been created using Slider verion
<code>0.80.0-incubating</code> or later and the scripts
+in the package has the pre-upgrade hook defined.</p>
+</blockquote>
+<h3 id="post-upgrade-hook-optional-not-yet-supported">Post-upgrade hook
(optional) - <code>not yet supported</code></h3>
+<blockquote>
<p>This allows applications to perform simple housekeeping tasks prior to
calling start on the new
version of the application component. This is helpful only if such tasks are
required to be
-performed in every single container and 1000s of them are running. This is
required only in the
-upgrade scenario and should not be called on new containers created using flex
up in non-upgrade
-scenarios. This is tricky, how will Slider determine this? I donât think we
need a post-upgrade
-hook? Clarify with Rakesh and remove?</p>
+performed in every single container and 1000s of them are running. This hook
will be triggered only in the
+upgrade scenario. It will not be called on new containers created using flex
up, in non-upgrade
+scenarios. This makes triggering of post-upgrade hook a little tricky and
hence is not supported
+today. This will be looked into, in future releases.</p>
+</blockquote>
<h2 id="upgrade-support-of-simple-apps-with-no-packages">Upgrade support of
simple apps with no packages</h2>
<ul>
<li>Upgrade of such simple apps does not involve binaries</li>
<li>Upgrade of such simple apps basically means re-configuration only</li>
<li>The following can be done to perform the reconfiguration (or upgrade if
you prefer to call so)<ul>
-<li>Download the app package that was internally created by Slider from HDFS
(First class Slider
- command-line support will not be provided for download. So refer to
documentation or get
- Slider/Yarn admin support to find the app package location in HDFS.)</li>
+<li>Download the app package that was internally created by Slider from HDFS.
This is the path
+ defined in <code>application.def</code> property in the
<em>appConfig.json</em> file. Slider client command-line
+ support is not available to perform this download (yet). So, refer to
documentation or get
+ Slider/Yarn admin support if required.</li>
+<li>Unzip the application package</li>
<li>Make necessary changes to metainfo.json and repackage the app</li>
<li>Make necessary updates to appConfig.json and resources.json</li>
-<li>Follow the upgrade steps provided in this document to upgrade the simple
app</li>
+<li>Re-package the application package (this is the newer version of your
package, say v2.0)</li>
+<li>Follow the upgrade steps provided in this document to upgrade the simple
app to v2.0</li>
</ul>
</li>
</ul>
@@ -350,11 +387,11 @@ hook? Clarify with Rakesh and remove?</p
(on which the upgrade command was not called yet) fails, then its
replacement container will
come up with the newer version of the application.</li>
<li>It is recommended to not call <code>flex</code> (to increase or decrease
the no of containers assigned
- to a specific role) after the upgrade command has been issued and until
upgrade command is
+ to a specific role) after the upgrade spec command has been issued and until
upgrade command is
issued on all currently running containers (as per current resource
specification). Basically
avoid calling flex until <code>slider list --containers --version
<old_ver></code> returns no more
containers. Note, flex command is not explicitly blocked by Slider, hence
unexpected
- behavior of the application might occur if called.</li>
+ behavior of the application might occur, if called.</li>
</ul>
</div>
