Author: buildbot
Date: Fri May 22 18:31:09 2015
New Revision: 952342
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/docs/slider_specs/hello_world_slider_app.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Fri May 22 18:31:09 2015
@@ -1 +1 @@
-1680469
+1681180
Modified:
websites/staging/slider/trunk/content/docs/slider_specs/hello_world_slider_app.html
==============================================================================
---
websites/staging/slider/trunk/content/docs/slider_specs/hello_world_slider_app.html
(original)
+++
websites/staging/slider/trunk/content/docs/slider_specs/hello_world_slider_app.html
Fri May 22 18:31:09 2015
@@ -376,8 +376,8 @@ Add one more entry for the component <co
<span class="n">env</span><span class="p">.</span><span
class="n">set_params</span><span class="p">(</span><span
class="n">params</span><span class="p">)</span>
<span class="n">check_process_status</span><span class="p">(</span><span
class="n">params</span><span class="p">.</span><span
class="n">pid_file</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">__name__</span> <span
class="o">==</span> "<span class="n">__main__</span>"<span
class="p">:</span>
- <span class="n">Memcached</span><span class="p">().</span><span
class="n">execute</span><span class="p">()</span>
+<span class="k">if</span> <span class="n">__name__</span> <span
class="o">==</span> "<span class="n">__main__</span>"<span
class="p">:</span>
+ <span class="n">Memcached</span><span class="p">().</span><span
class="n">execute</span><span class="p">()</span>
</pre></div>
@@ -402,6 +402,154 @@ Add one more entry for the component <co
<p>So, without Slider/YARN, one should be able to execute the command above
and see memcached up and running. It might be a better option to try the
command first to ensure that you have the right tarballs and environment to run
memcached.</p>
+<h2 id="add-on-package">Add on package</h2>
+<p>You can deploy the application with add on packages as well. Add on
packages are extension libraries, configurations, or scripts that the master
application can use. For example, HBase has Phoenix providing SQL access to it
and Ranger providing Authorization added to it. With Slider, you can deploy
HBase as the master application, and Phoenix and/or Ranger as the add on
package to it</p>
+<p>To do that, you need to package your add on packages in a similar way to
the master package. For example, in order to deploy Phoenix with HBase, you
need to create an add on package as below:</p>
+<p><code>unzip -l "$@" Phoenix.zip
+Archive: Archive.zip
+ Length Date Time Name
+ -------- ---- ---- ----
+ 2143 04-19-15 12:03 metainfo.xml
+ 0 11-19-14 15:17 package/
+ 0 04-19-15 11:41 package/files/
+ 1908 12-04-14 17:07 package/files/end2endTest.py
+ 840 12-04-14 17:07 package/files/hadoop-metrics2-hbase.properties
+ 2271 12-04-14 17:07 package/files/hadoop-metrics2-phoenix.properties
+ 1136 12-04-14 17:07 package/files/hbase-site.xml
+ 2342 12-04-14 17:07 package/files/log4j.properties
+ 3762 12-04-14 17:07 package/files/performance.py
+ 38015961 12-04-14 17:07 package/files/phoenix-4.2.2-client.jar
+ 4039324 12-04-14 17:07 package/files/phoenix-4.2.2-server.jar
+ 3171 12-04-14 17:07 package/files/phoenix_utils.py
+ 1669 12-04-14 17:07 package/files/psql.py
+ 1820 12-04-14 17:07 package/files/readme.txt
+ 2314 12-04-14 17:07 package/files/sqlline.py
+ 0 04-19-15 21:57 package/scripts/
+ 2287 04-19-15 21:57 package/scripts/addon_hbase_master.py
+ 1659 04-19-15 11:45 package/scripts/addon_hbase_regionserver.py
+ 1079 04-18-15 17:36 package/scripts/params.py
+ 0 04-17-15 18:42 package/templates/
+ -------- -------
+ 42106790 21 files</code></p>
+<p>Please note that different from the master package, the add on package
doesn't have appConfig.json and resources.json, because appConfig.json is not
supported for now, while you can define all the variables you need in the
master package's appConfig.json.
+For add on package, you should/can not add any new components to the master
application, so resources.json is not needed either.</p>
+<p>What is remaining is the metainfo.xml, in which you should/can not add any
new components. However, you can apply some components in the add on package's
metainfo.json belonging to all components in the master package's metainfo.json
by specifying 'ALL' in the component name</p>
+<p>```
+<metainfo>
+ <schemaVersion>2.0</schemaVersion>
+ <applicationPackage>
+ <name>PHOENIX</name>
+ <comment>
+ Apache Phoenix is a relational database layer over HBase delivered as a
client-embedded JDBC driver targeting low latency queries over HBase data.
+ </comment>
+ <version>2.0</version>
+ <type>ADDON-PACKAGE</type>
+ <minHadoopVersion>1.0</minHadoopVersion></p>
+<div class="codehilite"><pre><span class="nt"><components></span>
+ <span class="nt"><component></span>
+ <span class="nt"><name></span>HBASE_REGIONSERVER<span
class="nt"></name></span>
+ <span class="nt"><commandScript></span>
+ <span
class="nt"><script></span>scripts/addon_hbase_regionserver.py<span
class="nt"></script></span>
+ <span class="nt"><scriptType></span>PYTHON<span
class="nt"></scriptType></span>
+ <span class="nt"><timeout></span>600<span
class="nt"></timeout></span>
+ <span class="nt"></commandScript></span>
+ <span class="nt"></component></span>
+ <span class="nt"><component></span>
+ <span class="nt"><name></span>HBASE_MASTER<span
class="nt"></name></span>
+ <span class="nt"><commandScript></span>
+ <span class="nt"><script></span>scripts/addon_hbase_master.py<span
class="nt"></script></span>
+ <span class="nt"><scriptType></span>PYTHON<span
class="nt"></scriptType></span>
+ <span class="nt"><timeout></span>600<span
class="nt"></timeout></span>
+ <span class="nt"></commandScript></span>
+ <span class="nt"></component></span>
+<span class="nt"></components></span>
+
+<span class="nt"><osSpecifics></span>
+ <span class="nt"><osSpecific></span>
+ <span class="nt"><osType></span>any<span
class="nt"></osType></span>
+ <span class="nt"><packages></span>
+ <span class="nt"><package></span>
+ <span class="nt"><type></span>tarball<span
class="nt"></type></span>
+ <span
class="nt"><name></span>files/phoenix-4.2.2-server.jar<span
class="nt"></name></span>
+ <span class="nt"></package></span>
+ <span class="nt"></packages></span>
+ <span class="nt"></osSpecific></span>
+<span class="nt"></osSpecifics></span>
+</pre></div>
+
+
+<p></applicationPackage>
+</metainfo>
+```</p>
+<p>Please note the type of the application package is specified as
'ADDON-PACKAGE'.</p>
+<p>The add on package is expected to implement INSTALL for each component,
without other commands defined for the master application. Note, you can still
use Slider's extensive library that can be used to implement the commands. More
details can be found <a
href="/docs/slider_specs/writing_app_command_scripts">here</a>.</p>
+<p>Below is an example of the python script for HbaseMaster:
+```
+import sys
+import os
+from shutil import copyfile
+from resource_management import *</p>
+<p>class HbaseMaster(Script):
+ def install(self, env):
+ config = Script.get_config()</p>
+<div class="codehilite"><pre><span class="n">src</span> <span
class="p">=</span> <span class="n">config</span><span class="p">[</span><span
class="s">'commandParams'</span><span class="p">][</span><span
class="s">'addonPackageRoot'</span><span class="p">]</span> <span
class="o">+</span> "<span class="o">/</span><span
class="n">package</span><span class="o">/</span><span
class="n">files</span><span class="o">/</span><span
class="n">phoenix</span><span class="o">-</span>4<span class="p">.</span>2<span
class="p">.</span>2<span class="o">-</span><span class="n">client</span><span
class="p">.</span><span class="n">jar</span>"
+<span class="n">dst</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'configurations'</span><span class="p">][</span><span
class="s">'global'</span><span class="p">][</span><span
class="s">'app_root'</span><span class="p">]</span> <span
class="o">+</span> "<span class="o">/</span><span
class="n">lib</span><span class="o">/</span><span class="n">phoenix</span><span
class="o">-</span>4<span class="p">.</span>2<span class="p">.</span>2<span
class="o">-</span><span class="n">client</span><span class="p">.</span><span
class="n">jar</span>"
+<span class="n">copyfile</span><span class="p">(</span><span
class="n">src</span><span class="p">,</span> <span class="n">dst</span><span
class="p">)</span>
+
+<span class="n">filestocopytobin</span> <span class="p">=</span> <span
class="p">[</span>"<span class="n">end2endTest</span><span
class="p">.</span><span class="n">py</span>"<span
class="p">,</span>"<span class="n">hadoop</span><span
class="o">-</span><span class="n">metrics2</span><span class="o">-</span><span
class="n">hbase</span><span class="p">.</span><span
class="k">properties</span>"<span class="p">,</span>"<span
class="n">hadoop</span><span class="o">-</span><span
class="n">metrics2</span><span class="o">-</span><span
class="n">phoenix</span><span class="p">.</span><span
class="k">properties</span>"<span class="p">,</span>"<span
class="n">hbase</span><span class="o">-</span><span class="n">site</span><span
class="p">.</span><span class="n">xml</span>"<span
class="p">,</span>"<span class="n">log4j</span><span
class="p">.</span><span class="k">properties</span>"<span
class="p">,</span>"<span class="n">performance</span><sp
an class="p">.</span><span class="n">py</span>"<span
class="p">,</span>"<span class="n">phoenix_utils</span><span
class="p">.</span><span class="n">py</span>"<span
class="p">,</span>"<span class="n">psql</span><span
class="p">.</span><span class="n">py</span>"<span
class="p">,</span>"<span class="n">readme</span><span
class="p">.</span><span class="n">txt</span>"<span
class="p">,</span>"<span class="n">sqlline</span><span
class="p">.</span><span class="n">py</span>"<span class="p">]</span>
+
+<span class="k">for</span> <span class="n">file</span> <span
class="n">in</span> <span class="n">filestocopytobin</span><span
class="p">:</span>
+ <span class="n">src</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'commandParams'</span><span class="p">][</span><span
class="s">'addonPackageRoot'</span><span class="p">]</span> <span
class="o">+</span> "<span class="o">/</span><span
class="n">package</span><span class="o">/</span><span
class="n">files</span><span class="o">/</span>" <span class="o">+</span>
<span class="n">file</span>
+ <span class="n">dst</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'configurations'</span><span class="p">][</span><span
class="s">'global'</span><span class="p">][</span><span
class="s">'app_root'</span><span class="p">]</span> <span
class="o">+</span> "<span class="o">/</span><span
class="n">bin</span><span class="o">/</span>" <span class="o">+</span>
<span class="n">file</span>
+ <span class="n">copyfile</span><span class="p">(</span><span
class="n">src</span><span class="p">,</span> <span class="n">dst</span><span
class="p">)</span>
+<span class="n">bin_file</span> <span class="p">=</span> <span
class="n">config</span><span class="p">[</span><span
class="s">'configurations'</span><span class="p">][</span><span
class="s">'global'</span><span class="p">][</span><span
class="s">'app_root'</span><span class="p">]</span> <span
class="o">+</span> "<span class="o">/</span><span
class="n">bin</span><span class="o">/</span>" <span class="o">+</span>
"<span class="n">sqlline</span><span class="p">.</span><span
class="n">py</span>"
+<span class="n">os</span><span class="p">.</span><span
class="n">chmod</span><span class="p">(</span><span
class="n">bin_file</span><span class="p">,</span> 0555<span class="p">)</span>
+</pre></div>
+
+
+<p>if <strong>name</strong> == "<strong>main</strong>":
+ HbaseMaster().execute()
+ pass
+```</p>
+<p>Below is an example of the python script for HbaseRegionServer:
+```
+import sys
+from shutil import copyfile
+from resource_management import *</p>
+<p>class HbaseRegionserver(Script):
+ def install(self, env):
+ config = Script.get_config()
+ src = config['commandParams']['addonPackageRoot'] +
"/package/files/phoenix-4.2.2-server.jar"
+ dst = config['configurations']['global']['app_root'] +
"/lib/phoenix-4.2.2-server.jar"
+ copyfile(src, dst)</p>
+<p>def configure(self, env):
+ import params
+ env.set_params(params)</p>
+<p>def start(self, env):
+ import params
+ env.set_params(params)
+ self.configure(env) # for security</p>
+<p>def stop(self, env):
+ import params
+ env.set_params(params)</p>
+<p>def status(self, env):
+ import status_params
+ env.set_params(status_params)</p>
+<p>if <strong>name</strong> == "<strong>main</strong>":
+ HbaseRegionserver().execute()
+ pass
+```</p>
+<p>Similarly, you can provide params.py to provide all configuration needed to
run the install command</p>
+<p>```
+from resource_management import *</p>
+<h1 id="server-configurations">server configurations</h1>
+<p>config = Script.get_config()</p>
+<p>hbase_root = config['configurations']['global']['app_root']
+jar_location = config['commandParams']['addonPackageRoot'] +
"/package/files/phoenix-4.2.2-server.jar"</p>
+<p>print ('jar_location' + jar_location)
+```</p>
+<p>When submitting an application with add on package with Slider, you need to
use '--addon' option to specify the add on package name and path to its zipped
package
+<code>slider create [application_name] --template [path to appConfig.json]
--resources [path to resources.json] --addon PHOENIX [path to
Phoenix.zip]</code></p>
</div>
<div id="footer">
