Author: smohanty
Date: Wed Aug 27 22:07:54 2014
New Revision: 1620986
URL: http://svn.apache.org/r1620986
Log:
Changes to the document to reflect latest Slider package structures
Modified:
incubator/slider/site/trunk/content/docs/getting_started.md
incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md
incubator/slider/site/trunk/content/docs/slider_specs/index.md
Modified: incubator/slider/site/trunk/content/docs/getting_started.md
URL:
http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/getting_started.md?rev=1620986&r1=1620985&r2=1620986&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/getting_started.md (original)
+++ incubator/slider/site/trunk/content/docs/getting_started.md Wed Aug 27
22:07:54 2014
@@ -112,18 +112,15 @@ Browse to the Slider directory: `cd slid
Modify Slider configuration file
`${slider-install-dir}/slider-0.30.0/conf/slider-client.xml` to add the
following properties:
<property>
- <name>yarn.application.classpath</name>
-
<value>/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*</value>
+ <name>yarn.application.classpath</name>
+
<value>/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*</value>
</property>
<property>
- <name>slider.zookeeper.quorum</name>
- <value>yourZooKeeperHost:port</value>
+ <name>slider.zookeeper.quorum</name>
+ <value>yourZooKeeperHost:port</value>
</property>
-
-In addition, specify the scheduler and HDFS addresses as follows:
-
<property>
<name>yarn.resourcemanager.address</name>
<value>yourResourceManagerHost:8050</value>
@@ -154,10 +151,6 @@ Perform the following steps to create th
su hdfs
- hdfs dfs -mkdir /slider
-
- hdfs dfs -chown yarn:hdfs /slider
-
hdfs dfs -mkdir /user/yarn
hdfs dfs -chown yarn:hdfs /user/yarn
@@ -166,9 +159,9 @@ Perform the following steps to create th
su yarn
- hdfs dfs -mkdir /slider/agent
+ hdfs dfs -mkdir /user/yarn/agent
- hdfs dfs -copyFromLocal
${slider-install-dir}/slider-0.30.0/agent/slider-agent.tar.gz /slider/agent
+ hdfs dfs -copyFromLocal
${slider-install-dir}/slider-0.30.0/agent/slider-agent.tar.gz /user/yarn/agent
## <a name="downsample"></a>Create Application Packages
@@ -199,36 +192,35 @@ Create one or more Slider application pa
### <a name="load"></a>Load Sample Application Package
- hdfs dfs -copyFromLocal *sample-application-package* /slider
+ hdfs dfs -copyFromLocal *sample-application-package* /user/yarn
### <a name="create"></a>Create Application Specifications
Configuring a Slider application consists of two parts: the [Resource
Specification](#resspec),
- and the *[Application Configuration](#appconfig). Below are guidelines for
creating these files.
+ and the *[Application Configuration](#appconfig).
-*Note: There are sample Resource Specifications (**resources.json**) and
Application Configuration
-(**appConfig.json**) files in the root directory of the
-Sample Applications packages (e.g. /**resources.json** and
/**appConfig.json**). The Slider git repo also contains sample appConfig.json
and resources.json in the application packages directory (e.g.
app-packages/hbase/appConfig.json)*
+**Note**: There are sample Resource Specifications (**resources.json**) and
Application Configuration
+(**appConfig.json**) files in application packages (e.g. /**resources.json**
and /**appConfig.json**).
#### <a name="resspec"></a>Resource Specification
-Slider needs to know what components (and how many components) are in an
application package to deploy. For example, in HBase, the components are
**_master_** and **_worker_** -- the latter hosting **HBase RegionServers**,
and the former hosting the **HBase Master**.
+Slider needs to know what components (and how many components) are in an
application package to deploy.
As Slider creates each instance of a component in its own YARN container, it
also needs to know what to ask YARN for in terms of **memory** and **CPU** for
those containers.
All this information goes into the **Resources Specification** file ("Resource
Spec") named `resources.json`. The Resource Spec tells Slider how many
instances of each component in the application (such as an HBase RegionServer)
to deploy and the parameters for YARN.
+An application package should contain the default resources.json and you can
start from there. Or you can create one based on [Resource
Specification](slider_specs/resource_specification.html).
+
Store the Resource Spec file on your local disk (e.g. `/tmp/resources.json`).
-[Appendix B](#appendixb) provides some sample resources.json files.
#### <a name="appconfig"></a>Application Configuration
Alongside the Resource Spec there is the **Application Configuration** file
("App Config") which includes parameters that are specific to the application,
rather than YARN. The App Config is a file that contains all application
configuration. This configuration is applied to the default configuration
provided by the application definition and then handed off to the associated
component agent.
-For example, the heap sizes of the JVMs, The App Config defines the
configuration details **specific to the application and component** instances.
For HBase, this includes any values for the *to-be-generated *hbase-site.xml
file, as well as options for individual components, such as their heap size.
+For example, the heap sizes of the JVMs, The App Config defines the
configuration details **specific to the application and component** instances.
An application package should contain the default resources.json and you can
start from there.
-[Appendix C](#appendixc) provides a sample appConfig.json file.
### <a name="start"></a>Start the Application
@@ -240,7 +232,7 @@ Change directory to the "bin" directory
Execute the following command:
- ./slider create cl1 --image
hdfs://yourNameNodeHost:8020/slider/agent/slider-agent.tar.gz --template
appConfig.json --resources resources.json
+ ./slider create cl1 --image
hdfs://yourNameNodeHost:8020/user/yarn/agent/slider-agent.tar.gz --template
appConfig.json --resources resources.json
### <a name="verify"></a>Verify the Application
@@ -324,152 +316,15 @@ The sample agent.ini file can be used as
`log_dir` = the relative location of the agent and command log file
- [server]
- hostname=localhost
- port=8440
- secured_port=8441
- check_path=/ws/v1/slider/agents/
- register_path=/ws/v1/slider/agents/{name}/register
- heartbeat_path=/ws/v1/slider/agents/{name}/heartbeat
-
- [agent]
- app_pkg_dir=app/definition
- app_install_dir=app/install
- app_run_dir=app/run
- app_dbg_cmd=
- debug_mode_enabled=true
-
- app_task_dir=app/command-log
- app_log_dir=app/log
- app_tmp_dir=app/tmp
-
- log_dir=infra/log
- run_dir=infra/run
- version_file=infra/version
-
- log_level=INFO
-
- [python]
-
- [command]
- max_retries=2
- sleep_between_retries=1
-
- [security]
-
- [heartbeat]
- state_interval=6
- log_lines_count=300
-
-
Once created, deploy the agent.ini file to HDFS (it can be deployed in any
location accessible to application instance):
su yarn
- hdfs dfs -copyFromLocal agent.ini /slider/agent/conf
+ hdfs dfs -copyFromLocal agent.ini /user/yarn/agent/conf
Modify the --template json file (appConfig.json) and add the location of the
agent.ini file.
- "agent.conf": "/slider/agent/conf/agent.ini"
-
-# <a name="appendixb"></a>Appendix B: Resource template (resources.json)
-
-This sample is suitable for HBase application.
-
- {
- "schema": "http://example.org/specification/v2.0.0";,
- "metadata": {
- },
- "global": {
- },
- "components": {
- "HBASE_MASTER": {
- "yarn.role.priority": "1",
- "yarn.component.instances": "1",
- "yarn.memory": "256"
- },
- "slider-appmaster": {
- },
- "HBASE_REGIONSERVER": {
- "yarn.role.priority": "2",
- "yarn.component.instances": "1",
- "yarn.memory": "256"
- }
- }
- }
-
-# <a name="appendixc"></a>Appendix C: Config template (appCofig.json)
-
-This sample is suitable for HBase application. See [Application Instance
Configuration](/docs/slider_specs/application_instance_configuration.html) for
detailed explanation of how to create an appConfig.json file.
-
- {
- "schema": "http://example.org/specification/v2.0.0";,
- "metadata": {
- },
- "global": {
- "application.def": "${app.package.name}.zip",
- "create.default.zookeeper.node": "true",
- "config_types": "core-site,hdfs-site,hbase-site",
- "java_home": "/usr/jdk64/jdk1.7.0_45",
- "package_list": "files/hbase-${hbase.version}-bin.tar.gz",
- "site.global.app_user": "yarn",
- "site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
- "site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
- "site.global.app_root":
"${AGENT_WORK_ROOT}/app/install/hbase-${hbase.version}",
- "site.global.app_install_dir": "${AGENT_WORK_ROOT}/app/install",
- "site.global.hbase_master_heapsize": "1024m",
- "site.global.hbase_regionserver_heapsize": "1024m",
- "site.global.hbase_instance_name": "instancename",
- "site.global.hbase_root_password": "secret",
- "site.global.user_group": "hadoop",
- "site.global.security_enabled": "false",
- "site.global.monitor_protocol": "http",
- "site.global.ganglia_server_host": "${NN_HOST}",
- "site.global.ganglia_server_port": "8667",
- "site.global.ganglia_server_id": "Application1",
- "site.hbase-site.hbase.hstore.flush.retries.number": "120",
- "site.hbase-site.hbase.client.keyvalue.maxsize": "10485760",
- "site.hbase-site.hbase.hstore.compactionThreshold": "3",
- "site.hbase-site.hbase.rootdir": "${DEFAULT_DATA_DIR}/data",
- "site.hbase-site.hbase.stagingdir": "${DEFAULT_DATA_DIR}/staging",
- "site.hbase-site.hbase.regionserver.handler.count": "60",
- "site.hbase-site.hbase.regionserver.global.memstore.lowerLimit":
"0.38",
- "site.hbase-site.hbase.hregion.memstore.block.multiplier": "2",
- "site.hbase-site.hbase.hregion.memstore.flush.size": "134217728",
- "site.hbase-site.hbase.superuser": "yarn",
- "site.hbase-site.hbase.zookeeper.property.clientPort": "2181",
- "site.hbase-site.hbase.regionserver.global.memstore.upperLimit": "0.4",
- "site.hbase-site.zookeeper.session.timeout": "30000",
- "site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
- "site.hbase-site.hbase.local.dir": "${hbase.tmp.dir}/local",
- "site.hbase-site.hbase.hregion.max.filesize": "10737418240",
- "site.hbase-site.hfile.block.cache.size": "0.40",
- "site.hbase-site.hbase.security.authentication": "simple",
- "site.hbase-site.hbase.defaults.for.version.skip": "true",
- "site.hbase-site.hbase.zookeeper.quorum": "${ZK_HOST}",
- "site.hbase-site.zookeeper.znode.parent": "${DEF_ZK_PATH}",
- "site.hbase-site.hbase.hstore.blockingStoreFiles": "10",
- "site.hbase-site.hbase.hregion.majorcompaction": "86400000",
- "site.hbase-site.hbase.security.authorization": "false",
- "site.hbase-site.hbase.cluster.distributed": "true",
- "site.hbase-site.hbase.hregion.memstore.mslab.enabled": "true",
- "site.hbase-site.hbase.client.scanner.caching": "100",
- "site.hbase-site.hbase.zookeeper.useMulti": "true",
- "site.hbase-site.hbase.regionserver.info.port": "0",
- "site.hbase-site.hbase.master.info.port":
"${HBASE_MASTER.ALLOCATED_PORT}",
- "site.hbase-site.hbase.regionserver.port": "0"
- },
- "components": {
- "HBASE_MASTER": {
- },
- "slider-appmaster": {
- "jvm.heapsize": "256M"
- },
- "HBASE_REGIONSERVER": {
- }
- }
- }
-
+ "agent.conf": "/user/yarn/agent/conf/agent.ini"
Modified:
incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md
URL:
http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md?rev=1620986&r1=1620985&r2=1620986&view=diff
==============================================================================
---
incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md
(original)
+++
incubator/slider/site/trunk/content/docs/slider_specs/hello_world_slider_app.md
Wed Aug 27 22:07:54 2014
@@ -17,7 +17,7 @@
# Goal
-Define the Slider application package for YARN hosted
[jmemcached](https://code.google.com/p/jmemcache-daemon/). Jmemcached is a
functionally equivalent java implementation of memcached, a distributed memory
object caching system.
+Define the Slider application package for YARN hosted
[jmemcached](https://code.google.com/p/jmemcache-daemon/). Jmemcached is a
functionally equivalent java implementation of memcached, a distributed memory
object caching system. The memcached daemons export the host/port they are
listening on.
## Basic version
@@ -28,7 +28,7 @@ The structure of an app package is discu
In this example, the application package created looks as follows:
- unzip -l "$@"
~/enlistments/ambari-vagrant/centos6.4/memcached/jmemcached-1.0.0.zip
+ unzip -l "$@" jmemcached-1.0.0.zip
Archive: /jmemcached-1.0.0.zip
Length Date Time Name
-------- ---- ---- ----
@@ -54,11 +54,18 @@ The minimal metainfo contains some infor
<name>MEMCACHED</name>
<comment>Memcache is a network accessible key/value storage system,
often used as a distributed cache.</comment>
<version>1.0.0</version>
-
+ <exportedConfigs>None</exportedConfigs>
+
<components>
<component>
<name>MEMCACHED</name>
<category>MASTER</category>
+ <componentExports>
+ <componentExport>
+ <name>host_port</name>
+ <value>${THIS_HOST}:${site.global.listen_port}</value>
+ </componentExport>
+ </componentExports>
<commandScript>
<script>scripts/memcached.py</script>
<scriptType>PYTHON</scriptType>
@@ -129,6 +136,7 @@ Add other parameters needed by the appli
* site.global.xmx_val - value of Xmx
* site.global.xms_val - value of Xms
* site.global.memory_val - value of memory
+* site.global.listen_port - let Slider know that ports need to be allocated
You can add additional parameters as needed.
@@ -137,22 +145,22 @@ You can add additional parameters as nee
"metadata": {
},
"global": {
- "application.def": "/slider/jmemcached-1.0.0.zip",
+ "application.def": "jmemcached-1.0.0.zip",
"java_home": "/usr/jdk64/jdk1.7.0_45",
- "package_list": "files/jmemcached-1.0.0.tar",
- "config_types": "",
+
"site.global.app_user": "yarn",
"site.global.app_root":
"${AGENT_WORK_ROOT}/app/install/jmemcached-1.0.0",
+ "site.global.pid_file": "${AGENT_WORK_ROOT}/app/run/component.pid",
+
"site.global.additional_cp": "/usr/lib/hadoop/lib/*",
"site.global.xmx_val": "256m",
"site.global.xms_val": "128m",
- "site.global.memory_val": "200M"
+ "site.global.memory_val": "200M",
+ "site.global.listen_port":
"${MEMCACHED.ALLOCATED_PORT}{DO_NOT_PROPAGATE}"
},
"components": {
"slider-appmaster": {
"jvm.heapsize": "256M"
- },
- "MEMCACHED": {
}
}
}
@@ -173,10 +181,13 @@ The parameters file we will use is:
app_root = config['configurations']['global']['app_root']
java64_home = config['hostLevelParams']['java_home']
app_user = config['configurations']['global']['app_user']
+ pid_file = config['configurations']['global']['pid_file']
+
additional_cp = config['configurations']['global']['additional_cp']
xmx_val = config['configurations']['global']['xmx_val']
xms_val = config['configurations']['global']['xms_val']
memory_val = config['configurations']['global']['memory_val']
+ port = config['configurations']['global']['listen_port']
Note that the parameter.py file only reads the parameters needed by the
command implementations.
@@ -197,12 +208,13 @@ Note that the parameter.py file only rea
import params
env.set_params(params)
self.configure(env)
- process_cmd = format("{java64_home}/bin/java -Xmx{xmx_val}
-Xms{xms_val} -classpath {app_root}/*:{additional_cp}
com.thimbleware.jmemcached.Main --memory={memory_val}")
+ process_cmd = format("{java64_home}/bin/java -Xmx{xmx_val}
-Xms{xms_val} -classpath {app_root}/*:{additional_cp}
com.thimbleware.jmemcached.Main --memory={memory_val} --port={port}")
Execute(process_cmd,
user=params.app_user,
logoutput=False,
- wait_for_finish=False
+ wait_for_finish=False,
+ pid_file=params.pid_file
)
def stop(self, env):
@@ -212,9 +224,11 @@ Note that the parameter.py file only rea
def status(self, env):
import params
env.set_params(params)
+ check_process_status(params.pid_file)
Thats pretty much it. The script basically does the following:
* Expand the given tarball
* Reads the provided configuration and creates the command string
-* Executes the command to start jmemcached
\ No newline at end of file
+* Executes the command to start jmemcached
+* Start writes the PID into a file that is used to check status of the daemon
\ No newline at end of file
Modified: incubator/slider/site/trunk/content/docs/slider_specs/index.md
URL:
http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/index.md?rev=1620986&r1=1620985&r2=1620986&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/index.md (original)
+++ incubator/slider/site/trunk/content/docs/slider_specs/index.md Wed Aug 27
22:07:54 2014
@@ -36,8 +36,9 @@
- Use of slider minimizes custom code for deployment + lifecycle management
- Requires apps to follow Slider guidelines and packaging ("Sliderize")
-Specifications
----
+##Specifications
+
+Refer to [Creating a Slider package for
Memcached](hello_world_slider_app.html) for a quick over view of how to write a
Slider app.
The entry points to leverage Slider are:
@@ -48,7 +49,8 @@ The entry points to leverage Slider are:
- [Specification of Resources](resource_specification.html) How to write a
resource spec for an app?
- [Specifications
InstanceConfiguration](application_instance_configuration.html) How to write a
template config for an app?
- [Guidelines for Clients and Client Applications](canonical_scenarios.html)
-- [Specifications for Configuration](application_configuration.html) Default
application configuration?
+- [Specifications for Configuration](application_configuration.html) Default
application configuration
+- [Specifying Exports](specifying_exports.html) How to specify exports for an
application?
- [Documentation for "General Developer Guidelines"](/developing/index.html)
* [Configuring the Slider Chaos Monkey](slider_specs/chaosmonkey.md)
