Repository: incubator-myriad Updated Branches: refs/heads/master 420cb78a6 -> a2c0202a4
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/a2c0202a/docs/myriad-scheduler-architecture.md ---------------------------------------------------------------------- diff --git a/docs/myriad-scheduler-architecture.md b/docs/myriad-scheduler-architecture.md index c40cd03..3ee5733 100644 --- a/docs/myriad-scheduler-architecture.md +++ b/docs/myriad-scheduler-architecture.md @@ -1,69 +1,31 @@ -# Myriad Scheduler +# Fine-grained Scaling Architecture Myriad scheduler is comprised of components that interact with YARN and Mesos -services. The following diagram outlines the main interactions. - - +services. ## Mesos Master Interactions -* Myriad Scheduler - - This is the entry point to register a framework scheduler with Mesos. Upon - registration, we start receiving Mesos offers. - -* Offer Feed - - Mesos offers are accepted and stashed here. - -* Offer Lifecycle Manager - - Interface to retrieve stashed offers, return unused offers back to Mesos. - -* Consumed Offer - Represents offers that have been consumed for a slave prior to scheduling run. - When offers are consumed, they increase the capacity of YARN Node Manager by - that amount. Note that no Mesos tasks are launched when offers are consumed. - They are launched only after the scheduling run. +Class | Description | +------|------------ | +MyriadScheduler| This is the entry point to register a framework scheduler with Mesos. Upon registration, Mesos offers start to be received. | +OfferFeed | Mesos offers are accepted and stashed here. | +OfferLifecycleManager | Interface to retrieve stashed offers, return unused offers back to Mesos. | +ConsumedOffer | Represents offers that have been consumed for a slave prior to scheduling run. When offers are consumed, they increase the capacity of YARN Node Manager by that amount. Note that no Mesos tasks are launched when offers are consumed. They are launched only after the scheduling run. | -## Mesos Slave Interactions -* Pseudo Task Manager - Interface to manage pseudo Mesos tasks that are launched to satisfy the container - allocations done by YARN scheduler. +## Resource Manager Scheduler Interactions ## -* Pseudo Task Resource Manager - Interface to manage the resources consumed by psuedo Mesos tasks and reuse them - as needed. Pseudo Mesos tasks are launched only when there isn't enough - resources from existing tasks available to satisfy YARN container allocations. - Similarly, when YARN containers complete, their resources are not immediately - released back to Mesos. They are reused for the next scheduling run. When - resources don't get used up after a scheduling run, they are released back to - Mesos. +Class | Description | +------|------------ | +Myriad Fair Scheduler, Myriad Capacity Scheduler | Extension point for YARN schedulers to enable setting up hooks into scheduler events. | +YARN Node Capacity Manager | Controls the YARN Scheduler's view of the cluster node capacity. YARN schedules containers on nodes based on available capacity. Myriad enables YARN cluster elasticity by dynamically growing and shrinking the node capacities. | -## Resource Manager Scheduler Interactions -* Myriad Fair Scheduler, Myriad Capacity Scheduler - - Extension point for YARN schedulers to enable setting up hooks into scheduler events. - -* YARN Node Capacity Manager - - Controls the YARN Scheduler's view of the cluster node capacity. YARN schedules - containers on nodes based on available capacity. Myriad enables YARN cluster - elasticity by dynamically growing and shrinking the node capacities. ## Node Manager Interactions -* NM HB Handler - - Handles node manager heartbeat and responds to status updates of YARN - containers. For running containers, it sends status update to Mesos Executor and - for completed containers, it informs Pseudo Task Manager about available - resources. - -* Node Store - - Tracks the set of nodes that are part of YARN cluster. -* Node - Tracks metadata about a node. +Class | Description | +------|------------ | +NM HB Handler | Handles node manager heartbeat and responds to status updates of YARN containers. For running containers, it sends status update to Mesos Executor and for completed containers, it informs Pseudo Task Manager about available resources. | +Node Store | Tracks the set of nodes that are part of YARN cluster. | +Node | Tracks metadata about a node. | http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/a2c0202a/docs/node-manager-profiles.md ---------------------------------------------------------------------- diff --git a/docs/node-manager-profiles.md b/docs/node-manager-profiles.md new file mode 100644 index 0000000..6930c5d --- /dev/null +++ b/docs/node-manager-profiles.md @@ -0,0 +1,39 @@ +# Node Manager Profiles # + +The Node Manager profile is an abstraction for the amount of resources a Node Manager advertises to the Resource Manager. These profiles are set in the Myriad configuration file, myriad-config-default.yml. Each profile is identified by zero, small, medium, and large with default CPU and memory parameters. The cluster can be configured to start with an initial number of Node Managers. By default, Myriad requires at least 1 NodeManager instance with a medium profile. See Scaling Resources Dynamically and the Myriad Cluster API for information about scaling instance resources. + +**Note:**During initial startup, Myriad launches one NodeManager instance with a profile of medium through the parameter, nmInstances, which is specified in the myriad-config-default.yml file. + +``` +"nmInstances": { + "medium": 1 + }, +``` + +The following default profiles are configurable. To change a profile, modify the Myriad configuration file, `myriad-config-default.yml`. + +**Note:** If you modify the Myriad configuration file after the initial build, you must build and deploy again for the changes to take affect. + +Default profiles: + +``` +"profiles": { + "zero": { + "cpu": "0", + "mem": "0" + }, + "small": { + "cpu": "2", + "mem": "2048" + }, + "medium": { + "cpu": "4", + "mem": "4096" + }, + "large": { + "cpu": "10", + "mem": "12288" + } +}, +``` + http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/a2c0202a/docs/sample-yarn-site.md ---------------------------------------------------------------------- diff --git a/docs/sample-yarn-site.md b/docs/sample-yarn-site.md new file mode 100644 index 0000000..f3c6fdf --- /dev/null +++ b/docs/sample-yarn-site.md @@ -0,0 +1,117 @@ +# Sample: yarn-site.xml + +The following is a sample yarn-site.xml file. + + + +<pre> +<?xml version="1.0" encoding="UTF-8"?> +<configuration> + +<!-- Site-specific YARN configuration properties --> + <property> + <name>yarn.nodemanager.aux-services</name> + <value>mapreduce_shuffle,myriad_executor</value> + <!-- If using MapR distro, please use the following value: + <value>mapreduce_shuffle,mapr_direct_shuffle,myriad_executor</value> --> + </property> + <property> + <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> + <value>org.apache.hadoop.mapred.ShuffleHandler</value> + </property> + <property> + <name>yarn.nodemanager.aux-services.myriad_executor.class</name> + <value>org.apache.myriad.executor.MyriadExecutorAuxService</value> + </property> + <property> + <name>yarn.nm.liveness-monitor.expiry-interval-ms</name> + <value>2000</value> + </property> + <property> + <name>yarn.am.liveness-monitor.expiry-interval-ms</name> + <value>10000</value> + </property> + <property> + <name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name> + <value>1000</value> + </property> +<!-- (more) Site-specific YARN configuration properties --> + <property> + <name>yarn.nodemanager.resource.cpu-vcores</name> + <value>${nodemanager.resource.cpu-vcores}</value> + </property> + <property> + <name>yarn.nodemanager.resource.memory-mb</name> + <value>${nodemanager.resource.memory-mb}</value> + </property> +<!-- Dynamic Port Assignment enablement by Mesos --> + <property> + <name>yarn.nodemanager.address</name> + <value>${myriad.yarn.nodemanager.address}</value> + </property> + <property> + <name>yarn.nodemanager.webapp.address</name> + <value>${myriad.yarn.nodemanager.webapp.address}</value> + </property> + <property> + <name>yarn.nodemanager.webapp.https.address</name> + <value>${myriad.yarn.nodemanager.webapp.address}</value> + </property> + <property> + <name>yarn.nodemanager.localizer.address</name> + <value>${myriad.yarn.nodemanager.localizer.address}</value> + </property> + +<!-- Myriad Scheduler configuration --> + <property> + <name>yarn.resourcemanager.scheduler.class</name> + <value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value> + </property> +<!-- Needed for Fine Grain Scaling --> + <property> + <name>yarn.scheduler.minimum-allocation-vcores</name> + <value>0</value> + </property> + <property> + <name>yarn.scheduler.minimum-allocation-vcores</name> + <value>0</value> + </property> +<!-- Cgroups specific configuration --> +<!-- + <property> + <description>Who will execute(launch) the containers.</description> + <name>yarn.nodemanager.container-executor.class</name> + <value>${yarn.nodemanager.container-executor.class}</value> + </property> + <property> + <description>The class which should help the LCE handle resources.</description> + <name>yarn.nodemanager.linux-container-executor.resources-handler.class</name> + <value>${yarn.nodemanager.linux-container-executor.resources-handler.class}</value> + </property> + <property> + <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name> + <value>${yarn.nodemanager.linux-container-executor.cgroups.hierarchy}</value> + </property> + <property> + <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name> + <value>${yarn.nodemanager.linux-container-executor.cgroups.mount}</value> + </property> + <property> + <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name> + <value>${yarn.nodemanager.linux-container-executor.cgroups.mount-path}</value> + </property> + <property> + <name>yarn.nodemanager.linux-container-executor.group</name> + <value>${yarn.nodemanager.linux-container-executor.group}</value> + </property> + <property> + <name>yarn.nodemanager.linux-container-executor.path</name> + <value>${yarn.home}/bin/container-executor</value> + </property> +--> +</configuration> +</pre> + + + + http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/a2c0202a/docs/vagrant.md ---------------------------------------------------------------------- diff --git a/docs/vagrant.md b/docs/vagrant.md index 17be2d2..c42447f 100644 --- a/docs/vagrant.md +++ b/docs/vagrant.md @@ -1,11 +1,14 @@ -# Vagrant setup +# Installing using Vagrant -You can use following guide to setup a cluster in a virtual machine. +This section provides information for setting up a cluster in a virtual machine. Setting up a cluster in a virtual machine involves the following: -### Prerequisities + +## Prerequisities * Virtualbox * Vagrant +## Starting the Cluster ## + To start the cluster run following: ``` @@ -14,24 +17,24 @@ vagrant up At this point the VM will have a single node mesos cluster running. -To ssh in the cluster run following: +To ssh in the cluster, run following: ``` vagrant ssh ``` -The password for vagrant user is 'vagrant'. +The password for vagrant user is **vagrant'** -To setup YARN/Hadoop inside VM, run following: +## Setting Up YARN/Hadoop ## +To setup YARN/Hadoop inside VM, run following YARN setup shell files: + +1 Run the first YARN setup shell command from the vagrant directory to create a user hduser in group hadoop. Be sure to remember the password that you provide for this user. ``` cd /vagrant ./setup-yarn-1.sh ``` - -This will create a user hduser in group hadoop. Remember the password that you provide for this user. - -Now, do following: +2. Run the second YARN setup shell command as sudo. ``` sudo su - hduser @@ -39,7 +42,7 @@ cd /vagrant ./setup-yarn-2.sh ``` -If everything goes fine you'll see following processes running (process ids will be different): +The following processes should be running: ``` 9844 Jps @@ -48,54 +51,78 @@ If everything goes fine you'll see following processes running (process ids will 6874 DataNode ``` -To build myriad scheduler inside VM, you can do following: +**NOTE:** The process IDS are different. + +## Installing Myriad ## + +Installing Myriad involves downloading the Myriad binaries from GitHub. The Myriad location is currently at [https://github.com/mesos/myriad](https://github.com/mesos/myriad). + +To install Myriad: + +1. Create a new myriad directory. +2. Clone the Myriad GitHub. +3. Pull the contents to the new directory. + + +## Configuring Myriad ## + +To configure Myriad: + +### Step 1: Build Myriad ### + +To build Myriad Scheduler inside VM, run the gradlew build: ``` cd /vagrant ./gradlew build ``` -**Dealing with a build failure** -If you get a build failure which is not the build itself, but a failure to write to disk. This can happen when you built outside the vagrant instance first. Exit the user `hduser` by typing `exit` and build again as the `vagrant` user. +**NOTE:** If build failure failure occurs, the issue is not with the build itself, but a failure to write to disk. This can happen when you built outside the vagrant instance first. Exit the user `hduser` by typing `exit` and build again as the `vagrant` user. + +### Step 2: Deploy the Myriad Files ### + +The Myriad Schedule and Executer jar files and all the runtime dependences as well as the Myriad configuration file must be copied to $YARN_HOME. + +* The Myriad Scheduler jar and all the runtime dependencies are located at: + +``` +/vagrant/myriad-scheduler/build/libs/* +``` +* The Myriad configuration file is located at: + +``` +/vagrant/myriad-scheduler/src/main/resources/myriad-config-default.yml +``` -At this point, myriad's scheduler jar and all the runtime dependencies will be available here: `/vagrant/myriad-scheduler/build/libs/*`. Please copy these jars to `$YARN_HOME/share/hadoop/yarn/lib/`. The default `$YARN_HOME` is `/usr/local/hadoop/`. +* The Myriad Executor jar file are located at: ``` -cp /vagrant/myriad-scheduler/build/libs/* /usr/local/hadoop/share/hadoop/yarn/lib/ +/vagrant/myriad-scheduler/build/libs/myriad-executor-0.0.1.jar ``` -The self-contained myriad executor jar will be available here: `/vagrant/myriad-executor/build/libs/myriad-executor-runnable-x.y.z.jar`. Please copy this jar to `/usr/local/libexec/mesos/`. +For example, the files are copied to the following locations: ``` -sudo mkdir -p /usr/local/libexec/mesos/ -sudo cp /vagrant/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /usr/local/libexec/mesos/ -sudo chown hduser:hadoop -R /usr/local/libexec/mesos/ +cp /vagrant/myriad-scheduler/build/libs/* $YARN_HOME/share/hadoop/yarn/lib/ +cp /vagrant/myriad-executor/build/libs/myriad-executor-0.0.1.jar $YARN_HOME/share/hadoop/yarn/lib/ +cp /vagrant/myriad-scheduler/src/main/resources/myriad-config-default.yml $YARN_HOME/etc/hadoop/ ``` -To configure YARN to use Myriad, please update ```$YARN_HOME/etc/hadoop/yarn-site.xml``` with following: -```xml -<property> - <name>yarn.nodemanager.resource.cpu-vcores</name> - <value>${nodemanager.resource.cpu-vcores}</value> -</property> -<property> - <name>yarn.nodemanager.resource.memory-mb</name> - <value>${nodemanager.resource.memory-mb}</value> -</property> +## Step 3: Configure the Myriad Defaults ## -<!-- Configure Myriad Scheduler here --> -<property> - <name>yarn.resourcemanager.scheduler.class</name> - <value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value> - <description>One can configure other schedulers as well from following list: org.apache.myriad.scheduler.yarn.MyriadCapacityScheduler, org.apache.myriad.scheduler.yarn.MyriadFifoScheduler</description> -</property> -``` + As a minimum, the following Myriad configuration parameters must be set: + +* mesosMaster +* zkServers +* YARN_HOME -To configure Myriad itself, please add following file to ```$YARN_HOME/etc/hadoop/myriad-default-config.yml```: +**NOTE:** Enabling Cgroups involves modifying the **yarn-site.xml** and **myriad-config-default.yml** files. If you plan on using Cgroups, you could set that property at this time. See [Configuring Cgroups](cgroups.md) for more information. + +To configure Myriad itself, update **$YARN_HOME/etc/hadoop/myriad-default-config.yml** with the following content: ```yml -mesosMaster: 10.141.141.20:5050 +mesosMaster: <mesos Master IP address>:5050 checkpoint: false frameworkFailoverTimeout: 43200000 frameworkName: MyriadAlpha @@ -104,14 +131,14 @@ zkServers: localhost:2181 zkTimeout: 20000 profiles: small: - cpu: 1 - mem: 1100 - medium: cpu: 2 mem: 2048 - large: + medium: cpu: 4 mem: 4096 + large: + cpu: 10 + mem: 12288 rebalancer: true nodemanager: jvmMaxMemoryMB: 1024 @@ -123,28 +150,72 @@ executor: path: file://localhost/usr/local/libexec/mesos/myriad-executor-runnable-0.0.1.jar ``` -To launch Myriad, you can run following: + +### Step 4: Configure YARN to Use Myriad ### + +To configure YARN to use Myriad, update **$YARN_HOME/etc/hadoop/yarn-site.xml** with following content: + +```xml +<property> +<name>yarn.nodemanager.resource.cpu-vcores</name> +<value>${nodemanager.resource.cpu-vcores}</value> +</property> +<property> +<name>yarn.nodemanager.resource.memory-mb</name> +<value>${nodemanager.resource.memory-mb}</value> +</property> +<!-- The following properties enable dynamic port assignment by Mesos --> +<property> + <name>yarn.nodemanager.address</name> + <value>${myriad.yarn.nodemanager.address}</value> +</property> +<property> + <name>yarn.nodemanager.webapp.address</name> + <value>${myriad.yarn.nodemanager.webapp.address}</value> +</property> +<property> + <name>yarn.nodemanager.webapp.https.address</name> + <value>${myriad.yarn.nodemanager.webapp.address}</value> +</property> +<property> + <name>yarn.nodemanager.localizer.address</name> + <value>${myriad.yarn.nodemanager.localizer.address}</value> +</property> +<!-- The following properties configure Myriad Scheduler --> +<property> +<name>yarn.resourcemanager.scheduler.class</name> +<value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value> +<description>One can configure other schedulers as well from following list: org.apache.myriad.scheduler.yarn.MyriadCapacityScheduler, org.apache.myriad.scheduler.yarn.MyriadFifoScheduler</description> +</property> +``` + + +## Getting Started ## + +### Launching +To launch Myriad, run the following: ``` sudo su hduser yarn-daemon.sh start resourcemanager ``` +### Verifying Activity To check that things are running, from a browser on the host check out the following urls: -* [Myriad UI](http://10.141.141.20:8192/) -* [Mesos UI - Frameworks](http://10.141.141.20:5050/#/frameworks) +* Myriad UI - Use the 8192 port. For example: <pre> http://<IP address>:8192/ (http://10.141.141.20:8192/)</pre> + * Without the Mesos-DNS service, this IP is slave node's IP address where the ResourceManager is running. + * With the Mesos-DNS service, Mesos-DNS discovers the node where the ResourceManager is running by using Mesos DNS FQDN (<app>.<framework>.mesos). +* Mesos UI - Use the 5050 port and Frameworks directory. For example: <pre>http://<IP address>:5050/#/frameworks (http://10.141.141.20:5050/#/frameworks).</pre> + +### Shutting Down ### -To shut down, from the vagrant ssh console: +To shut down, from the vagrant ssh console, run the following: ``` yarn-daemon.sh stop resourcemanager - ./shutdown.sh - exit - exit - vagrant halt ```
