A full DC/OS cluster for Apache Myriad development and testing with resources
such as Mesos-DNS, Minuteman load balancer (VIPs), Marathon meta-scheduler and
so on.
Advertising
JIRA:
[MYRIAD-262] https://issues.apache.org/jira/browse/MYRIAD-262
Pull Request:
Closes #108
Author: Javi Roman <jroman.espi...@gmail.com>
Date: Sat Apr 7 10:59:50 2018 +0200
Project: http://git-wip-us.apache.org/repos/asf/incubator-myriad/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-myriad/commit/1c3857a8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-myriad/tree/1c3857a8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-myriad/diff/1c3857a8
Branch: refs/heads/master
Commit: 1c3857a8dde10cf162a7abef08170679b7ee1512
Parents: 2ce260e
Author: Javi Roman <jroman.espi...@gmail.com>
Authored: Sat Apr 7 10:59:50 2018 +0200
Committer: Javi Roman <javiro...@apache.org>
Committed: Tue Apr 10 08:14:15 2018 +0200
----------------------------------------------------------------------
vagrant/libvirt/README.md | 55 +++++
vagrant/libvirt/dcos/README.md | 81 ++++++
vagrant/libvirt/dcos/Vagrantfile | 94 +++++++
vagrant/libvirt/dcos/ansible.cfg | 76 ++++++
vagrant/libvirt/dcos/check-env.sh | 29 +++
vagrant/libvirt/dcos/config/cluster-1m4a1p.yaml | 70 ++++++
vagrant/libvirt/dcos/config/cluster-3m4a1p.yaml | 82 +++++++
.../group_vars/all/dcos-versions.yaml | 29 +++
.../provisioning/group_vars/all/defaults.yml | 44 ++++
vagrant/libvirt/dcos/provisioning/inventory.py | 232 ++++++++++++++++++
.../provisioning/roles/common/handlers/main.yml | 21 ++
.../provisioning/roles/common/tasks/main.yml | 78 ++++++
.../roles/common/templates/hostname-fix.j2 | 23 ++
.../roles/common/templates/hosts.j2 | 27 ++
.../roles/common/templates/ntp.conf.j2 | 25 ++
.../roles/dcos-install/tasks/main.yml | 43 ++++
.../roles/dcos-setup/tasks/main.yml | 130 ++++++++++
.../provisioning/roles/docker/tasks/main.yml | 63 +++++
.../roles/passwordless/tasks/main.yml | 38 +++
.../provisioning/roles/registry/tasks/main.yml | 32 +++
vagrant/libvirt/dcos/provisioning/site.yml | 63 +++++
vagrant/libvirt/dcos/requirements.txt | 1 +
vagrant/libvirt/dcos/setup-env | 24 ++
vagrant/libvirt/mesos/README.md | 245 +++++++++++++++++++
vagrant/libvirt/mesos/Vagrantfile | 92 +++++++
vagrant/libvirt/mesos/ansible.cfg | 79 ++++++
vagrant/libvirt/mesos/check-env.sh | 28 +++
vagrant/libvirt/mesos/config/cluster-1m4a.yaml | 64 +++++
vagrant/libvirt/mesos/config/cluster-3m4a.yaml | 76 ++++++
vagrant/libvirt/mesos/docs/ansible.txt | 1 +
vagrant/libvirt/mesos/docs/build-mesos.md | 135 ++++++++++
vagrant/libvirt/mesos/docs/notes-myriad.txt | 11 +
vagrant/libvirt/mesos/docs/steps-hadoop.txt | 145 +++++++++++
.../provisioning/group_vars/all/defaults.yml | 100 ++++++++
vagrant/libvirt/mesos/provisioning/inventory.py | 219 +++++++++++++++++
.../provisioning/roles/common/handlers/main.yml | 20 ++
.../provisioning/roles/common/tasks/main.yml | 104 ++++++++
.../roles/common/templates/hostname-fix.j2 | 23 ++
.../roles/common/templates/hosts.j2 | 26 ++
.../roles/common/templates/ntp.conf.j2 | 24 ++
.../provisioning/roles/common/vars/main.yml | 2 +
.../provisioning/roles/devel/handlers/main.yml | 29 +++
.../provisioning/roles/devel/tasks/main.yml | 70 ++++++
.../roles/devel/templates/exports.j2 | 34 +++
.../provisioning/roles/dfs-start/tasks/main.yml | 39 +++
.../provisioning/roles/docker/tasks/main.yml | 63 +++++
.../roles/hadoop/files/core-site.xml | 23 ++
.../roles/hadoop/files/hdfs-site.xml | 34 +++
.../provisioning/roles/hadoop/files/java.sh | 20 ++
.../roles/hadoop/files/mapred-site.xml | 24 ++
.../hadoop/files/myriad-config-default.yml | 76 ++++++
.../provisioning/roles/hadoop/files/slaves | 3 +
.../provisioning/roles/hadoop/files/workers | 3 +
.../roles/hadoop/files/yarn-site.xml | 40 +++
.../provisioning/roles/hadoop/tasks/build.yml | 51 ++++
.../roles/hadoop/tasks/datanode.yml | 50 ++++
.../provisioning/roles/hadoop/tasks/main.yml | 37 +++
.../roles/hadoop/tasks/namenode.yml | 68 +++++
.../roles/nfs-client/tasks/main.yml | 31 +++
.../roles/passwordless/tasks/build.yml | 38 +++
.../roles/passwordless/tasks/main.yml | 24 ++
.../roles/passwordless/tasks/master.yml | 83 +++++++
.../roles/yarn-start/tasks/main.yml | 30 +++
.../provisioning/roles/zookeeper/tasks/main.yml | 51 ++++
vagrant/libvirt/mesos/provisioning/site.yml | 63 +++++
vagrant/libvirt/mesos/setup-env | 23 ++
vagrant/mesos/libvirt/README.md | 55 -----
vagrant/mesos/libvirt/mesos/README.md | 245 -------------------
vagrant/mesos/libvirt/mesos/Vagrantfile | 92 -------
vagrant/mesos/libvirt/mesos/ansible.cfg | 79 ------
vagrant/mesos/libvirt/mesos/check-env.sh | 28 ---
.../libvirt/mesos/config/cluster-1m4a.yaml | 64 -----
.../libvirt/mesos/config/cluster-3m4a.yaml | 76 ------
vagrant/mesos/libvirt/mesos/docs/ansible.txt | 1 -
vagrant/mesos/libvirt/mesos/docs/build-mesos.md | 135 ----------
.../mesos/libvirt/mesos/docs/notes-myriad.txt | 11 -
.../mesos/libvirt/mesos/docs/steps-hadoop.txt | 145 -----------
.../provisioning/group_vars/all/defaults.yml | 100 --------
.../libvirt/mesos/provisioning/inventory.py | 219 -----------------
.../provisioning/roles/common/handlers/main.yml | 20 --
.../provisioning/roles/common/tasks/main.yml | 104 --------
.../roles/common/templates/hostname-fix.j2 | 23 --
.../roles/common/templates/hosts.j2 | 26 --
.../roles/common/templates/ntp.conf.j2 | 24 --
.../provisioning/roles/common/vars/main.yml | 2 -
.../provisioning/roles/devel/handlers/main.yml | 29 ---
.../provisioning/roles/devel/tasks/main.yml | 70 ------
.../roles/devel/templates/exports.j2 | 34 ---
.../provisioning/roles/dfs-start/tasks/main.yml | 39 ---
.../provisioning/roles/docker/tasks/main.yml | 63 -----
.../roles/hadoop/files/core-site.xml | 23 --
.../roles/hadoop/files/hdfs-site.xml | 34 ---
.../provisioning/roles/hadoop/files/java.sh | 20 --
.../roles/hadoop/files/mapred-site.xml | 24 --
.../hadoop/files/myriad-config-default.yml | 76 ------
.../provisioning/roles/hadoop/files/slaves | 3 -
.../provisioning/roles/hadoop/files/workers | 3 -
.../roles/hadoop/files/yarn-site.xml | 40 ---
.../provisioning/roles/hadoop/tasks/build.yml | 51 ----
.../roles/hadoop/tasks/datanode.yml | 50 ----
.../provisioning/roles/hadoop/tasks/main.yml | 37 ---
.../roles/hadoop/tasks/namenode.yml | 68 -----
.../roles/nfs-client/tasks/main.yml | 31 ---
.../roles/passwordless/tasks/build.yml | 38 ---
.../roles/passwordless/tasks/main.yml | 24 --
.../roles/passwordless/tasks/master.yml | 83 -------
.../roles/yarn-start/tasks/main.yml | 30 ---
.../provisioning/roles/zookeeper/tasks/main.yml | 51 ----
.../mesos/libvirt/mesos/provisioning/site.yml | 63 -----
vagrant/mesos/libvirt/mesos/setup-env | 23 --
vagrant/mesos/virtualbox/Vagrantfile | 79 ------
.../virtualbox/provisioning/format_namenode.sh | 23 --
.../provisioning/install_default_jdk.sh | 32 ---
.../virtualbox/provisioning/install_docker.sh | 25 --
.../virtualbox/provisioning/install_hadoop.sh | 115 ---------
.../virtualbox/provisioning/install_mesos.sh | 38 ---
.../mesos/virtualbox/provisioning/shutdown.sh | 30 ---
.../virtualbox/provisioning/start_datanode.sh | 24 --
.../provisioning/start_historyserver.sh | 23 --
.../provisioning/start_mesos_master.sh | 34 ---
.../provisioning/start_mesos_slave.sh | 38 ---
.../virtualbox/provisioning/start_namenode.sh | 27 --
vagrant/virtualbox/mesos/Vagrantfile | 79 ++++++
.../mesos/provisioning/format_namenode.sh | 23 ++
.../mesos/provisioning/install_default_jdk.sh | 32 +++
.../mesos/provisioning/install_docker.sh | 25 ++
.../mesos/provisioning/install_hadoop.sh | 115 +++++++++
.../mesos/provisioning/install_mesos.sh | 38 +++
.../virtualbox/mesos/provisioning/shutdown.sh | 30 +++
.../mesos/provisioning/start_datanode.sh | 24 ++
.../mesos/provisioning/start_historyserver.sh | 23 ++
.../mesos/provisioning/start_mesos_master.sh | 34 +++
.../mesos/provisioning/start_mesos_slave.sh | 38 +++
.../mesos/provisioning/start_namenode.sh | 27 ++
134 files changed, 4249 insertions(+), 2944 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/README.md
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/README.md b/vagrant/libvirt/README.md
new file mode 100644
index 0000000..ced3779
--- /dev/null
+++ b/vagrant/libvirt/README.md
@@ -0,0 +1,55 @@
+# Myriad Vagrant based testbed setup
+
+This is a development environment for the Apache Myriad project based on
+Vagrant and libvirt.
+
+You have to setup properly your environment in order to use Vagrant and
+libvirt. Follow the steps in the nex section for different environments.
+
+# Requeriments
+
+Download Vagrant from HashiCorp site:
+
+https://www.vagrantup.com/downloads.html
+
+## For Ubuntu users
+
+```
+$ sudo apt-get install software-properties-common
+$ sudo apt-add-repository ppa:ansible/ansible
+$ sudo apt-get update
+$ sudo apt-get install ansible
+$ sudo apt-get install qemu-kvm libvirt-bin libvirt-dev
+$ sudo apt-get install ruby-libvirt
+
+$ vagrant plugin install vagrant-libvirt
+Installing the 'vagrant-libvirt' plugin. This can take a few minutes...
+$ vagrant plugin list
+vagrant-libvirt (0.0.40, system)
+
+$ sudo usermod -G libvirtd -a $USER
+$ newgrp -
+```
+
+## For Fedora users
+
+```
+dnf -y install qemu libvirt libvirt-devel ruby-devel gcc
+dnf -y install libxslt-devel libxml2-devel libvirt-devel
+dnf -y install libguestfs-tools-c ruby-devel gcc
+```
+
+Testing Vagrant-libvirt:
+
+```
+$ mkdir test && cd test
+$ vagrant init centos/7
+$ vagrant up --provider libvirt
+$ vagrant box list
+centos/7 (libvirt, 1702.01)
+$ vagrant status
+$ vagrant ssh
+$ vagrant destroy
+$ cd .. && rm -fr test
+```
+
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/README.md
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/README.md b/vagrant/libvirt/dcos/README.md
new file mode 100644
index 0000000..fd43ee4
--- /dev/null
+++ b/vagrant/libvirt/dcos/README.md
@@ -0,0 +1,81 @@
+# Mesosphere DC/OS Community Environment
+
+This is the Vagrant-libvirt based environment for testing Apache Myriad
+project in a DC/OS cluster.
+
+# Setup the environment
+
+In order to use this vagrant environment we have to enable a suitable
+environment. The key points of this environment are:
+
+- **DCOS_ARCH**: This variable has to match with the VMs architecture defined
in
+ the config folder: *cluster-1m4a1p.yaml*, *cluster-3m4a1p.yaml*, and so on.
For
+ example **DCOS_ARCH: 1m4a1p** matchs with the architecture
*cluster-1m4a.yaml*, which
+ is a DC/OS master node (1m), four DC/OS private agents (4a), and one DC/OS
+ public agent (1p).
+
+- **DCOS_VERSION**: This variable set the DC/OS community to download. The next
+ section shows an example of configuration.
+
+```
+$ source setup-env
+
+This Vagrant environment is ready for the following settings:
+
+- DCOS_ARCH: 1m4a1p
+- DCOS_VERSION: 1.11.0
+- HADOOP_VERSION: 2.7.0
+
+'vagrant up --provider=libvirt' and happy hacking!
+```
+
+# Running the deployment
+
+```
+$ vagrant up --provider=libvirt
+[...]
+PLAY RECAP
+*********************************************************************
+bt : ok=36 changed=28 unreachable=0 failed=0
+m1 : ok=28 changed=25 unreachable=0 failed=0
+a1 : ok=31 changed=28 unreachable=0 failed=0
+a2 : ok=31 changed=28 unreachable=0 failed=0
+a3 : ok=31 changed=28 unreachable=0 failed=0
+a4 : ok=31 changed=28 unreachable=0 failed=0
+p1 : ok=31 changed=28 unreachable=0 failed=0
+```
+
+## Final VMs set
+
+```
+$ vagrant status
+Current machine states:
+
+bt running (libvirt)
+m1 running (libvirt)
+a1 running (libvirt)
+a2 running (libvirt)
+a3 running (libvirt)
+a4 running (libvirt)
+p1 running (libvirt)
+```
+
+## Running DC/OS CLI
+
+```
+$ vagrant ssh bt
+[vagrant@bt ~]$ dcos auth login
+If your browser didn't open, please go to the following link:
+
+ http://m1/login?redirect_uri=urn:ietf:wg:oauth:2.0:oob
+
+[vagrant@bt ~]$ dcos node
+ HOSTNAME IP ID TYPE
REGION ZONE
+ 100.1.10.102 100.1.10.102 637fb1cd-f4d4-427a-9418-092b80eb6000-S2 agent
None None
+ 100.1.10.103 100.1.10.103 637fb1cd-f4d4-427a-9418-092b80eb6000-S3 agent
None None
+ 100.1.10.104 100.1.10.104 637fb1cd-f4d4-427a-9418-092b80eb6000-S0 agent
None None
+ 100.1.10.105 100.1.10.105 637fb1cd-f4d4-427a-9418-092b80eb6000-S1 agent
None None
+ 100.1.10.106 100.1.10.106 637fb1cd-f4d4-427a-9418-092b80eb6000-S4 agent
None None
+master.mesos. 100.1.10.101 5ee304dc-18a0-4f7f-b1ef-591c249cbbeb master
(leader) None None
+```
+
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/Vagrantfile
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/Vagrantfile b/vagrant/libvirt/dcos/Vagrantfile
new file mode 100644
index 0000000..886c227
--- /dev/null
+++ b/vagrant/libvirt/dcos/Vagrantfile
@@ -0,0 +1,94 @@
+# -*- mode: ruby -*-
+# vim: set ft=ruby ts=2 et :
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+unless defined? VAGRANTFILE_API_VERSION
+ VAGRANTFILE_API_VERSION = "2"
+end
+
+# Tested with Vagrant version:
+Vagrant.require_version ">= 1.7.2"
+
+# Require YAML module
+require 'yaml'
+
+def fail_with_message(msg)
+ fail Vagrant::Errors::VagrantError.new, msg
+end
+
+def check_minimal_env()
+ if ENV['DCOS_ARCH'].nil? || ENV['DCOS_ARCH'].empty?
+ fail_with_message "You have to set DCOS_ARCH environment variable!"
+ end
+ if ENV['DCOS_VERSION'].nil? || ENV['DCOS_VERSION'].empty?
+ fail_with_message "You have to set DCOS_VERSION environment variable!"
+ end
+ if ENV['HADOOP_VERSION'].nil? || ENV['HADOOP_VERSION'].empty?
+ fail_with_message "You have to set HADOOP_VERSION environment
variable!"
+ end
+end
+
+# Sanity checks
+check_minimal_env()
+
+# Read YAML file with cluster details: The cluster flavor.
+config_file = 'config/cluster-' + ENV['DCOS_ARCH'] + ".yaml"
+if File.exists?(config_file)
+ cluster = YAML.load_file(config_file)
+else
+ fail_with_message "Cluster definition file: #{config_file} was not found!"
+end
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+ # Disabling the default /vagrant share
+ config.vm.synced_folder ".", "/vagrant", disabled: true
+ config.ssh.insert_key = false
+
+ i = 0
+ cluster.each do |servers|
+ # VM definition
+ config.vm.define servers["name"] do |node|
+ node.vm.box = servers["box"]
+ node.vm.hostname = servers["name"]
+ node.vm.network "private_network", ip: servers["ip"]
+ node.vm.provider :libvirt do |domain|
+ domain.uri = 'qemu+unix:///system'
+ domain.driver = 'kvm'
+ domain.memory = servers["mem"]
+ domain.cpus = servers['cpu']
+ if servers.key?('aditional_disk')
+ for disk in servers['aditional_disk']
+ domain.storage :file, :size => disk
+ end
+ end
+ end
+ i += 1
+ if i == cluster.length
+ node.vm.provision :ansible do |ansible|
+ ansible.verbose = "vv"
+ ansible.inventory_path = "provisioning/inventory.py"
+ ansible.playbook = "provisioning/site.yml"
+ ansible.limit = "all"
+ end # provison
+ end # if
+ end # config
+ end # cluster
+end # vagrant
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/ansible.cfg
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/ansible.cfg b/vagrant/libvirt/dcos/ansible.cfg
new file mode 100644
index 0000000..109fedb
--- /dev/null
+++ b/vagrant/libvirt/dcos/ansible.cfg
@@ -0,0 +1,76 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+[defaults]
+# Disable host key checking. If a host is not initially in âknown_hostsâ
this
+# will result in prompting for confirmation of the key, which results in an
+# interactive experience
+host_key_checking = False
+
+# This is the default SSH timeout to use on connection attempts.
+timeout = 30
+
+# This options forces color mode even when running without a TTY.
+force_color = 1
+
+# Number of parallel processes to spawn when
+# communicating with remote hosts.
+forks = 10
+
+# Facts gathering: The value âsmartâ means each new
+# host that has no facts discovered will be scanned,
+# but if the same host is addressed in multiple plays
+# it will not be # contacted again in the playbook run.
+# This ave fact gathering time.
+gathering = smart
+
+# Ansible will log information about executions at the designated location
+log_path=.vagrant/ansible.log
+
+# The retry files save path is where Ansible will save .retry files when a
+# playbook fails. The directory will be created if it does not already exist.
+retry_files_save_path = .vagrant/retry-files
+
+# Play execution strategies: Plays execution control for many hosts, controls
+# how plays run their multiple hosts.
+#
+# linear (default): All hosts of a play have to finish one task before any of
them can
+# begin the next one. Using the number of forks (default 5)
+# to parallelize.
+# serial: Ensures one group of hosts in a play finishes its work before
another group
+# can begin. This strategy will take a subset of hosts (the default is
+# five) and execute all tasks (in the linear fashion) against those
+# hosts before moving to the next subset and starting from the
beginning.
+# free: Allows each host to run until the end of the play as fast as it can.
+# Allows all hosts to run independently of each other. âfreeâ lets
each
+# host run independently, starting its next task as soon as it finishes
+# the previous one, regardless of how far other hosts have gotten.
+#
+# By default Ansible will attempt to run on all hosts from a play in parallel.
+# With the serial strategy ansible will attempt to run on by defauult 5 of the
+# play.
+# It is designed so that each task will be run on all hosts before continuing
on
+# to the next task. So if you have 3 tasks it will ensure task 1 runs on all
+# your hosts first, then task 2 is run, then task 3 is run. This is the
"linear"
+# strategy.
+# Each play contains a list of tasks. Tasks are executed in order, one at a
+# time, against all machines matched by the host pattern of the play, before
moving on to
+# the next task. It is important to understand that, within a play, all hosts
+# are going to get the same task directives. It is the purpose of a play to map
+# a selection of hosts to tasks.
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/check-env.sh
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/check-env.sh
b/vagrant/libvirt/dcos/check-env.sh
new file mode 100644
index 0000000..d4c0d9a
--- /dev/null
+++ b/vagrant/libvirt/dcos/check-env.sh
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+echo
+echo "This Vagrant environment is ready for the following settings:"
+echo
+
+[ -z $DCOS_ARCH ] || echo "- DCOS_ARCH: $DCOS_ARCH"
+[ -z $DCOS_VERSION ] || echo "- DCOS_VERSION: $DCOS_VERSION"
+[ -z $HADOOP_VERSION ] || echo "- HADOOP_VERSION: $HADOOP_VERSION"
+
+echo
+echo "'vagrant up --provider=libvirt' and happy hacking!"
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/config/cluster-1m4a1p.yaml
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/config/cluster-1m4a1p.yaml
b/vagrant/libvirt/dcos/config/cluster-1m4a1p.yaml
new file mode 100644
index 0000000..ecef066
--- /dev/null
+++ b/vagrant/libvirt/dcos/config/cluster-1m4a1p.yaml
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+- name: bt
+ type: bootstrap
+ box: centos/7
+ ip: 100.1.10.100
+ mem: 2048
+ cpu: 2
+- name: m1
+ type: master
+ box: centos/7
+ ip: 100.1.10.101
+ mem: 2048
+ cpu: 2
+- name: a1
+ type: agent-private
+ box: centos/7
+ ip: 100.1.10.102
+ mem: 2048
+ cpu: 2
+- name: a2
+ type: agent-private
+ box: centos/7
+ ip: 100.1.10.103
+ mem: 2048
+ cpu: 2
+ aditional_disk:
+ - 50G
+ - 50G
+- name: a3
+ type: agent-private
+ box: centos/7
+ ip: 100.1.10.104
+ mem: 2048
+ cpu: 2
+ aditional_disk:
+ - 50G
+ - 50G
+- name: a4
+ type: agent-private
+ box: centos/7
+ ip: 100.1.10.105
+ mem: 2048
+ cpu: 2
+ aditional_disk:
+ - 50G
+ - 50G
+- name: p1
+ type: agent-public
+ box: centos/7
+ ip: 100.1.10.106
+ mem: 2048
+ cpu: 2
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/config/cluster-3m4a1p.yaml
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/config/cluster-3m4a1p.yaml
b/vagrant/libvirt/dcos/config/cluster-3m4a1p.yaml
new file mode 100644
index 0000000..48d438f
--- /dev/null
+++ b/vagrant/libvirt/dcos/config/cluster-3m4a1p.yaml
@@ -0,0 +1,82 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+- name: bt
+ type: bootstrap
+ box: centos/7
+ ip: 100.1.10.100
+ mem: 512
+ cpu: 1
+- name: m1
+ type: master
+ box: centos/7
+ ip: 100.1.10.101
+ mem: 1024
+ cpu: 2
+- name: m2
+ type: master
+ box: centos/7
+ ip: 100.1.10.102
+ mem: 1024
+ cpu: 2
+- name: m3
+ type: master
+ box: centos/7
+ ip: 100.1.10.103
+ mem: 1024
+ cpu: 2
+- name: a1
+ type: agent-private
+ box: centos/7
+ ip: 100.1.10.104
+ mem: 2048
+ cpu: 2
+- name: a2
+ type: agent-private
+ box: centos/7
+ ip: 100.1.10.105
+ mem: 2048
+ cpu: 2
+ aditional_disk:
+ - 50G
+ - 50G
+- name: a3
+ type: agent-private
+ box: centos/7
+ ip: 100.1.10.106
+ mem: 2048
+ cpu: 2
+ aditional_disk:
+ - 50G
+ - 50G
+- name: a4
+ type: agent-private
+ box: centos/7
+ ip: 100.1.10.107
+ mem: 2048
+ cpu: 2
+ aditional_disk:
+ - 50G
+ - 50G
+- name: p1
+ type: agent-public
+ box: centos/7
+ ip: 100.1.10.108
+ mem: 2048
+ cpu: 2
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/group_vars/all/dcos-versions.yaml
----------------------------------------------------------------------
diff --git
a/vagrant/libvirt/dcos/provisioning/group_vars/all/dcos-versions.yaml
b/vagrant/libvirt/dcos/provisioning/group_vars/all/dcos-versions.yaml
new file mode 100644
index 0000000..c4d5dcb
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/group_vars/all/dcos-versions.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+versions:
+ '1.11.0':
+ channel: 'stable/1.11.0'
+ ref: 'b6d6ad4722600877fde2860122f870031d109da3'
+ sha256: 'f42800c773a79c8fa4b5ae837626805ac67420d388433bbc9891520b17668ba9'
+ cli: 'dcos-1.11'
+ '1.10.4':
+ channel: 'stable/1.10.4'
+ ref: '2d45a8f9e277a60007f277f70f01d076c913a7fe'
+ sha256: '108b0d66af1a96508219627f7dd897b3110235eb1d9729ae5a576025192ccb61'
+ cli: 'dcos-1.10'
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/group_vars/all/defaults.yml
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/provisioning/group_vars/all/defaults.yml
b/vagrant/libvirt/dcos/provisioning/group_vars/all/defaults.yml
new file mode 100644
index 0000000..07e69e5
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/group_vars/all/defaults.yml
@@ -0,0 +1,44 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+# Minimal CentOS system packages
+system_packages:
+ - nc
+ - tar
+ - wget
+ - git
+ - docker-python
+ - epel-release
+ - vim
+ - tree
+ - ntp
+ - net-tools
+ - bind-utils
+ - xz
+ - unzip
+ - curl
+ - ipset
+ - sysstat
+
+system_epel_packages:
+ - pdsh
+ - sshpass
+
+with_dcos_version: "{{ lookup('env','DCOS_VERSION') }}"
+with_hadoop_version: "{{ lookup('env','HADOOP_VERSION') }}"
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/inventory.py
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/provisioning/inventory.py
b/vagrant/libvirt/dcos/provisioning/inventory.py
new file mode 100755
index 0000000..bee7313
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/inventory.py
@@ -0,0 +1,232 @@
+#!/usr/bin/env python2
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import yaml
+import os
+import sys
+import paramiko
+import socket
+import time
+import logging
+
+
+def get_cluster_config_file():
+ '''Returns the file used as DCOS architecture'''
+ try:
+ ret = os.environ["DCOS_ARCH"]
+ except KeyError:
+ logging.debug(
+ "[Ansible] Please set environment variable DCOS_ARCH")
+ sys.exit(1)
+
+ return "config/cluster-" + ret + ".yaml"
+
+
+def get_cluster_config_yml():
+ '''Returns a list with hostname and IP'''
+ with open(get_cluster_config_file(), 'r') as f:
+ ret = yaml.load(f)
+
+ return ret
+
+
+def check_ssh(ip, user, key_file, initial_wait=0, interval=0, retries=1):
+ logging.debug("[Ansible] checking SSH availability for %s", ip)
+ ssh = paramiko.SSHClient()
+ ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+ time.sleep(initial_wait)
+
+ for x in range(retries):
+ try:
+ ssh.connect(ip, username=user, key_filename=key_file)
+ return True
+ except (paramiko.BadHostKeyException,
+ paramiko.AuthenticationException,
+ paramiko.SSHException,
+ socket.error) as e:
+ logging.debug(e)
+ time.sleep(interval)
+
+ return False
+
+
+def check_ssh_available(cluster_yml):
+ ssh_key_path = os.environ["HOME"] + "/.vagrant.d/insecure_private_key"
+
+ # check if the hosts are ssh accesibles
+ for item in cluster_yml:
+ if check_ssh(item.get('ip'),
+ "vagrant",
+ ssh_key_path,
+ 2, 2, 3):
+ logging.debug(
+ "[Ansible] %s: SSH is OK for provisioning", item)
+ else:
+ logging.debug("[Ansible] %s: SSH not ready", item)
+ return False
+
+ return True
+
+
+logging.basicConfig(filename='.vagrant/inventory.log', level=logging.DEBUG)
+
+all_vm_accesibles = False
+logging.debug('[Ansible] getting host list from configuration')
+cluster_yml = get_cluster_config_yml()
+
+
+logging.debug("[Ansible] Sanity check loop for Ansible hosts")
+while not all_vm_accesibles:
+ logging.debug("[Ansible] Waiting for SSH to become available in all hosts")
+ if check_ssh_available(cluster_yml):
+ all_vm_accesibles = True
+
+
+class InventoryTemplate:
+ '''
+ {
+ "all": {
+ "hosts": ["bt","a1","a2","p1","m1","m2","m3"],
+ "vars": {
+ "ansible_user": "vagrant",
+ "ansible_become": "true"
+ }
+ },
+ "bootstrap": {
+ "hosts": ["bt"]
+ },
+ "dcos-masters": {
+ "hosts": ["m1","m3","m2"]
+ },
+ "dcos-agents-private": {
+ "hosts": ["a1","a2"]
+ },
+ "dcos-agents-public": {
+ "hosts": ["p1"]
+ },
+ "_meta": {
+ "hostvars": {
+ "bt": {"ansible_host": "192.168.121.56"},
+ "p1": {"ansible_host": "192.168.121.30"},
+ "a1": {"ansible_host": "192.168.121.248"},
+ "a2": {"ansible_host": "192.168.121.112"},
+ "m1": {"ansible_host": "192.168.121.80"},
+ "m2": {"ansible_host": "192.168.121.105"},
+ "m3": {"ansible_host": "192.168.121.150"}
+ }
+ }
+ }
+ '''
+
+ _template = """
+ {
+ "all": {
+ "hosts": [%(_get_all|_pattern_a)s],
+ "vars": {
+ "ansible_user": "vagrant",
+ "ansible_become": "true"
+ }
+ },
+ "bootstrap": {
+ "hosts": ["bt"]
+ },
+ "dcos-masters": {
+ "hosts": [%(_get_masters|_pattern_a)s]
+ },
+ "dcos-agents-private": {
+ "hosts": [%(_get_privates|_pattern_a)s]
+ },
+ "dcos-agents-public": {
+ "hosts": [%(_get_publics|_pattern_a)s]
+ },
+ "_meta": {
+ "hostvars": {
+ %(_get_all_hostip|_pattern_b)s
+ }
+ }
+ }
+ """
+
+ def __init__(self, dict={}):
+ self.dict = dict
+
+ def __str__(self):
+ return self._template % self
+
+ def __getitem__(self, key):
+ return self._process(key.split("|"))
+
+ def _process(self, l):
+ arg = l[0]
+ if len(l) == 1:
+ if arg in self.dict:
+ return self.dict[arg]
+ elif hasattr(self, arg) and callable(getattr(self, arg)):
+ return getattr(self, arg)()
+ else:
+ raise KeyError(arg)
+ else:
+ func = l[1]
+ return getattr(self, func)(self._process([arg]))
+
+ def _get_all(self):
+ cad = []
+ for i in cluster_yml:
+ cad.append(i.get('name'))
+ return cad
+
+ def _get_masters(self):
+ cad = []
+ for i in cluster_yml:
+ if i.get('type') == 'master':
+ cad.append(i.get('name'))
+ return cad
+
+ def _get_privates(iself):
+ cad = []
+ for i in cluster_yml:
+ if i.get('type') == 'agent-private':
+ cad.append(i.get('name'))
+ return cad
+
+ def _get_publics(iself):
+ cad = []
+ for i in cluster_yml:
+ if i.get('type') == 'agent-public':
+ cad.append(i.get('name'))
+ return cad
+
+ def _get_all_hostip(self):
+ cad = []
+ for i in cluster_yml:
+ cad.append("\"%s\": {\"ansible_host\": \"%s\"}" % (i.get('name'),
+ i.get('ip')))
+ return cad
+
+ def _pattern_a(self, l):
+ return ",".join(["\"%s\"" % x for x in l])
+
+ def _pattern_b(self, l):
+ return ",".join(["%s" % x for x in l])
+
+
+print InventoryTemplate()
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/roles/common/handlers/main.yml
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/provisioning/roles/common/handlers/main.yml
b/vagrant/libvirt/dcos/provisioning/roles/common/handlers/main.yml
new file mode 100644
index 0000000..869f483
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/roles/common/handlers/main.yml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+- name: Restart ntp
+ service: name=ntpd state=restarted
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/roles/common/tasks/main.yml
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/provisioning/roles/common/tasks/main.yml
b/vagrant/libvirt/dcos/provisioning/roles/common/tasks/main.yml
new file mode 100644
index 0000000..f814085
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/roles/common/tasks/main.yml
@@ -0,0 +1,78 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+- name: Sanity checking Ansible version
+ run_once: true
+ delegate_to: localhost
+ assert:
+ that:
+ - ansible_version.full|version_compare('2.2.0.0', '>')
+ msg: "Please install at least ansible 2.2.x.x in your host"
+
+- name: Install CentOS prerequisites
+ yum: name={{ item }} state=latest
+ with_items:
+ - "{{ system_packages }}"
+
+- name: Install CentOS EPEL prerequisites
+ yum: name={{ item }} state=latest
+ with_items:
+ - "{{ system_epel_packages }}"
+
+- name: Disabling SELinux (anti-pattern)
+ selinux:
+ policy: targeted
+ state: permissive
+
+- name: Effective disabling of SELinux (avoiting reboot)
+ command: /usr/sbin/setenforce 0
+
+- name: Adding group nogroup (similar to nobody)
+ group:
+ name: nogroup
+ state: present
+
+- name: Stopping Firewall rules (anti-pattner)
+ service: name=firewalld state=stopped enabled=no
+
+- name: Effective disabling of iptables rules
+ command: /usr/sbin/iptables -F
+
+- name: Disabling Chronyd NTP server
+ service: name=chronyd state=stopped enabled=no
+
+- name: Passwordless pre-step PasswordAuthentication enabling
+ lineinfile:
+ dest: /etc/ssh/sshd_config
+ backup: yes
+ regexp: '^PasswordAuthentication'
+ line: 'PasswordAuthentication yes'
+
+- name: Restarting SSH daemon
+ service: name=sshd state=restarted
+
+- name: Configure /etc/host files
+ template: src=hosts.j2 dest=/etc/hosts
+
+- name: Configure ntp servers
+ template: src=ntp.conf.j2 dest=/etc/ntp.conf
+ notify: Restart ntp
+
+- name: Enabled ntpd service
+ service: name=ntpd enabled=yes
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/roles/common/templates/hostname-fix.j2
----------------------------------------------------------------------
diff --git
a/vagrant/libvirt/dcos/provisioning/roles/common/templates/hostname-fix.j2
b/vagrant/libvirt/dcos/provisioning/roles/common/templates/hostname-fix.j2
new file mode 100644
index 0000000..6a39a55
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/roles/common/templates/hostname-fix.j2
@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+if ! grep -e "^127\.0\.1\.1.*`hostname`.*" /etc/hosts > /dev/null ; then
+ echo "127.0.1.1 `hostname`" >> /etc/hosts
+fi
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/roles/common/templates/hosts.j2
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/provisioning/roles/common/templates/hosts.j2
b/vagrant/libvirt/dcos/provisioning/roles/common/templates/hosts.j2
new file mode 100644
index 0000000..b273ea1
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/roles/common/templates/hosts.j2
@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# {{ ansible_managed }}
+127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
+::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
+
+{% for item in play_hosts %}
+{% set short_name = item.split('.') %}
+{{ hostvars[item]['ansible_host'] }} {{ item }}
+{% endfor %}
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/roles/common/templates/ntp.conf.j2
----------------------------------------------------------------------
diff --git
a/vagrant/libvirt/dcos/provisioning/roles/common/templates/ntp.conf.j2
b/vagrant/libvirt/dcos/provisioning/roles/common/templates/ntp.conf.j2
new file mode 100644
index 0000000..0c54033
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/roles/common/templates/ntp.conf.j2
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+driftfile /var/lib/ntp/ntp.drift
+
+server 0.pool.ntp.org
+server 1.pool.ntp.org
+server 2.pool.ntp.org
+server 3.pool.ntp.org
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/roles/dcos-install/tasks/main.yml
----------------------------------------------------------------------
diff --git
a/vagrant/libvirt/dcos/provisioning/roles/dcos-install/tasks/main.yml
b/vagrant/libvirt/dcos/provisioning/roles/dcos-install/tasks/main.yml
new file mode 100644
index 0000000..5ec6580
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/roles/dcos-install/tasks/main.yml
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+- name: Waiting for bootstrap healthy state
+ wait_for:
+ host: bt
+ port: 8080
+ delay: 5
+ timeout: 1200
+ tags: install
+
+- name: Download dcos_install.sh DC/OS Installer
+ get_url:
+ url: http://bt:8080/dcos_install.sh
+ dest: /home/vagrant/dcos_install.sh
+ mode: 0655
+ tags: install
+
+# master, slave, slave_public
+- name: Executing DC/OS Node Installation
+ shell: >
+ bash dcos_install.sh "{{ node_type }}" &&
+ touch .skip
+ args:
+ chdir: /home/vagrant
+ creates: .skip
+ tags: install
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/roles/dcos-setup/tasks/main.yml
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/provisioning/roles/dcos-setup/tasks/main.yml
b/vagrant/libvirt/dcos/provisioning/roles/dcos-setup/tasks/main.yml
new file mode 100644
index 0000000..35e8af7
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/roles/dcos-setup/tasks/main.yml
@@ -0,0 +1,130 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+- name: Install CentOS prerequisites for Docker Ansible Module
+ yum: name={{ item }} state=latest
+ with_items:
+ - docker-python
+
+- name: Start and enable DNSMasq in Bootstrap
+ service: name=dnsmasq state=started enabled=yes
+
+- name: Pull NGINX Docker Image
+ docker_image:
+ name: nginx
+
+- name: Create working folder (genconf)
+ file: path=/home/vagrant/genconf state=directory
+
+- name: Download dcos_generate_config.sh installer (~1GB this process might
takes a long time)
+ shell: >
+ curl -fLsS --retry 20 -Y 100000 -y 60
+ https://downloads.dcos.io/dcos/{{ versions[with_dcos_version].channel
}}/commit/{{ versions[with_dcos_version].ref }}/dcos_generate_config.sh
+ -o dcos_generate_config.sh && touch .installerskip
+ args:
+ chdir: /home/vagrant
+ creates: .installerskip
+
+- name: Configure ip-detect helper for correct interface attaching
+ blockinfile:
+ dest: /home/vagrant/genconf/ip-detect
+ create: yes
+ marker: REMOVE
+ block: |
+ #!/usr/bin/env bash
+ set -o nounset -o errexit
+ export PATH=/usr/sbin:/usr/bin:$PATH
+ echo $(ip addr show eth1 | grep -Eo
'[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | head -1)
+
+- name: Configure config.yaml for DC/OS deployment
+ blockinfile:
+ dest: /home/vagrant/genconf/config.yaml
+ create: yes
+ marker: REMOVE
+ block: |
+ ---
+ bootstrap_url: http://bt:8080
+ cluster_name: 'dcos-vagrant'
+ exhibitor_storage_backend: static
+ ip_detect_filename: /genconf/ip-detect
+ master_discovery: static
+ master_list:
+ - 100.1.10.101
+ resolvers:
+ - 8.8.8.8
+ ssh_port: 22
+ ssh_user: vagrant
+ superuser_username: admin
+ superuser_password_hash:
$6$rounds=656000$J1uOqAb95/iblDzm$ejRz.4wZiBJyKK0Wlmv4zH4c1bRbO0/P1KbOLSm8NJfQChzWcDJ.twmbCq9LS83bWrt7S6ZaOzicwN/bBnC3S1
+ check_time: false
+
+- name: Remove markers from module blockinfile (workaround)
+ lineinfile:
+ dest: "{{ item }}"
+ regexp: "^REMOVE"
+ state: absent
+ with_items:
+ - "/home/vagrant/genconf/ip-detect"
+ - "/home/vagrant/genconf/config.yaml"
+
+- name: Generating DC/OS config
+ shell: bash dcos_generate_config.sh && touch .skip
+ args:
+ chdir: /home/vagrant
+ creates: .skip
+
+# FIXME: hardcoded IP of bootstrap node
+- name: Running dockerized Nginx for installation
+ docker_container:
+ name: nginx-bootstrap
+ image: nginx
+ ports:
+ - "100.1.10.100:8080:80"
+ volumes:
+ - /home/vagrant/genconf/serve:/usr/share/nginx/html
+
+- name: Downloading DC/OS CLI
+ shell: >
+ curl -fLsS --retry 20 -Y 100000 -y 60
+ https://downloads.dcos.io/binaries/cli/linux/x86-64/dcos-1.11/dcos
+ https://downloads.dcos.io/binaries/cli/linux/x86-64/{{
versions[with_dcos_version].cli }}/dcos
+ -o dcos && touch .cliskip
+ args:
+ chdir: /home/vagrant
+ creates: .cliskip
+ tags: cli
+
+- name: Copy DC/OS CLI to suitable PATH location
+ shell: cp -f dcos /usr/local/bin/dcos && chmod 755 /usr/local/bin/dcos &&
touch .skipcopy
+ args:
+ chdir: /home/vagrant
+ creates: .skipcopy
+ tags: cli
+
+- name: Delete DC/OS CLI downloaded
+ file:
+ path: /home/vagrant/dcos
+ state: absent
+ tags: cli
+
+- name: Configure current configuration for master
+ shell: /usr/local/bin/dcos config set core.dcos_url http://m1
+ become: yes
+ become_user: vagrant
+ tags: cli
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/roles/docker/tasks/main.yml
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/provisioning/roles/docker/tasks/main.yml
b/vagrant/libvirt/dcos/provisioning/roles/docker/tasks/main.yml
new file mode 100644
index 0000000..e9e0b68
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/roles/docker/tasks/main.yml
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+- name: Setup OverlayFS storage driver automatic loading
+ lineinfile:
+ dest: /etc/modules-load.d/overlay.conf
+ create: yes
+ state: present
+ line: overlay
+
+- name: Load overlay kernel module (avoiting reboot)
+ modprobe: name=overlay state=present
+
+- name: Setup offial Docker repository
+ blockinfile:
+ dest: /etc/yum.repos.d/docker.repo
+ backup: yes
+ create: yes
+ block: |
+ [dockerrepo]
+ name=Docker Repository
+ baseurl=https://yum.dockerproject.org/repo/main/centos/7/
+ enabled=1
+ gpgcheck=1
+ gpgkey=https://yum.dockerproject.org/gpg
+
+- name: Configure systemd to run the Docker Daemon with OverlayFS (folder)
+ file: path=/etc/systemd/system/docker.service.d state=directory
+
+- name: Configure systemd to run the Docker Daemon with OverlayFS (control
file)
+ blockinfile:
+ dest: /etc/systemd/system/docker.service.d/override.conf
+ backup: yes
+ create: yes
+ block: |
+ [Service]
+ ExecStart=
+ ExecStart=/usr/bin/docker daemon --insecure-registry bt:5000
--storage-driver=overlay -H fd://
+
+- name: Installing Docker 1.11.12 version
+ yum: name=docker-engine-1.11.2 state=installed
+
+- name: Starting Docker daemon
+ service: name=docker state=started
+
+- name: Enabling Docker daemon
+ service: name=docker enabled=yes
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/roles/passwordless/tasks/main.yml
----------------------------------------------------------------------
diff --git
a/vagrant/libvirt/dcos/provisioning/roles/passwordless/tasks/main.yml
b/vagrant/libvirt/dcos/provisioning/roles/passwordless/tasks/main.yml
new file mode 100644
index 0000000..6330a85
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/roles/passwordless/tasks/main.yml
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+- name: Generate RSA key-pair for passwordless
+ shell: ssh-keygen -t rsa -f .ssh/id_rsa -q -N "" && touch .skiprsa
+ become: yes
+ become_user: vagrant
+ args:
+ chdir: /home/vagrant
+ creates: .skiprsa
+
+- name: Install boot host public key in a remote authorized_keys
+ shell: >
+ sshpass -p vagrant ssh-copy-id -i .ssh/id_rsa {{ item }} -o
StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+ become: yes
+ become_user: vagrant
+ args:
+ chdir: /home/vagrant
+ with_inventory_hostnames:
+ - all
+
+
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/roles/registry/tasks/main.yml
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/provisioning/roles/registry/tasks/main.yml
b/vagrant/libvirt/dcos/provisioning/roles/registry/tasks/main.yml
new file mode 100644
index 0000000..d365b4e
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/roles/registry/tasks/main.yml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+- name: Pull Docker Registry Image
+ docker_image:
+ name: registry
+ tags: registy
+
+- name: Running dockerized Nginx for installation
+ docker_container:
+ name: registry
+ image: registry
+ ports:
+ - "5000:5000"
+ restart_policy: always
+ tags: registry
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/provisioning/site.yml
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/provisioning/site.yml
b/vagrant/libvirt/dcos/provisioning/site.yml
new file mode 100644
index 0000000..416aadf
--- /dev/null
+++ b/vagrant/libvirt/dcos/provisioning/site.yml
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+# - Plays are map roles to hosts.
+# - Roles are just abstractions around tasks
+# - Each "hosts:"-block (a play) will be executed
+# one by one. So you can use multiple
+# blocks to force the desired execution order.
+# - The plays are executed from top to down order.
+# - Multiple roles declarations per play are executed
+# non in special order, you cannot control the order
+# of roles within a play.
+
+# Play 0
+- hosts: all
+ roles:
+ - common
+ - docker
+
+# Play 1
+- hosts: bootstrap
+ roles:
+ - passwordless
+ - dcos-setup
+ - registry
+
+# Play 2
+- hosts: dcos-masters
+ strategy: free
+ vars:
+ node_type: master
+ roles:
+ - dcos-install
+
+- hosts: dcos-agents-private
+ strategy: free
+ vars:
+ node_type: slave
+ roles:
+ - dcos-install
+
+- hosts: dcos-agents-public
+ strategy: free
+ vars:
+ node_type: slave_public
+ roles:
+ - dcos-install
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/requirements.txt
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/requirements.txt
b/vagrant/libvirt/dcos/requirements.txt
new file mode 100644
index 0000000..cd696f0
--- /dev/null
+++ b/vagrant/libvirt/dcos/requirements.txt
@@ -0,0 +1 @@
+sudo dnf install ansible -y
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/dcos/setup-env
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/dcos/setup-env b/vagrant/libvirt/dcos/setup-env
new file mode 100644
index 0000000..6296221
--- /dev/null
+++ b/vagrant/libvirt/dcos/setup-env
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+export DCOS_ARCH=1m4a1p
+export DCOS_VERSION=1.11.0
+export HADOOP_VERSION=2.7.0
+
+sh check-env.sh
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/mesos/README.md
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/mesos/README.md b/vagrant/libvirt/mesos/README.md
new file mode 100644
index 0000000..02d6584
--- /dev/null
+++ b/vagrant/libvirt/mesos/README.md
@@ -0,0 +1,245 @@
+# Plain Apache Mesos Environment
+
+This is the Vagrant-libvirt based environment for testing Apache Myriad
+project in a Mesos cluster.
+
+# Setup the environment
+
+In order to use this vagrant environment we have to enable a suitable
+environment. The key points of this environment are:
+
+- **MESOS_ARCH**: This variable has to match with the VMs architecture defined
in
+ the config folder: *cluster-1m4a.yaml*, *cluster-3m4a.yaml*, and so on. For
+ example **MESOS_ARCH: 1m4a** matchs with the architecture
*cluster-1m4a.yaml*, which
+ is a Mesos master node (1m), and four Mesos agents (4a).
+
+- **HADOOP_VERSION**: This variable is used for downloading a binary
distribution
+ of Apache HAdoop.
+
+- **ZOOKEEPER_VERSION**: For dowloading an Apache Zookeeper binary
distribution.
+
+```
+$ source setup-env
+
+This Vagrant environment is ready for the following settings:
+
+- MESOS_ARCH: 1m4a
+- HADOOP_VERSION: 2.7.0
+- ZOOKEEPER_VERSION: 3.4.11
+
+'vagrant up --provider=libvirt' and happy hacking!
+```
+
+# Running the deployment
+
+The VMs provisioning is based on Ansible, the provisioning is guided by the
+main playbook **provisioning/site.yml** and one special
**provisioing/inventory.py**
+for creating a dynamic inventory. This dynamic inventory is created for
+managing the Ansible provisioning with a multi-machine Vagrant environment,
+because the Vagrant Ansible support is sometime problematic.
+
+```
+$ vagrant up --provider=libvirt
+[...]
+PLAY RECAP
+*********************************************************************
+build : ok=36 changed=28 unreachable=0 failed=0
+mesos-a1 : ok=28 changed=25 unreachable=0 failed=0
+mesos-a2 : ok=31 changed=28 unreachable=0 failed=0
+mesos-a3 : ok=31 changed=28 unreachable=0 failed=0
+mesos-a4 : ok=31 changed=28 unreachable=0 failed=0
+mesos-m1 : ok=41 changed=38 unreachable=0 failed=0
+```
+
+## Final VMs set
+
+```
+$ vagrant status
+Current machine states:
+
+build running (libvirt)
+mesos-m1 running (libvirt)
+mesos-a1 running (libvirt)
+mesos-a2 running (libvirt)
+mesos-a3 running (libvirt)
+mesos-a4 running (libvirt)
+```
+
+The VM dedicated for development is *build*, the development is maded in the
+*/opt* folder:
+
+```
+$ vagrant ssh build
+[vagrant@build ~]$ tree -d -L 1 /opt/
+/opt/
+âââ hadoop
+âââ mesos
+âââ myriad
+```
+
+# Building the ecosystem
+
+The strategy is to build the target Apache Mesos version from source, and
+build Myriad framework as follow:
+
+## Building Apache Mesos
+
+```
+[vagrant@build ~]$ cd /opt/mesos
+[vagrant@build ~]$ git checkout 0.28.1
+[vagrant@build ~]$ ./bootstrap
+[vagrant@build ~]$ mkdir build
+[vagrant@build ~]$ cd build
+[vagrant@build ~]$ ../configure
+[vagrant@build ~]$ make
+[...]
+running install_scripts
+creating build/bdist.linux-x86_64/wheel/mesos.native-0.28.1.dist-info/WHEEL
+make[2]: Leaving directory `/opt/mesos/build/src'
+make[1]: Leaving directory `/opt/mesos/build/src'
+```
+
+## Building Myriad Framework
+
+```
+[vagrant@build myriad]$ cd /opt/myriad
+[vagrant@build myriad]$ ./gradlew build
+[...]
+:myriad-scheduler:test
+:myriad-scheduler:check
+:myriad-scheduler:build
+
+BUILD SUCCESSFUL
+
+Total time: 3 mins 56.948 secs
+```
+
+# Running the ecosystem
+
+The ecosystem was built from **build** VM, however the execution of ecosystem
+is carried out from **mesos-m1** VM and mesos agents.
+
+## Running Mesos 3rdparty shipped Zookeeper
+
+We have to run a Zookeeper server, we can use the shipped Zookeeper expecified
+at ZOOKEEPER_VERSION, or we can use the already shipped Zookeeper at 3rdparty
+Mesos folder. Let see who to run the 3rdparty Zookeeper shipped with Mesos.
+
+```
+$ vagrant ssh mesos-m1
+[vagrant@mesos-m1 ~]$ cd /opt/mesos/build/3rdparty/zookeeper-3.4.5/
+[vagrant@mesos-m1 zookeeper-3.4.5]$ cp conf/zoo_sample.cfg conf/zoo.cfg
+[vagrant@mesos-m1 zookeeper-3.4.5]$ echo "server.1=mesos-m1:2888:3888" >>
conf/zoo.cfg
+[vagrant@mesos-m1 zookeeper-3.4.5]$ bin/zkServer.sh start
+[vagrant@mesos-m1 zookeeper-3.4.5]$ echo ruok | nc 127.0.0.1 2181
+imok
+[vagrant@mesos-m1 ~]$ jps
+16290 Jps
+16253 QuorumPeerMain
+```
+
+## Running Apache Mesos
+
+- At mesos-m1
+
+```
+[vagrant@mesos-m1 ~]$ cd /opt/mesos/build
+[vagrant@mesos-m1 build]$ sudo ./bin/mesos-master.sh --ip=100.0.10.101
--work_dir=/var/lib/mesos --zk=zk://mesos-m1:2181/mesos --quorum=1
+```
+
+- At agents mesos-a[1..4]:
+
+```
+[vagrant@mesos-a1 ~]$ cd /opt/mesos/build
+[vagrant@mesos-a1 build]$ sudo ./bin/mesos-slave.sh
--master=zk://mesos-m1:2181/mesos --work_dir=/var/lib/mesos
+```
+
+*Note*: mesos-slave.sh was changed for newer versions of Mesos for
mesos-agent.sh
+
+The Mesos Master web interface can be accessed via the URL:
+
+http://100.0.10.101:5050
+
+## Running Hadoop HDFS
+
+The Ansible provisioning must have Apache HDFS already running in the platform.
+For checking we can do the following:
+
+- At mesos-m1 master:
+
+```
+[vagrant@mesos-m1 ~]$ su - hdfs
+Password: (vagrant is the password)
+[hdfs@mesos-m1 ~]$ jps
+21249 NameNode
+16374 Jps
+21448 SecondaryNameNode
+[hdfs@mesos-m1 hadoop]$ cd /opt/hadoop
+[hdfs@mesos-m1 hadoop]$ bin/hdfs dfsadmin -printTopology
+Rack: /default-rack
+ 100.0.10.103:50010 (mesos-a2)
+ 100.0.10.104:50010 (mesos-a3)
+ 100.0.10.105:50010 (mesos-a4)
+[hdfs@mesos-m1 hadoop]$ bin/hdfs dfsadmin -report
+[...]
+```
+
+The NN web interface can be accessed via the URL:
+
+http://100.0.10.101:50070/
+
+*Note*: Hadoop HDFS is running in mesos-a2, mesos-a3 and mesos-a4.
+The Mesos agent mesos-a1 is dedicated for running Mesos tasks not related
+with Hadoop.
+
+*Note*: You can stop or start HDFS manually from master node with:
+
+```
+[hdfs@mesos-m1 hadoop]$ cd /opt/hadoop
+[hdfs@mesos-m1 hadoop]$ sbin/stop-dfs.sh
+[hdfs@mesos-m1 hadoop]$ sbin/stop-dfs.sh
+```
+
+
+## Running Hadoop YARN Resource Manager (RM) with Myriad
+
+- At mesos-m1:
+
+```
+cd /opt
+
+# Copy the Myriad Scheduler
+cp -vf myriad/myriad-scheduler/build/libs/* hadoop/share/hadoop/yarn/lib/
+
+# Copy the Myriad Executor
+cp -vf myriad/myriad-executor/build/libs/myriad-executor-0.2.0.jar
hadoop/share/hadoop/yarn/lib/
+
+# Hadoop 2.7.x:
+$ su - yarn
+Password: (vagrant is the password)
+[yarn@mesos-m1 ~]$ cd /opt/hadoop/
+[yarn@mesos-m1 hadoop]$ sbin/yarn-daemon.sh --config /opt/hadoop/etc/hadoop/
start resourcemanager
+[yarn@mesos-m1 hadoop]$ jps
+19283 ResourceManager
+19519 Jps
+```
+
+The YARN web interface can be accessed via the URL:
+
+http://100.0.10.101:8088
+
+The Myriad web interface can be accessd via the URL:
+
+http://100.0.10.101:8192
+
+# Happy Hacking
+
+At this point you have a local development environment ready for Myriad
+Framework hacking!
+
+
+
+
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/mesos/Vagrantfile
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/mesos/Vagrantfile
b/vagrant/libvirt/mesos/Vagrantfile
new file mode 100644
index 0000000..c5cc2af
--- /dev/null
+++ b/vagrant/libvirt/mesos/Vagrantfile
@@ -0,0 +1,92 @@
+# -*- mode: ruby -*-
+# vim: set ft=ruby ts=2 et :
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+VAGRANTFILE_API_VERSION = "2"
+
+# Tested with Vagrant version:
+Vagrant.require_version ">= 1.7.2"
+
+# Require YAML module
+require 'yaml'
+
+def fail_with_message(msg)
+ fail Vagrant::Errors::VagrantError.new, msg
+end
+
+def check_minimal_env()
+ if ENV['MESOS_ARCH'].nil? || ENV['MESOS_ARCH'].empty?
+ fail_with_message "You have to set MESOS_ARCH environment variable!"
+ end
+ if ENV['HADOOP_VERSION'].nil? || ENV['HADOOP_VERSION'].empty?
+ fail_with_message "You have to set HADOOP_VERSION environment
variable!"
+ end
+ if ENV['ZOOKEEPER_VERSION'].nil? || ENV['ZOOKEEPER_VERSION'].empty?
+ fail_with_message "You have to set ZOOKEEPER environment variable!"
+ end
+end
+
+# Sanity checks
+check_minimal_env()
+
+# Read YAML file with cluster details: The cluster flavor.
+config_file = 'config/cluster-' + ENV['MESOS_ARCH'] + ".yaml"
+if File.exists?(config_file)
+ cluster = YAML.load_file(config_file)
+else
+ fail_with_message "Cluster definition file: #{config_file} was not found!"
+end
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+ # Disabling the default /vagrant share
+ config.vm.synced_folder ".", "/vagrant", disabled: true
+ config.ssh.insert_key = false
+
+ i = 0
+ cluster.each do |servers|
+ # VM definition
+ config.vm.define servers["name"] do |node|
+ node.vm.box = servers["box"]
+ node.vm.hostname = servers["name"]
+ node.vm.network "private_network", ip: servers["ip"]
+ node.vm.provider :libvirt do |domain|
+ domain.uri = 'qemu+unix:///system'
+ domain.driver = 'kvm'
+ domain.memory = servers["mem"]
+ domain.cpus = servers['cpu']
+ if servers.key?('aditional_disk')
+ for disk in servers['aditional_disk']
+ domain.storage :file, :size => disk
+ end
+ end
+ end
+ i += 1
+ if i == cluster.length
+ node.vm.provision :ansible do |ansible|
+ ansible.verbose = "vv"
+ ansible.inventory_path = "provisioning/inventory.py"
+ ansible.playbook = "provisioning/site.yml"
+ ansible.limit = "all"
+ end # provison
+ end # if
+ end # config
+ end # cluster
+end # vagrant
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/mesos/ansible.cfg
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/mesos/ansible.cfg
b/vagrant/libvirt/mesos/ansible.cfg
new file mode 100644
index 0000000..acbdc2e
--- /dev/null
+++ b/vagrant/libvirt/mesos/ansible.cfg
@@ -0,0 +1,79 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+[defaults]
+# Disable host key checking. If a host is not initially in âknown_hostsâ
this
+# will result in prompting for confirmation of the key, which results in an
+# interactive experience
+host_key_checking = False
+
+# This is the default SSH timeout to use on connection attempts.
+timeout = 30
+
+# This options forces color mode even when running without a TTY.
+force_color = 1
+
+# Number of parallel processes to spawn when
+# communicating with remote hosts.
+forks = 10
+
+# Facts gathering: The value âsmartâ means each new
+# host that has no facts discovered will be scanned,
+# but if the same host is addressed in multiple plays
+# it will not be contacted again in the playbook run.
+# This enhance fact gathering time.
+gathering = smart
+
+# Ansible will log information about executions at the designated location
+log_path=.vagrant/ansible.log
+
+# The retry files save path is where Ansible will save .retry files when a
+# playbook fails. The directory will be created if it does not already exist.
+retry_files_save_path = .vagrant/retry-files
+
+#
+# Play execution strategies explained:
+#
+# Plays execution control for many hosts, controls how plays run their multiple
+# hosts:
+#
+# linear (default): All hosts of a play have to finish one task before any of
them can
+# begin the next one. Using the number of forks (default 5)
+# to parallelize.
+# serial: Ensures one group of hosts in a play finishes its work before
another group
+# can begin. This strategy will take a subset of hosts (the default is
+# five) and execute all tasks (in the linear fashion) against those
+# hosts before moving to the next subset and starting from the
beginning.
+# free: Allows each host to run until the end of the play as fast as it can.
+# Allows all hosts to run independently of each other. âfreeâ lets
each
+# host run independently, starting its next task as soon as it finishes
+# the previous one, regardless of how far other hosts have gotten.
+#
+# By default Ansible will attempt to run on all hosts from a play in parallel.
+#
+# With the serial strategy ansible will attempt to run on by defauult 5 of the
play.
+# It is designed so that each task will be run on all hosts before continuing
on
+# to the next task. So if you have 3 tasks it will ensure task 1 runs on all
+# your hosts first, then task 2 is run, then task 3 is run. This is the
"linear" strategy.
+#
+# Each play contains a list of tasks. Tasks are executed in order, one at a
+# time, against all machines matched by the host pattern of the play, before
moving on to
+# the next task. It is important to understand that, within a play, all hosts
+# are going to get the same task directives. It is the purpose of a play to map
+# a selection of hosts to tasks.
+
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/mesos/check-env.sh
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/mesos/check-env.sh
b/vagrant/libvirt/mesos/check-env.sh
new file mode 100644
index 0000000..c51ae75
--- /dev/null
+++ b/vagrant/libvirt/mesos/check-env.sh
@@ -0,0 +1,28 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+echo
+echo "This Vagrant environment is ready for the following settings:"
+echo
+
+[ -z $MESOS_ARCH ] || echo "- MESOS_ARCH: $MESOS_ARCH"
+[ -z $HADOOP_VERSION ] || echo "- HADOOP_VERSION: $HADOOP_VERSION"
+[ -z $ZOOKEEPER_VERSION ] || echo "- ZOOKEEPER_VERSION: $ZOOKEEPER_VERSION"
+
+echo
+echo "'vagrant up --provider=libvirt' and happy hacking!"
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/mesos/config/cluster-1m4a.yaml
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/mesos/config/cluster-1m4a.yaml
b/vagrant/libvirt/mesos/config/cluster-1m4a.yaml
new file mode 100644
index 0000000..3164e9f
--- /dev/null
+++ b/vagrant/libvirt/mesos/config/cluster-1m4a.yaml
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+- name: build
+ type: build
+ box: centos/7
+ ip: 100.0.10.100
+ mem: 4096
+ cpu: 2
+- name: mesos-m1
+ type: master
+ box: centos/7
+ ip: 100.0.10.101
+ mem: 2048
+ cpu: 2
+- name: mesos-a1
+ type: agent
+ box: centos/7
+ ip: 100.0.10.102
+ mem: 2048
+ cpu: 2
+- name: mesos-a2
+ type: agent
+ box: centos/7
+ ip: 100.0.10.103
+ mem: 2048
+ cpu: 2
+ aditional_disk:
+ - 50G
+ - 50G
+- name: mesos-a3
+ type: agent
+ box: centos/7
+ mem: 2048
+ ip: 100.0.10.104
+ aditional_disk:
+ - 50G
+ - 50G
+ cpu: 2
+- name: mesos-a4
+ type: agent
+ box: centos/7
+ mem: 2048
+ ip: 100.0.10.105
+ aditional_disk:
+ - 50G
+ - 50G
+ cpu: 2
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/mesos/config/cluster-3m4a.yaml
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/mesos/config/cluster-3m4a.yaml
b/vagrant/libvirt/mesos/config/cluster-3m4a.yaml
new file mode 100644
index 0000000..cb51758
--- /dev/null
+++ b/vagrant/libvirt/mesos/config/cluster-3m4a.yaml
@@ -0,0 +1,76 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+- name: build
+ type: build
+ box: centos/7
+ ip: 100.0.10.100
+ mem: 2048
+ cpu: 2
+- name: mesos-m1
+ type: master
+ box: centos/7
+ ip: 100.0.10.101
+ mem: 2048
+ cpu: 2
+- name: mesos-m2
+ type: master
+ box: centos/7
+ ip: 100.0.10.102
+ mem: 2048
+ cpu: 2
+- name: mesos-m3
+ type: master
+ box: centos/7
+ ip: 100.0.10.103
+ mem: 2048
+ cpu: 2
+- name: mesos-a1
+ type: agent
+ box: centos/7
+ ip: 100.0.10.104
+ mem: 2048
+ cpu: 2
+- name: mesos-a2
+ type: agent
+ box: centos/7
+ ip: 100.0.10.105
+ mem: 2048
+ cpu: 2
+ aditional_disk:
+ - 50G
+ - 50G
+- name: mesos-a3
+ type: agent
+ box: centos/7
+ ip: 100.0.10.106
+ mem: 1024
+ cpu: 2
+ aditional_disk:
+ - 50G
+ - 50G
+- name: mesos-a4
+ type: agent
+ box: centos/7
+ mem: 2048
+ ip: 100.0.10.107
+ aditional_disk:
+ - 50G
+ - 50G
+ cpu: 2
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/mesos/docs/ansible.txt
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/mesos/docs/ansible.txt
b/vagrant/libvirt/mesos/docs/ansible.txt
new file mode 100644
index 0000000..d2c1d5b
--- /dev/null
+++ b/vagrant/libvirt/mesos/docs/ansible.txt
@@ -0,0 +1 @@
+ansible-playbook --private-key $HOME/.vagrant.d/insecure_private_key -i
provisioning/inventory.py provisioning/site.yml --tags hadoop
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/mesos/docs/build-mesos.md
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/mesos/docs/build-mesos.md
b/vagrant/libvirt/mesos/docs/build-mesos.md
new file mode 100644
index 0000000..99981f4
--- /dev/null
+++ b/vagrant/libvirt/mesos/docs/build-mesos.md
@@ -0,0 +1,135 @@
+# Building Apache Mesos
+
+Note: Last tested Mesos version for Myriad -> 0.28.1
+
+Notes about build Apache Mesos and some development hacking notes.
+
+# Building with autotools system
+
+```
+# Change working directory.
+$ cd mesos
+
+# Bootstrap (Only required if building from git repository).
+$ ./bootstrap
+
+# Configure and build.
+$ mkdir build
+$ cd build
+$ ../configure
+$ make
+
+# Run test suite.
+$ make check
+
+# Install (Optional).
+$ make install
+```
+
+# Building with new CMake system
+
+```
+curl -O https://cmake.org/files/v3.9/cmake-3.9.4-Linux-x86_64.tar.gz
+tar xvzf cmake-3.9.4-Linux-x86_64.tar.gz
+export PATH=$PATH:$HOME/cmake-3.9.4-Linux-x86_64/bin
+
+cd mesos
+mkdir build && cd build
+cmake ..
+cmake --build .
+
+
+For cleaning generated build system files:
+
+rm build/CMakeCache.txt
+```
+
+# Running Apache Mesos
+
+## Master
+
+```
+cd /home/vagrant/mesos.git/build
+sudo ./bin/mesos-master.sh --ip=100.0.10.101 --work_dir=/var/lib/mesos
+```
+
+## Agents
+
+```
+cd /home/vagrant/mesos.git/build
+sudo ./bin/mesos-agent.sh --master=100.0.10.101:5050 --work_dir=/var/lib/mesos
+```
+
+## Running with 3rdparty shipped Zookeeper
+
+At master:
+
+```
+cd /home/vagrant/mesos.git/build/3rdparty/zookeeper-3.4.8
+cp conf/zoo_sample.cfg conf/zoo.cfg
+echo "server.1=mesos-m1:2888:3888" >> conf/zoo.cfg
+bin/zkServer.sh start
+echo ruok | nc 127.0.0.1 2181
+
+sudo ./bin/mesos-master.sh --ip=100.0.10.101 --work_dir=/var/lib/mesos
--zk=zk://mesos-m1:2181/mesos --quorum=1
+```
+
+At agents:
+
+```
+sudo ./bin/mesos-agent.sh --master=zk://mesos-m1:2181/mesos
--work_dir=/var/lib/mesos
+```
+
+# Debugging
+
+```
+gdbserver -> exec ${LIBTOOL} --mode=execute gdbserver 100.0.10.101:1234 \
+ /home/vagrant/mesos.git/build/src/mesos-master "${@}"
+
+
+(gdb) target remote mesos-m1:1234
+(gdb) continue
+(gdb) monitor exit
+```
+
+# Development Environment with Vim
+
+
+## Building clang-format
+
+```
+svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
+cd llvm/tools
+svn co http://llvm.org/svn/llvm-project/cfe/trunk clang
+cd ../..
+mkdir build
+cd build
+cmake -G "Unix Makefiles" ../llvm
+```
+
+## Code navigation: cscope and tags
+
+```
+sudo yum install cscope ctags -y
+
+cd build
+make cscope
+make ctags
+```
+
+## Vimrc with clang-format for C++11 and Google Style
+
+```
+cd
+curl -O
https://raw.githubusercontent.com/javiroman/reproducible-research/master/vim-tmux/vimrc
+mv vimrc .vimrc
+mkdir -p .vim/bundle
+git clone https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim
+vim
+:PluginInstall
+or from command line: vim +PluginInstall +qall
+```
+
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-myriad/blob/1c3857a8/vagrant/libvirt/mesos/docs/notes-myriad.txt
----------------------------------------------------------------------
diff --git a/vagrant/libvirt/mesos/docs/notes-myriad.txt
b/vagrant/libvirt/mesos/docs/notes-myriad.txt
new file mode 100644
index 0000000..1dd51d5
--- /dev/null
+++ b/vagrant/libvirt/mesos/docs/notes-myriad.txt
@@ -0,0 +1,11 @@
+# Copy the Myriad Scheduler
+cp -vf myriad/myriad-scheduler/build/libs/* hadoop/share/hadoop/yarn/lib/
+
+# Copy the Myriad Executor
+cp -vf myriad/myriad-executor/build/libs/myriad-executor-0.2.0.jar
hadoop/share/hadoop/yarn/lib/
+
+# Copy the myriad-config-default.yml
+cp myriad/myriad-scheduler/build/resources/main/myriad-config-default.yml
hadoop/etc/hadoop/
+
+#Hadoop 2.7.x:
+#sbin/yarn-daemon.sh --config /opt/hadoop/etc/hadoop/ start resourcemanager
