csantanapr closed pull request #3105: auto detect docker-for-mac
URL: https://github.com/apache/incubator-openwhisk/pull/3105
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.gitignore b/.gitignore
index f65f024838..bfb7f48f58 100644
--- a/.gitignore
+++ b/.gitignore
@@ -54,6 +54,7 @@ out/
# Ansible
ansible/environments/docker-machine/hosts
+ansible/environments/local/hosts
ansible/db_local.ini*
ansible/tmp/*
ansible/roles/nginx/files/openwhisk-client*
diff --git a/ansible/README.md b/ansible/README.md
index 8bb9596608..95bbcf9b9b 100644
--- a/ansible/README.md
+++ b/ansible/README.md
@@ -73,20 +73,32 @@ In all instructions, replace `<openwhisk_home>` with the
base directory of your
#### Setup
-This step needs to be done only once per development environment. It will
generate configuration files based on your local settings. Notice that for the
following playbook you don't need to specify a target environment as it will
run only local actions.
-After the playbook is done you should see a file called `db_local.ini` in your
`ansible` directory. It will by default contain settings for a local ephemeral
CouchDB setup. Afterwards, you can change the values directly in `db_local.ini`.
-
-##### Ephemeral CouchDB
-
-If you want to use the ephemeral CouchDB, run this command
+The following step must be executed once per development environment.
+It will generate the `hosts` configuration file based on your environment
settings.
```
ansible-playbook -i environments/<environment> setup.yml
```
-##### Persistent CouchDB
+The default configuration does not run multiple instances of core components
(e.g., controller, invoker, kafka).
+You may elect to enable high-availability (HA) mode by passing tne ansible
option `-e mode=HA` when executing this playbook.
+This will configure your deployment with multiple instances (e.g., two kafka
instancess, and two invokers).
+
+In addition to the host file generation, you need to configure the database
for your deployment. This is done
+by creating a file `ansible/db_local.ini` to provide the following properties.
+
+```bash
+[db_creds]
+db_provider=
+db_username=
+db_password=
+db_protocol=
+db_host=
+db_port=
+```
-If you want to use the persistent CouchDB instead, you can use env variables
that are read by the playbook:
+This file is generated automatically if you are using an ephermeral CouchDB
instance. Otherwise, you must create it explicitly.
+For convenience, you can use shell environment variables that are read by the
playbook to generate the required `db_local.ini` file as shown below.
```
export OW_DB=CouchDB
@@ -96,14 +108,10 @@ export OW_DB_PROTOCOL=<your couchdb protocol>
export OW_DB_HOST=<your couchdb host>
export OW_DB_PORT=<your couchdb port>
-ansible-playbook -i environments/<environment> setup.yml
+ansible-playbook -i environments/<environment> couchdb.yml --tags ini
```
-If you deploy CouchDB manually (i.e., without using the deploy CouchDB
playbook), you must set the `reduce_limit` property on views to `false`. This
may be done via the REST API, as in: `curl -X PUT
${OW_DB_PROTOCOL}://${OW_DB_HOST}:${OW_DB_PORT}/_config/query_server_config/reduce_limit
-d '"false"' -u ${OW_DB_USERNAME}:${OW_DB_PASSWORD}`.
-
-##### Cloudant
-
-If you want to use Cloudant instead, you can use env variables that are read
by the playbook:
+Alternatively, if you want to use Cloudant as your datastore:
```
export OW_DB=Cloudant
@@ -113,7 +121,7 @@ export OW_DB_PROTOCOL=https
export OW_DB_HOST=<your cloudant user>.cloudant.com
export OW_DB_PORT=443
-ansible-playbook -i environments/<environment> setup.yml
+ansible-playbook -i environments/<environment> couchdb.yml --tags ini
```
#### Install Prerequisites
@@ -129,6 +137,8 @@ ansible-playbook -i environments/<environment> prereq.yml
### Deploying Using CouchDB
- Make sure your `db_local.ini` file is set up for CouchDB. See [Setup](#setup)
+- If you deploy CouchDB manually (i.e., without using the deploy CouchDB
playbook), you must set the `reduce_limit` property on views to `false`.
+This may be done via the REST API, as in: `curl -X PUT
${OW_DB_PROTOCOL}://${OW_DB_HOST}:${OW_DB_PORT}/_config/query_server_config/reduce_limit
-d '"false"' -u ${OW_DB_USERNAME}:${OW_DB_PASSWORD}`.
- Then execute
```
@@ -143,7 +153,7 @@ ansible-playbook -i environments/<environment> openwhisk.yml
ansible-playbook -i environments/<environment> postdeploy.yml
```
-You need to run `initdb.yml` on couchdb **every time** you do a fresh deploy
CouchDB to initialize the subjects database.
+You need to run `initdb.yml` **every time** you do a fresh deploy CouchDB to
initialize the subjects database.
The playbooks `wipe.yml` and `postdeploy.yml` should be run on a fresh
deployment only, otherwise all transient
data that include actions and activations are lost.
diff --git a/ansible/couchdb.yml b/ansible/couchdb.yml
index 70faba99db..b75cd099cd 100644
--- a/ansible/couchdb.yml
+++ b/ansible/couchdb.yml
@@ -1,6 +1,18 @@
---
# This playbook deploys a CouchDB for Openwhisk.
+- hosts: localhost
+ tasks:
+ - name: check if db_local.ini exists?
+ tags: ini
+ stat: path="{{ playbook_dir }}/db_local.ini"
+ register: db
+
+ - name: prepare db_local.ini
+ tags: ini
+ local_action: template src="db_local.ini.j2" dest="{{ playbook_dir
}}/db_local.ini"
+ when: not db.stat.exists
+
- hosts: db
roles:
- couchdb
\ No newline at end of file
diff --git a/ansible/environments/docker-machine/group_vars/all
b/ansible/environments/docker-machine/group_vars/all
index 391fa1e1e3..abffd2fe3c 100644
--- a/ansible/environments/docker-machine/group_vars/all
+++ b/ansible/environments/docker-machine/group_vars/all
@@ -30,3 +30,12 @@ controller_arguments: '-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxre
invoker_arguments: "{{ controller_arguments }}"
invoker_allow_multiple_instances: true
+
+# Set kafka configuration
+kafka_heap: '512m'
+kafka_topics_completed_retentionBytes: 104857600
+kafka_topics_completed_retentionMS: 300000
+kafka_topics_health_retentionBytes: 104857600
+kafka_topics_health_retentionMS: 300000
+kafka_topics_invoker_retentionBytes: 104857600
+kafka_topics_invoker_retentionMS: 300000
diff --git a/ansible/environments/docker-machine/hosts.j2.ini
b/ansible/environments/docker-machine/hosts.j2.ini
index a4990ae359..1a49698ced 100644
--- a/ansible/environments/docker-machine/hosts.j2.ini
+++ b/ansible/environments/docker-machine/hosts.j2.ini
@@ -8,17 +8,24 @@ ansible ansible_connection=local
[controllers]
controller0 ansible_host={{ docker_machine_ip }}
-controller1 ansible_host={{ docker_machine_ip }}
+;{% if mode is defined and 'HA' in mode %}
+;controller1 ansible_host={{ docker_machine_ip }}
+;{% endif %}
[kafkas]
-{{ docker_machine_ip }} ansible_host={{ docker_machine_ip }}
+kafka0 ansible_host={{ docker_machine_ip }}
+{% if mode is defined and 'HA' in mode %}
+kafka1 ansible_host={{ docker_machine_ip }}
+{% endif %}
[zookeepers:children]
kafkas
[invokers]
invoker0 ansible_host={{ docker_machine_ip }}
+{% if mode is defined and 'HA' in mode %}
invoker1 ansible_host={{ docker_machine_ip }}
+{% endif %}
[db]
{{ docker_machine_ip }} ansible_host={{ docker_machine_ip }}
diff --git a/ansible/environments/local/hosts
b/ansible/environments/local/hosts.j2.ini
similarity index 79%
rename from ansible/environments/local/hosts
rename to ansible/environments/local/hosts.j2.ini
index 4cf54aa811..a687adf413 100644
--- a/ansible/environments/local/hosts
+++ b/ansible/environments/local/hosts.j2.ini
@@ -8,17 +8,24 @@ ansible ansible_connection=local
[controllers]
controller0 ansible_host=172.17.0.1 ansible_connection=local
+;{% if mode is defined and 'HA' in mode %}
+;controller1 ansible_host=172.17.0.1 ansible_connection=local
+;{% endif %}
[kafkas]
kafka0 ansible_host=172.17.0.1 ansible_connection=local
+{% if mode is defined and 'HA' in mode %}
kafka1 ansible_host=172.17.0.1 ansible_connection=local
+{% endif %}
[zookeepers:children]
kafkas
[invokers]
invoker0 ansible_host=172.17.0.1 ansible_connection=local
+{% if mode is defined and 'HA' in mode %}
invoker1 ansible_host=172.17.0.1 ansible_connection=local
+{% endif %}
; db group is only used if db_provider is CouchDB
[db]
diff --git a/ansible/setup.yml b/ansible/setup.yml
index 28d02c50bc..86904920e9 100644
--- a/ansible/setup.yml
+++ b/ansible/setup.yml
@@ -3,6 +3,10 @@
- hosts: localhost
tasks:
+ - name: gen hosts if 'local' env is used
+ local_action: template
src="{{playbook_dir}}/environments/local/hosts.j2.ini" dest="{{ playbook_dir
}}/environments/local/hosts"
+ when: "'environments/local' in inventory_dir"
+
- name: find the ip of docker-machine
local_action: shell "docker-machine" "ip" "{{docker_machine_name |
default('whisk')}}"
register: result
@@ -17,19 +21,6 @@
local_action: template
src="{{playbook_dir}}/environments/docker-machine/hosts.j2.ini" dest="{{
playbook_dir }}/environments/docker-machine/hosts"
when: "'environments/docker-machine' in inventory_dir"
- # this step is needed to generate db_local.ini which is required by later
steps
- - name: add new db host on-the-fly
- add_host: name={{ docker_machine_ip }} groups=db
- when: "'environments/docker-machine' in inventory_dir"
-
- - name: check if db_local.ini exists?
- stat: path="{{ playbook_dir }}/db_local.ini"
- register: db
-
- - name: prepare db_local.ini
- local_action: template src="db_local.ini.j2" dest="{{ playbook_dir
}}/db_local.ini"
- when: not db.stat.exists
-
- name: gen untrusted server certificate for host
local_action: shell "{{ playbook_dir }}/roles/nginx/files/genssl.sh" "*.{{
whisk_api_localhost_name | default(whisk_api_host_name) |
default(whisk_api_localhost_name_default) }}" "server"
when: nginx.ssl.cert == "openwhisk-server-cert.pem"
diff --git a/tools/build/redo b/tools/build/redo
index 8b877b14aa..b152c24e34 100755
--- a/tools/build/redo
+++ b/tools/build/redo
@@ -75,7 +75,12 @@ def getArgs():
if osname == 'Linux':
return 'local'
elif osname == 'Darwin':
- return 'docker-machine'
+ if os.getenv('DOCKER_HOST', None) is not None:
+ # docker-machine typically has docker host set in the
environment
+ return 'docker-machine'
+ else:
+ # otherwise assume docker-for-mac
+ return 'local'
else:
return None
@@ -83,7 +88,7 @@ def getArgs():
parser.add_argument('-b', '--build', help='build component',
action='store_const', const=True, default=False)
parser.add_argument('-x', '--teardown', help='teardown component',
action='store_const', const=True, default=False)
parser.add_argument('-d', '--deploy', help='deploy component',
action='store_const', const=True, default=False)
- parser.add_argument('-t', '--target', help='deploy target (one of [mac,
docker-machine, local])', default=detectDeployTarget())
+ parser.add_argument('-t', '--target', help='deploy target (one of
[docker-machine, local])', default=detectDeployTarget())
parser.add_argument('-y', '--yaml', help='deploy target using inferred
YAML file if component is not one of known targets', action='store_const',
const=True, default=False)
parser.add_argument('-g', '--gradle', help='use target using inferred
gradle file if component is not one of known targets', action='store_const',
const=True, default=False)
parser.add_argument('-n', '--just-print', help='prints the component
configuration but does not run any targets', action='store_const', const=True,
default=False, dest='skiprun')
diff --git a/tools/travis/build.sh b/tools/travis/build.sh
index 043c1cc4d6..4870acb72a 100755
--- a/tools/travis/build.sh
+++ b/tools/travis/build.sh
@@ -23,7 +23,7 @@ cd $ROOTDIR/ansible
ANSIBLE_CMD="ansible-playbook -i environments/local -e
docker_image_prefix=testing"
GRADLE_PROJS_SKIP="-x :actionRuntimes:pythonAction:distDocker -x
:actionRuntimes:python2Action:distDocker -x
actionRuntimes:swift3.1.1Action:distDocker -x
:actionRuntimes:javaAction:distDocker"
-$ANSIBLE_CMD setup.yml
+$ANSIBLE_CMD setup.yml -e mode=HA
$ANSIBLE_CMD prereq.yml
$ANSIBLE_CMD couchdb.yml
$ANSIBLE_CMD initdb.yml
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
