This is an automated email from the ASF dual-hosted git repository.
dgrove pushed a commit to branch master
in repository
https://gitbox.apache.org/repos/asf/incubator-openwhisk-deploy-kube.git
The following commit(s) were added to refs/heads/master by this push:
new 8363056 DockerContainerFactory DNS configuration (#392)
8363056 is described below
commit 8363056019a365fc3f976a47c3f97cd8a82cd4eb
Author: David Grove <[email protected]>
AuthorDate: Wed Feb 13 16:08:13 2019 -0500
DockerContainerFactory DNS configuration (#392)
By default, automatically configure the Invoker so that
user action containers created by the DockerContainerFactory
will have the same DNS configuration as the Invoker itself.
This (a) enables all event providers to be enabled by default
and (b) provides a more Kubernetes-native experience since the
user action containers can access other Kubernetes services
running on the cluster via the usual Kubernetes DNS mechanisms.
---
README.md | 12 +---
docs/configurationChoices.md | 82 +++++-----------------
.../configMapFiles/configureDNS/configureDNS.sh | 24 +++++++
helm/openwhisk/templates/_invoker-helpers.tpl | 6 +-
helm/openwhisk/templates/invoker-agent-pod.yaml | 4 +-
helm/openwhisk/templates/invoker-cm.yaml | 3 +-
helm/openwhisk/templates/invoker-pod.yaml | 35 +++++++--
helm/openwhisk/values-metadata.yaml | 50 +++++++++----
helm/openwhisk/values.yaml | 19 +++--
tools/travis/deploy-chart.sh | 8 ---
10 files changed, 129 insertions(+), 114 deletions(-)
diff --git a/README.md b/README.md
index ddb2d12..9407491 100644
--- a/README.md
+++ b/README.md
@@ -142,16 +142,6 @@ deploy.
4. [Configure the `wsk` CLI](#configure-the-wsk-cli). You need to
tell the `wsk` CLI how to connect to your OpenWhisk deployment.
-Although you are deploying OpenWhisk to Kubernetes, by default the
-executing actions will not be aware they are running on a Kubernetes
-cluster. In particular, they will not be configured to do DNS resolution
-of Kubernetes services. However, if you want executing actions to
-easily access services in a Kubernetes-native way, you can configure
-your OpenWhisk deployment to enable that by either using the
-[KubernetesContainerFactory](docs/configurationChoices.md#invoker-container-factory)
-or setting the value of `invoker.DNS` when you create the `mycluster.yaml`
-to customize your deployment ([see DNS
options](docs/configurationChoices.md#user-action-container-dns)).
-
## Initial setup
Indicate the Kubernetes worker nodes that should be used to execute
@@ -280,7 +270,7 @@ $ kubectl -n openwhisk -ti exec owdev-wskadmin -- wskadmin
user list guest
Check
[here](https://github.com/apache/incubator-openwhisk/tree/master/tools/admin)
for details about the available commands.
-# Development and Testing
+# Development and Testing OpenWhisk on Kubernetes
This section outlines how common OpenWhisk development tasks are
supported when OpenWhisk is deployed on Kubernetes using Helm.
diff --git a/docs/configurationChoices.md b/docs/configurationChoices.md
index 1723171..7b3ab2b 100644
--- a/docs/configurationChoices.md
+++ b/docs/configurationChoices.md
@@ -22,54 +22,6 @@ supported by the OpenWhisk Helm chart. In general, you
customize your
deployment by adding stanzas to `mycluster.yaml` that override default
values in the `helm/openwhisk/values.yaml` file.
-### Deploying Event Providers
-
-OpenWhisk has several standard Event Providers that can be optionally enabled.
-The OpenWhisk Helm Chart currently includes optional support for
-deploying the alarm, cloudant, and kafka providers.
-To deploy a provider, you must add a stanza to your `mycluster.yaml` that
enables it,
-for example:
-```yaml
-providers:
- alarm:
- enabled: true
-```
-
-The deployment of the event providers is not enabled by default because they
-are not fully functional with OpenWhisk's default
-`DockerContainerFactory` without additional configuration (the issue is that
-user action containers created by the DockerContainerFactory are not
configured to
-themselves be able to invoke Kubernetes services). To work around this you
must do one
-of the following three alternatives:
-1. Deploy a CouchDB instance external to your Kubernetes cluster and configure
the event
-providers to use it by adding a stanza like the following to your
`mycluster.yaml`:
-```yaml
-providers:
- db:
- external: true
- host: "0.0.0.0"
- port: 5984
- protocol: "http"
- username: "admin"
- password: "secret"
-```
-2. Configure the DNS nameservers for the user containers created by
DockerContainerFactory to
-use Kubernetes's DNS service. For example, if your cluster uses kube-dns,
then first
-get the IP address of Kubernetes DNS server by `echo $(kubectl get svc
kube-dns -n kube-system -o 'jsonpath={.spec.clusterIP}')`
-and then add below stanza to your `mycluster.yaml`:
-```yaml
-invoker:
- containerFactory:
- nameservers: "<IP_Address_Of_Kube_DNS>"
-```
-3. Use the lower performance `KubernetesContainerFactory` by adding the
following stanza
-to your `mycluster.yaml`
-```yaml
-invoker:
- containerFactory:
- impl: "kubernetes"
-```
-
### Replication factor
By default the OpenWhisk Helm Chart will deploy a single replica of each
@@ -157,6 +109,19 @@ k8s:
enabled: false
```
+### Selectively Deploying Event Providers
+
+The default settings of the Helm chart will deploy OpenWhisk's alarm,
+cloudant, and kafka event providers. If you want to disable the
+deployment of one or more event providers, you can add
+a stanza to your `mycluster.yaml` for example:
+```yaml
+providers:
+ alarm:
+ enabled: false
+```
+will disable the deployment of the alarm provider.
+
### Invoker Container Factory
The Invoker is responsible for creating and managing the containers
@@ -236,18 +201,9 @@ to your `mycluster.yaml`
### User action container DNS
-If you are using the DockerContainerFactory, by default your user actions will
-not be able to connect to other Kubernetes services running in your cluster.
-To enable a more Kubernetes-native variant of the DockerContainerFactory, you
-need to configure the DNS nameservers for the user containers to use
Kubernetes's
-DNS service. Currently this requires you to discover the InternalIP
-used for the DNS service and record this numeric ip address in `values.yaml`.
-
-For example, if your cluster uses kube-dns, then first
-get the IP address of Kubernetes DNS server by `echo $(kubectl get svc
kube-dns -n kube-system -o 'jsonpath={.spec.clusterIP}')`
-and then add below stanza to your `mycluster.yaml`:
-```yaml
-invoker:
- containerFactory:
- nameservers: "<IP_Address_Of_Kube_DNS>"
-```
+By default, your user actions containers will be configured to use the same
+DNS nameservers, search path, and options as the Invoker pod that spawned them.
+If you want to override this default when using the DockerContainerFactory,
+you can set `invoker.containerFactory.networkConfig.dns.inheritInvokerConfig`
to `false`
+and explicitly configure the child values of
`invoker.containerFactory.networkConfig.dns.overrides`
+instead.
diff --git a/helm/openwhisk/configMapFiles/configureDNS/configureDNS.sh
b/helm/openwhisk/configMapFiles/configureDNS/configureDNS.sh
new file mode 100644
index 0000000..0f23866
--- /dev/null
+++ b/helm/openwhisk/configMapFiles/configureDNS/configureDNS.sh
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
contributor
+# license agreements; and to You under the Apache License, Version 2.0.
+
+exportVars() {
+ vname=$1
+ vals=$2
+
+ let "idx=0"
+ for v in $vals; do
+ export $(echo $vname$idx)=$v
+ let "idx=idx+1"
+ done
+}
+
+echo "The contents of /etc/resolv.conf are:"
+cat /etc/resolv.conf
+
+nameservers=$(grep -e ^nameserver /etc/resolv.conf | sed 's/nameserver //')
+search=$(grep -e ^search /etc/resolv.conf | sed 's/search //')
+options=$(grep -e ^option /etc/resolv.conf | sed 's/option //')
+
+exportVars "CONFIG_whisk_containerFactory_containerArgs_dnsServers_"
"$nameservers"
+exportVars "CONFIG_whisk_containerFactory_containerArgs_dnsSearch_" "$search"
+exportVars "CONFIG_whisk_containerFactory_containerArgs_dnsOptions_" "$options"
diff --git a/helm/openwhisk/templates/_invoker-helpers.tpl
b/helm/openwhisk/templates/_invoker-helpers.tpl
index 3547f62..5e5ba7d 100644
--- a/helm/openwhisk/templates/_invoker-helpers.tpl
+++ b/helm/openwhisk/templates/_invoker-helpers.tpl
@@ -35,12 +35,12 @@
- name: docker-pull-runtimes
imagePullPolicy: {{ .Values.utility.imagePullPolicy | quote }}
image: "{{- .Values.utility.imageName -}}:{{- .Values.utility.imageTag -}}"
- command: ["/usr/local/bin/ansible-playbook", "/task/playbook.yml"]
+ command: ["/usr/local/bin/ansible-playbook", "/invoker-scripts/playbook.yml"]
volumeMounts:
- name: dockersock
mountPath: "/var/run/docker.sock"
- - name: task-dir
- mountPath: "/task/playbook.yml"
+ - name: scripts-dir
+ mountPath: "/invoker-scripts/playbook.yml"
subPath: "playbook.yml"
env:
# action runtimes
diff --git a/helm/openwhisk/templates/invoker-agent-pod.yaml
b/helm/openwhisk/templates/invoker-agent-pod.yaml
index c24d544..c6ea081 100644
--- a/helm/openwhisk/templates/invoker-agent-pod.yaml
+++ b/helm/openwhisk/templates/invoker-agent-pod.yaml
@@ -29,9 +29,9 @@ spec:
{{ include "openwhisk.docker_volumes" . | indent 6 }}
- name: userlogs
emptyDir: {}
- - name: task-dir
+ - name: scripts-dir
configMap:
- name: {{ .Release.Name }}-docker-pull-runtimes
+ name: {{ .Release.Name }}-invoker-scripts
initContainers:
# Pull images for all default runtimes before starting invoker
diff --git a/helm/openwhisk/templates/invoker-cm.yaml
b/helm/openwhisk/templates/invoker-cm.yaml
index 078db80..090a194 100644
--- a/helm/openwhisk/templates/invoker-cm.yaml
+++ b/helm/openwhisk/templates/invoker-cm.yaml
@@ -4,8 +4,9 @@
apiVersion: v1
kind: ConfigMap
metadata:
- name: {{ .Release.Name }}-docker-pull-runtimes
+ name: {{ .Release.Name }}-invoker-scripts
labels:
{{ include "openwhisk.label_boilerplate" .| indent 4 }}
data:
{{ (.Files.Glob "configMapFiles/dockerPullRuntimes/playbook.yml").AsConfig |
indent 2 }}
+{{ (.Files.Glob "configMapFiles/configureDNS/configureDNS.sh").AsConfig |
indent 2 }}
diff --git a/helm/openwhisk/templates/invoker-pod.yaml
b/helm/openwhisk/templates/invoker-pod.yaml
index 4d47e75..1247ec8 100644
--- a/helm/openwhisk/templates/invoker-pod.yaml
+++ b/helm/openwhisk/templates/invoker-pod.yaml
@@ -41,9 +41,9 @@ spec:
{{- if eq .Values.invoker.containerFactory.impl "docker" }}
volumes:
{{ include "openwhisk.docker_volumes" . | indent 6 }}
- - name: task-dir
+ - name: scripts-dir
configMap:
- name: {{ .Release.Name }}-docker-pull-runtimes
+ name: {{ .Release.Name }}-invoker-scripts
{{- end }}
initContainers:
@@ -58,7 +58,11 @@ spec:
- name: invoker
image: "{{- .Values.invoker.imageName -}}:{{- .Values.invoker.imageTag
-}}"
imagePullPolicy: {{ .Values.invoker.imagePullPolicy | quote }}
+{{- if and (eq .Values.invoker.containerFactory.impl "docker")
.Values.invoker.containerFactory.networkConfig.dns.inheritInvokerConfig }}
+ command: [ "/bin/bash", "-c", ". /invoker-scripts/configureDNS.sh &&
/init.sh --uniqueName $INVOKER_NAME" ]
+{{- else }}
command: [ "/bin/bash", "-c", "/init.sh --uniqueName $INVOKER_NAME" ]
+{{- end }}
env:
- name: "PORT"
value: {{ .Values.invoker.port | quote }}
@@ -86,15 +90,31 @@ spec:
- name: "CONFIG_whisk_containerPool_userMemory"
value: {{ .Values.whisk.containerPool.userMemory | quote }}
-{{- if ne .Values.invoker.containerFactory.nameservers "" }}
+{{- if not
.Values.invoker.containerFactory.networkConfig.dns.inheritInvokerConfig }}
+{{- if ne
.Values.invoker.containerFactory.networkConfig.dns.overrides.nameservers "" }}
# DNS Server(s) to be used by action containers
- {{- range $index, $element := splitList ":"
.Values.invoker.containerFactory.nameservers }}
+ {{- range $index, $element := splitList " "
.Values.invoker.containerFactory.networkConfig.dns.overrides.nameservers }}
- name: "CONFIG_whisk_containerFactory_containerArgs_dnsServers_{{
$index }}"
value: {{ $element | quote }}
{{- end }}
{{- end }}
+{{- if ne .Values.invoker.containerFactory.networkConfig.dns.overrides.search
"" }}
+ # DNS search path to be used by action containers
+ {{- range $index, $element := splitList " "
.Values.invoker.containerFactory.networkConfig.dns.overrides.search }}
+ - name: "CONFIG_whisk_containerFactory_containerArgs_dnsSearch_{{
$index }}"
+ value: {{ $element | quote }}
+ {{- end }}
+{{- end }}
+{{- if ne .Values.invoker.containerFactory.networkConfig.dns.overrides.options
"" }}
+ # DNS options to be used by action containers
+ {{- range $index, $element := splitList " "
.Values.invoker.containerFactory.networkConfig.dns.overrides.options }}
+ - name: "CONFIG_whisk_containerFactory_containerArgs_dnsOptions_{{
$index }}"
+ value: {{ $element | quote }}
+ {{- end }}
+{{- end }}
+{{- end }}
- name: "CONFIG_whisk_docker_containerFactory_containerArgs_network"
- value: {{ .Values.invoker.containerFactory.containerNetwork |
quote }}
+ value: {{ .Values.invoker.containerFactory.networkConfig.name |
quote }}
# Invoker name is the name of the node (DaemonSet) or pod
(StatefulSet)
- name: "INVOKER_NAME"
@@ -153,4 +173,9 @@ spec:
{{- if eq .Values.invoker.containerFactory.impl "docker" }}
volumeMounts:
{{ include "openwhisk.docker_volume_mounts" . | indent 8 }}
+{{- if .Values.invoker.containerFactory.networkConfig.dns.inheritInvokerConfig
}}
+ - name: scripts-dir
+ mountPath: "/invoker-scripts/configureDNS.sh"
+ subPath: "configureDNS.sh"
+{{- end }}
{{- end }}
diff --git a/helm/openwhisk/values-metadata.yaml
b/helm/openwhisk/values-metadata.yaml
index 54df8bf..791e417 100644
--- a/helm/openwhisk/values-metadata.yaml
+++ b/helm/openwhisk/values-metadata.yaml
@@ -436,8 +436,8 @@ k8s:
required: true
dns:
__metadata:
- label: "DNS nameserver"
- description: "The Internal IP of your clusters DNS service"
+ label: "Kubernetes DNS service"
+ description: "The name of your cluster's DNS service"
type: "string"
required: true
persistence:
@@ -1022,6 +1022,39 @@ invoker:
value: "docker"
- label: "kubernetes"
value: "kubernetes"
+ networkConfig:
+ name:
+ __metadata:
+ label: "Container Network Name"
+ description: "Name of the Docker network to be used by user action
containers"
+ type: "string"
+ required: false
+ dns:
+ inheritInvokerConfig:
+ __metadata:
+ label: "Inherit Invoker DNS Configuration"
+ description: "Should the user action containers use the same DNS
configuration as the Invoker's container?"
+ type: "boolean"
+ required: false
+ overrides:
+ nameservers:
+ __metadata:
+ label: "DNS nameservers"
+ description: "A list of IP addresses of DNS nameserver(s) to be
used by user action containers"
+ type: "string"
+ required: false
+ search:
+ __metadata:
+ label: "DNS search domains"
+ description: "A list of DNS search domains to be used by user
action containers"
+ type: "string"
+ required: false
+ options:
+ __metadata:
+ label: "options"
+ description: "A list of DNS options to be used by user action
containers"
+ type: "string"
+ required: false
kubernetes:
replicaCount:
__metadata:
@@ -1067,19 +1100,6 @@ invoker:
description: "The primary invoker agent port"
type: "string"
required: true
- containerNetwork:
- __metadata:
- label: "Container Network"
- description: "Docker network to be used by user action containers"
- type: "string"
- required: false
- nameservers:
- __metadata:
- label: "Nameservers"
- description: "The DNS nameserver to be used by user action containers"
- type: "string"
- required: false
-
apigw:
__metadata:
diff --git a/helm/openwhisk/values.yaml b/helm/openwhisk/values.yaml
index 6aced76..0baa2f1 100644
--- a/helm/openwhisk/values.yaml
+++ b/helm/openwhisk/values.yaml
@@ -245,6 +245,16 @@ invoker:
dind: false
useRunc: false
impl: "docker"
+ networkConfig:
+ name: "bridge"
+ dns:
+ inheritInvokerConfig: true
+ overrides: # NOTE: if inheritInvokerConfig is true, all
overrides are ignored
+ # Nameservers, search, and options are space-separated lists
+ # eg nameservers: "1.2.3.4 1.2.3.5 1.2.3.6" is a list of 3
nameservers
+ nameservers: ""
+ search: ""
+ options: ""
kubernetes:
replicaCount: 1
agent:
@@ -253,9 +263,6 @@ invoker:
imagePullPolicy: "IfNotPresent"
enabled: false
port: 3233
- containerNetwork: "bridge"
- # nameservers is an empty string or a colon separated list of IP addresses
of DNS servers.
- nameservers: ""
# API Gateway configurations
apigw:
@@ -293,7 +300,7 @@ providers:
password: "secret"
# Alarm provider configurations
alarm:
- enabled: false
+ enabled: true
imageName: "openwhisk/alarmprovider"
imageTag: "1.12.2"
imagePullPolicy: "IfNotPresent"
@@ -306,7 +313,7 @@ providers:
size: 1Gi
# Cloudant provider configurations
cloudant:
- enabled: false
+ enabled: true
imageName: "openwhisk/cloudantprovider"
imageTag: "1.9.2"
imagePullPolicy: "IfNotPresent"
@@ -319,7 +326,7 @@ providers:
size: 1Gi
# Kafka provider configurations
kafka:
- enabled: false
+ enabled: true
imageName: "openwhisk/kafkaprovider"
imageTag: "1.4.20"
imagePullPolicy: "IfNotPresent"
diff --git a/tools/travis/deploy-chart.sh b/tools/travis/deploy-chart.sh
index 78e2412..1f2d00e 100755
--- a/tools/travis/deploy-chart.sh
+++ b/tools/travis/deploy-chart.sh
@@ -200,14 +200,6 @@ invoker:
agent:
enabled: false
-providers:
- alarm:
- enabled: true
- cloudant:
- enabled: true
- kafka:
- enabled: true
-
nginx:
httpsNodePort: $WSK_PORT
EOF
