This is an automated email from the ASF dual-hosted git repository.
dgrove pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/openwhisk-deploy-kube.git
The following commit(s) were added to refs/heads/master by this push:
new ba72e9a Deploy with new scheduler (#729)
ba72e9a is described below
commit ba72e9af15bf940652b398762ccb2d88563b4462
Author: Erika Hunhoff <[email protected]>
AuthorDate: Mon Sep 26 08:48:11 2022 -0600
Deploy with new scheduler (#729)
---
.travis.yml | 1 +
docs/configurationChoices.md | 12 +
docs/k8s-custom-build-cluster-scaleup.md | 2 +-
docs/k8s-kind.md | 4 +-
helm/openwhisk/templates/_helpers.tpl | 43 ++++
helm/openwhisk/templates/_readiness.tpl | 22 ++
helm/openwhisk/templates/controller-pod.yaml | 29 ++-
helm/openwhisk/templates/etcd-pod.yaml | 114 +++++++++
helm/openwhisk/templates/etcd-pvc.yaml | 34 +++
helm/openwhisk/templates/etcd-svc.yaml | 32 +++
helm/openwhisk/templates/install-packages-job.yaml | 3 +
helm/openwhisk/templates/invoker-pod.yaml | 22 +-
helm/openwhisk/templates/scheduler-pdb.yaml | 34 +++
helm/openwhisk/templates/scheduler-pod.yaml | 283 +++++++++++++++++++++
helm/openwhisk/templates/scheduler-svc.yaml | 36 +++
helm/openwhisk/values.schema.json | 83 +++++-
helm/openwhisk/values.yaml | 92 ++++++-
tools/travis/collect-logs.sh | 1 +
tools/travis/deploy-chart.sh | 13 +
19 files changed, 844 insertions(+), 16 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 5d3295d..e823cb7 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -26,6 +26,7 @@ env:
- TRAVIS_KUBE_VERSION=v1.19 OW_INCLUDE_SYSTEM_TESTS=false
OW_CONTAINER_FACTORY=kubernetes OW_LEAN_MODE=true
- TRAVIS_KUBE_VERSION=v1.20 OW_INCLUDE_SYSTEM_TESTS=false
OW_CONTAINER_FACTORY=kubernetes
- TRAVIS_KUBE_VERSION=v1.21 OW_INCLUDE_SYSTEM_TESTS=false
OW_CONTAINER_FACTORY=kubernetes
+ - TRAVIS_KUBE_VERSION=v1.21 OW_INCLUDE_SYSTEM_TESTS=false
OW_CONTAINER_FACTORY=kubernetes OW_SCHEDULER_ENABLED=true
services:
- docker
diff --git a/docs/configurationChoices.md b/docs/configurationChoices.md
index 9244fab..5ade12c 100644
--- a/docs/configurationChoices.md
+++ b/docs/configurationChoices.md
@@ -45,6 +45,16 @@ components is not currently supported:
better management of the database and decouples its lifecycle from that of
the OpenWhisk deployment.
- The event providers: alarmprovider and kafkaprovider.
+### Openwhisk Scheduler
+
+By default, the scheduler is disabled. To enable the scheduler, add the
following
+to your `mycluster.yaml`
+
+```yaml
+scheduler:
+ enabled: true
+```
+
### Using an external database
You may want to use an external CouchDB or Cloudant instance instead
@@ -180,6 +190,8 @@ k8s:
enabled: false
```
+Currently, etcd persistence is not supported.
+
### Selectively Deploying Event Providers
The default settings of the Helm chart will deploy OpenWhisk's alarm
diff --git a/docs/k8s-custom-build-cluster-scaleup.md
b/docs/k8s-custom-build-cluster-scaleup.md
index 3632794..62040b4 100644
--- a/docs/k8s-custom-build-cluster-scaleup.md
+++ b/docs/k8s-custom-build-cluster-scaleup.md
@@ -40,7 +40,7 @@ Modifying the above mentioned parameters, one can easily
increase the concurrenc
In order to further increase the scale-up beyond `Small Scale`, one needs to
modify the following additional configurations appropriately (on top of the
above mentioned):
* `invoker:jvmHeapMB`: jvmHeap memory available to each invoker instance. May
or may not require increase based on running functions. For more information
check `troubleshooting` below.
* `invoker:containerFactory:_:replicaCount`: number of invoker instances that
will be used to handle the incoming workload. By default, there is only one
invoker instance which can become overwhelmed if workload goes beyond a certain
threshold.
-* `controller:replicaCount`: number of controller instances that will be used
to handle the incoming workload. Same as invoker instances.
+* `controller:replicaCount`: number of controller instances that will be used
to handle the incoming workload. Same as invoker and scheduler instances.
* `invoker:options`: Log processing at the invoker can become a bottleneck for
the KubernetesContainerFactory. One might try disabling invoker log processing
by setting it to
`-Dwhisk.spi.LogStoreProvider=org.apache.openwhisk.core.containerpool.logging.LogDriverLogStoreProvider`.
In general, one needs to offload log processing from the invoker to a
node-level log store provider if one is trying to push a large load through the
system.
## Troubleshooting
diff --git a/docs/k8s-kind.md b/docs/k8s-kind.md
index 769a36c..7d5b4f5 100644
--- a/docs/k8s-kind.md
+++ b/docs/k8s-kind.md
@@ -94,8 +94,8 @@ OpenWhisk apihost property to be set to localhost:31001
## Hints and Tips
If you are working on the core OpenWhisk system and want
-to use a locally built controller or invoker image to test
-your changes, you need to push the image to the docker image
+to use a locally built controller, invoker, or scheduler image
+to test your changes, you need to push the image to the docker image
repository inside the `kind` cluster.
For example, suppose I had a local change to the controller
diff --git a/helm/openwhisk/templates/_helpers.tpl
b/helm/openwhisk/templates/_helpers.tpl
index 847ef8e..121e199 100644
--- a/helm/openwhisk/templates/_helpers.tpl
+++ b/helm/openwhisk/templates/_helpers.tpl
@@ -42,6 +42,11 @@ app: {{ template "openwhisk.fullname" . }}
{{ .Release.Name }}-controller.{{ .Release.Namespace }}.svc.{{
.Values.k8s.domain }}
{{- end -}}
+{{/* hostname for scheduler */}}
+{{- define "openwhisk.scheduler_host" -}}
+{{ .Release.Name }}-scheduler.{{ .Release.Namespace }}.svc.{{
.Values.k8s.domain }}
+{{- end -}}
+
{{/* hostname for database */}}
{{- define "openwhisk.db_host" -}}
{{- if .Values.db.external -}}
@@ -68,6 +73,15 @@ app: {{ template "openwhisk.fullname" . }}
{{- end -}}
{{- end -}}
+{{/* hostname for etcd */}}
+{{- define "openwhisk.etcd_host" -}}
+{{- if .Values.etcd.external -}}
+{{ .Values.etcd.host }}
+{{- else -}}
+{{ .Release.Name }}-etcd.{{ .Release.Namespace }}.svc.{{ .Values.k8s.domain }}
+{{- end -}}
+{{- end -}}
+
{{/* client connection string for zookeeper cluster
(server1:port,server2:port, ... serverN:port)*/}}
{{- define "openwhisk.zookeeper_connect" -}}
{{- if .Values.zookeeper.external -}}
@@ -196,10 +210,24 @@ app: {{ template "openwhisk.fullname" . }}
value: {{ .Values.whisk.limits.activation.payload.max | quote }}
{{- end -}}
+{{/* Environment variables for configuring etcd */}}
+{{- define "openwhisk.etcdConfigEnvVars" -}}
+- name: "CONFIG_whisk_cluster_name"
+ value: {{ .Values.etcd.clusterName | quote }}
+- name: "CONFIG_whisk_etcd_hosts"
+ value: {{ include "openwhisk.etcd_host" . }}:{{ .Values.etcd.port }}
+- name: "CONFIG_whisk_etcd_lease_timeout"
+ value: {{ .Values.etcd.leaseTimeout | quote }}
+- name: "CONFIG_whisk_etcd_pool_threads"
+ value: {{ .Values.etcd.poolThreads | quote }}
+{{- end -}}
+
{{/* Environment variables for configuring kafka topics */}}
{{- define "openwhisk.kafkaConfigEnvVars" -}}
- name: "CONFIG_whisk_kafka_replicationFactor"
value: {{ .Values.whisk.kafka.replicationFactor | quote }}
+- name: "CONFIG_whisk_kafka_topics_prefix"
+ value: {{ .Values.whisk.kafka.topics.prefix | quote }}
- name: "CONFIG_whisk_kafka_topics_cacheInvalidation_retentionBytes"
value: {{ .Values.whisk.kafka.topics.cacheInvalidation.retentionBytes |
quote }}
- name: "CONFIG_whisk_kafka_topics_cacheInvalidation_retentionMs"
@@ -224,12 +252,27 @@ app: {{ template "openwhisk.fullname" . }}
value: {{ .Values.whisk.kafka.topics.health.retentionMs | quote }}
- name: "CONFIG_whisk_kafka_topics_health_segmentBytes"
value: {{ .Values.whisk.kafka.topics.health.segmentBytes | quote }}
+
- name: "CONFIG_whisk_kafka_topics_invoker_retentionBytes"
value: {{ .Values.whisk.kafka.topics.invoker.retentionBytes | quote }}
- name: "CONFIG_whisk_kafka_topics_invoker_retentionMs"
value: {{ .Values.whisk.kafka.topics.invoker.retentionMs | quote }}
- name: "CONFIG_whisk_kafka_topics_invoker_segmentBytes"
value: {{ .Values.whisk.kafka.topics.invoker.segmentBytes | quote }}
+
+- name: "CONFIG_whisk_kafka_topics_scheduler_retentionBytes"
+ value: {{ .Values.whisk.kafka.topics.scheduler.retentionBytes | quote }}
+- name: "CONFIG_whisk_kafka_topics_scheduler_retentionMs"
+ value: {{ .Values.whisk.kafka.topics.scheduler.retentionMs | quote }}
+- name: "CONFIG_whisk_kafka_topics_scheduler_segmentBytes"
+ value: {{ .Values.whisk.kafka.topics.scheduler.segmentBytes | quote }}
+
+- name: "CONFIG_whisk_kafka_topics_creationAck_retentionBytes"
+ value: {{ .Values.whisk.kafka.topics.creationAck.retentionBytes | quote }}
+- name: "CONFIG_whisk_kafka_topics_creationAck_retentionMs"
+ value: {{ .Values.whisk.kafka.topics.creationAck.retentionMs | quote }}
+- name: "CONFIG_whisk_kafka_topics_creationAck_segmentBytes"
+ value: {{ .Values.whisk.kafka.topics.creationAck.segmentBytes | quote }}
{{- end -}}
{{/* tlssecretname for ingress */}}
diff --git a/helm/openwhisk/templates/_readiness.tpl
b/helm/openwhisk/templates/_readiness.tpl
index da6a0fe..8de9d8c 100644
--- a/helm/openwhisk/templates/_readiness.tpl
+++ b/helm/openwhisk/templates/_readiness.tpl
@@ -38,6 +38,17 @@
command: ["sh", "-c",
'cacert="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"; token="$(cat
/var/run/secrets/kubernetes.io/serviceaccount/token)"; while true; do rc=$(curl
-sS --cacert $cacert --header "Authorization: Bearer $token"
https://kubernetes.default.svc/api/v1/namespaces/{{ .Release.Namespace
}}/endpoints/{{ .Release.Name }}-kafka | jq -r ".subsets[].addresses |
length"); echo "num ready kafka endpoints is $rc"; if [ $rc -gt 0 ]; then echo
"Success: ready kafka endpoin [...]
{{- end -}}
+{{/* Init container that waits for etcd to be ready */}}
+{{- define "openwhisk.readiness.waitForEtcd" -}}
+- name: "wait-for-etcd"
+ image: "{{- .Values.docker.registry.name -}}{{- .Values.utility.imageName
-}}:{{- .Values.utility.imageTag -}}"
+ imagePullPolicy: "IfNotPresent"
+ env:
+ - name: "READINESS_URL"
+ value: http://{{ include "openwhisk.etcd_host" . }}:{{ .Values.etcd.port
}}/health
+ command: ["sh", "-c", "while true; do echo 'checking etcd readiness';
health_result=$(curl -m 5 $READINESS_URL) && echo $health_result | jq -e '. |
select(.health==\"true\")'; result=$?; if [ $result -eq 0 ]; then echo
'Success: etcd is ready!'; break; fi; echo '...not ready yet; sleeping 3
seconds before retry'; sleep 3; done;"]
+{{- end -}}
+
{{/* Init container that waits for zookeeper to be ready */}}
{{- define "openwhisk.readiness.waitForZookeeper" -}}
- name: "wait-for-zookeeper"
@@ -57,6 +68,17 @@
command: ["sh", "-c", "result=1; until [ $result -eq 0 ]; do echo 'Checking
controller readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1;
done; echo 'Success: controller is ready'"]
{{- end -}}
+{{/* Init container that waits for scheduler to be ready */}}
+{{- define "openwhisk.readiness.waitForScheduler" -}}
+- name: "wait-for-scheduler"
+ image: "{{- .Values.docker.registry.name -}}{{- .Values.busybox.imageName
-}}:{{- .Values.busybox.imageTag -}}"
+ imagePullPolicy: "IfNotPresent"
+ env:
+ - name: "READINESS_URL"
+ value: http://{{ include "openwhisk.scheduler_host" . }}:{{
.Values.scheduler.endpoints.port }}/ping
+ command: ["sh", "-c", "result=1; until [ $result -eq 0 ]; do echo 'Checking
scheduler readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1;
done; echo 'Success: scheduler is ready'"]
+{{- end -}}
+
{{/* Init container that waits for at least 1 healthy invoker */}}
{{- define "openwhisk.readiness.waitForHealthyInvoker" -}}
- name: "wait-for-healthy-invoker"
diff --git a/helm/openwhisk/templates/controller-pod.yaml
b/helm/openwhisk/templates/controller-pod.yaml
index 6681370..ab81fb5 100644
--- a/helm/openwhisk/templates/controller-pod.yaml
+++ b/helm/openwhisk/templates/controller-pod.yaml
@@ -60,6 +60,9 @@ spec:
{{- if not .Values.controller.lean }}
# The controller must wait for kafka and/or couchdb to be ready before
it starts
{{ include "openwhisk.readiness.waitForKafka" . | indent 6 }}
+{{- if .Values.scheduler.enabled }}
+{{ include "openwhisk.readiness.waitForEtcd" . | indent 6 }}
+{{- end }}
{{- end }}
{{ include "openwhisk.readiness.waitForCouchDB" . | indent 6 }}
{{- if eq .Values.activationStoreBackend "ElasticSearch" }}
@@ -85,7 +88,7 @@ spec:
- name: controller
containerPort: {{ .Values.controller.port }}
- name: akka-remoting
- containerPort: 2552
+ containerPort: 25520
- name: akka-mgmt-http
containerPort: 19999
{{- if .Values.controller.lean }}
@@ -114,6 +117,11 @@ spec:
- name: "TZ"
value: {{ .Values.docker.timezone | quote }}
+ - name: "POD_IP"
+ valueFrom:
+ fieldRef:
+ fieldPath: status.podIP
+
- name: "CONFIG_whisk_info_date"
valueFrom:
configMapKeyRef:
@@ -137,6 +145,15 @@ spec:
- name: "RUNTIMES_MANIFEST"
value: {{ template "openwhisk.runtimes_manifest" . }}
+ # scheduler settings
+{{ if .Values.scheduler.enabled }}
+ - name: "CONFIG_whisk_spi_LoadBalancerProvider"
+ value: "org.apache.openwhisk.core.loadBalancer.FPCPoolBalancer"
+
+ - name: "CONFIG_whisk_spi_EntitlementSpiProvider"
+ value: "org.apache.openwhisk.core.entitlement.FPCEntitlementProvider"
+{{ end }}
+
# Action limits
{{ include "openwhisk.limitsEnvVars" . | indent 8 }}
@@ -151,11 +168,17 @@ spec:
value: "{{ include "openwhisk.kafka_connect" . }}"
{{ include "openwhisk.kafkaConfigEnvVars" . | indent 8 }}
+ # etcd properties
+{{- if .Values.scheduler.enabled }}
+{{ include "openwhisk.etcdConfigEnvVars" . | indent 8 }}
+{{- end }}
+
# properties for DB connection
{{ include "openwhisk.dbEnvVars" . | indent 8 }}
- name: "CONTROLLER_INSTANCES"
value: {{ .Values.controller.replicaCount | quote }}
+
{{- if gt (int .Values.controller.replicaCount) 1 }}
- name: "CONFIG_whisk_cluster_useClusterBootstrap"
value: "true"
@@ -169,7 +192,11 @@ spec:
value: "name={{ .Release.Name }}-controller"
- name: "CONFIG_akka_discovery_kubernetesApi_podPortName"
value: "akka-mgmt-http"
+{{- else }}
+ - name: "CONFIG_akka_cluster_seedNodes_0"
+ value: "akka://controller-actor-system@$(POD_IP):25520"
{{- end }}
+
{{- if .Values.metrics.prometheusEnabled }}
- name: "OPENWHISK_ENCODED_CONFIG"
value: {{ template "openwhisk.whiskconfig" . }}
diff --git a/helm/openwhisk/templates/etcd-pod.yaml
b/helm/openwhisk/templates/etcd-pod.yaml
new file mode 100644
index 0000000..dea4552
--- /dev/null
+++ b/helm/openwhisk/templates/etcd-pod.yaml
@@ -0,0 +1,114 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{ if not .Values.etcd.external }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ .Release.Name }}-etcd
+ labels:
+ name: {{ .Release.Name }}-etcd
+{{ include "openwhisk.label_boilerplate" . | indent 4 }}
+spec:
+ replicas: {{ .Values.etcd.replicaCount }}
+ selector:
+ matchLabels:
+ name: {{ .Release.Name }}-etcd
+ {{- if .Values.k8s.persistence.enabled }}
+ strategy:
+ type: "Recreate"
+ {{- end }}
+ template:
+ metadata:
+ labels:
+ name: {{ .Release.Name }}-etcd
+{{ include "openwhisk.label_boilerplate" . | indent 8 }}
+ spec:
+ restartPolicy: {{ .Values.etcd.restartPolicy }}
+
+ {{- if .Values.affinity.enabled }}
+ affinity:
+{{ include "openwhisk.affinity.core" . | indent 8 }}
+{{ include "openwhisk.affinity.selfAntiAffinity" ( printf "%s-etcd"
.Release.Name | quote ) | indent 8 }}
+ {{- end }}
+
+ {{- if .Values.toleration.enabled }}
+ tolerations:
+{{ include "openwhisk.toleration.core" . | indent 8 }}
+ {{- end }}
+
+{{- if .Values.k8s.persistence.enabled }}
+ volumes:
+ - name: etcd-data
+ persistentVolumeClaim:
+ claimName: {{ .Release.Name }}-etcd-pvc
+{{- end }}
+
+{{- if .Values.k8s.persistence.enabled }}
+ initContainers:
+ - name: etcd-init
+ image: "{{- .Values.docker.registry.name -}}{{-
.Values.busybox.imageName -}}:{{- .Values.busybox.imageTag -}}"
+ command:
+ - chown
+ - -v
+ - -R
+ - 999:999
+ - /data
+ volumeMounts:
+ - mountPath: /data
+ name: etcd-data
+ readOnly: false
+{{- end }}
+{{ include "openwhisk.docker.imagePullSecrets" . | indent 6 }}
+ # current command will always restart from scratch (no persistence)
+ containers:
+ - name: etcd
+ image: "{{- .Values.docker.registry.name -}}{{-
.Values.etcd.imageName -}}:{{- .Values.etcd.imageTag -}}"
+ command:
+ - /usr/local/bin/etcd
+ - --data-dir=/data
+ - --name
+ - etcd0
+ - --initial-advertise-peer-urls
+ - http://127.0.0.1:2480
+ - --advertise-client-urls
+ - http://0.0.0.0:{{ .Values.etcd.port }}
+ - --listen-peer-urls
+ - http://127.0.0.1:2480
+ - --listen-client-urls
+ - http://0.0.0.0:{{ .Values.etcd.port }}
+ - --initial-cluster
+ - etcd0=http://127.0.0.1:2480
+ - --initial-cluster-state
+ - new
+ - --initial-cluster-token
+ - openwhisk-etcd-token
+ - --quota-backend-bytes
+ - "0"
+ - --snapshot-count
+ - "100000"
+ - --auto-compaction-retention
+ - "1"
+ - --auto-compaction-mode
+ - periodic
+ - --log-level
+ - info
+ imagePullPolicy: {{ .Values.etcd.imagePullPolicy | quote }}
+ ports:
+ - name: etcd
+ containerPort: {{ .Values.etcd.port }}
+{{ end }}
diff --git a/helm/openwhisk/templates/etcd-pvc.yaml
b/helm/openwhisk/templates/etcd-pvc.yaml
new file mode 100644
index 0000000..860360c
--- /dev/null
+++ b/helm/openwhisk/templates/etcd-pvc.yaml
@@ -0,0 +1,34 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{- if and (not .Values.etcd.external) .Values.k8s.persistence.enabled }}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: {{ .Release.Name }}-etcd-pvc
+ labels:
+{{ include "openwhisk.label_boilerplate" . | indent 4 }}
+spec:
+{{- if not .Values.k8s.persistence.hasDefaultStorageClass }}
+ storageClassName: {{ .Values.k8s.persistence.explicitStorageClass }}
+{{- end }}
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: {{ .Values.etcd.persistence.size }}
+{{- end }}
diff --git a/helm/openwhisk/templates/etcd-svc.yaml
b/helm/openwhisk/templates/etcd-svc.yaml
new file mode 100644
index 0000000..0765c97
--- /dev/null
+++ b/helm/openwhisk/templates/etcd-svc.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{ if not .Values.etcd.external }}
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ .Release.Name }}-etcd
+ labels:
+ name: {{ .Release.Name }}-etcd
+{{ include "openwhisk.label_boilerplate" . | indent 4 }}
+spec:
+ selector:
+ name: {{ .Release.Name }}-etcd
+ ports:
+ - port: {{ .Values.etcd.port }}
+ name: etcd
+{{ end }}
diff --git a/helm/openwhisk/templates/install-packages-job.yaml
b/helm/openwhisk/templates/install-packages-job.yaml
index fc7cb30..ba3d025 100644
--- a/helm/openwhisk/templates/install-packages-job.yaml
+++ b/helm/openwhisk/templates/install-packages-job.yaml
@@ -41,6 +41,9 @@ spec:
{{ include "openwhisk.readiness.waitForController" . | indent 6 }}
{{- else }}
{{ include "openwhisk.readiness.waitForHealthyInvoker" . | indent 6 }}
+{{- if .Values.scheduler.enabled }}
+{{ include "openwhisk.readiness.waitForScheduler" . | indent 6 }}
+{{ end }}
{{ end }}
{{ include "openwhisk.docker.imagePullSecrets" . | indent 6 }}
containers:
diff --git a/helm/openwhisk/templates/invoker-pod.yaml
b/helm/openwhisk/templates/invoker-pod.yaml
index dbb8028..4102ff3 100644
--- a/helm/openwhisk/templates/invoker-pod.yaml
+++ b/helm/openwhisk/templates/invoker-pod.yaml
@@ -76,7 +76,7 @@ spec:
# Pull images for all default runtimes before starting invoker
{{ include "openwhisk.docker_pull_runtimes" . | indent 6 }}
{{- end }}
- # Wait for a controller to be up (which implies kafka, zookeeper,
couchdb are all up as well).
+ # Wait for a controller to be up (which implies kafka, zookeeper,
couchdb, etcd are all up as well).
{{ include "openwhisk.readiness.waitForController" . | indent 6 }}
{{ include "openwhisk.docker.imagePullSecrets" . | indent 6 }}
containers:
@@ -140,6 +140,21 @@ spec:
- name: "JAVA_OPTS"
value: "-Xmx{{- .Values.invoker.jvmHeapMB -}}M {{
.Values.invoker.jvmOptions }}"
+{{- if .Values.scheduler.enabled }}
+ # Options only needed when new scheduler is enabled
+ - name: "CONFIG_whisk_scheduler_dataManagementService_retryInterval"
+ value: "{{ .Values.scheduler.dataManagementService.retryInterval
}}"
+
+ - name: "CONFIG_whisk_spi_InvokerProvider"
+ value: "org.apache.openwhisk.core.invoker.FPCInvokerReactive"
+
+ - name: "CONFIG_whisk_spi_InvokerServerProvider"
+ value: "org.apache.openwhisk.core.invoker.FPCInvokerServer"
+
+ - name: "CONFIG_whisk_invoker_containerCreation_maxPeek"
+ value: "500"
+{{- end }}
+
# Invoker options
- name: "INVOKER_OPTS"
value: "{{ .Values.invoker.options }} {{ include
"openwhisk.invoker.add_opts" . }}"
@@ -170,6 +185,11 @@ spec:
value: "{{ include "openwhisk.kafka_connect" . }}"
{{ include "openwhisk.kafkaConfigEnvVars" . | indent 10 }}
+ # etcd properties
+{{- if .Values.scheduler.enabled }}
+{{ include "openwhisk.etcdConfigEnvVars" . | indent 10 }}
+{{- end }}
+
# properties for zookeeper connection
- name: "ZOOKEEPER_HOSTS"
value: "{{ include "openwhisk.zookeeper_connect" . }}"
diff --git a/helm/openwhisk/templates/scheduler-pdb.yaml
b/helm/openwhisk/templates/scheduler-pdb.yaml
new file mode 100644
index 0000000..ccaf41f
--- /dev/null
+++ b/helm/openwhisk/templates/scheduler-pdb.yaml
@@ -0,0 +1,34 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{- if .Values.scheduler.enabled }}
+{{- if and .Values.pdb.enable (gt (int .Values.scheduler.replicaCount) 1) }}
+---
+apiVersion: policy/v1beta1
+kind: PodDisruptionBudget
+metadata:
+ name: {{ .Release.Name }}-scheduler-pdb
+ labels:
+ name: {{ .Release.Name }}-scheduler-pdb
+{{ include "openwhisk.label_boilerplate" . | indent 4 }}
+spec:
+ selector:
+ matchLabels:
+ name: {{ .Release.Name }}-scheduler
+ maxUnavailable: {{ .Values.pdb.scheduler.maxUnavailable | default 1 }}
+{{- end }}
+{{- end }}
diff --git a/helm/openwhisk/templates/scheduler-pod.yaml
b/helm/openwhisk/templates/scheduler-pod.yaml
new file mode 100644
index 0000000..dcafe80
--- /dev/null
+++ b/helm/openwhisk/templates/scheduler-pod.yaml
@@ -0,0 +1,283 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{- if .Values.scheduler.enabled }}
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: {{ .Release.Name }}-scheduler
+ labels:
+ name: {{ .Release.Name }}-scheduler
+{{ include "openwhisk.label_boilerplate" . | indent 4 }}
+spec:
+ serviceName: {{ .Release.Name }}-scheduler
+ podManagementPolicy: "Parallel"
+ replicas: {{ .Values.scheduler.replicaCount }}
+ selector:
+ matchLabels:
+ name: {{ .Release.Name }}-scheduler
+ template:
+ metadata:
+ labels:
+ name: {{ .Release.Name }}-scheduler
+{{ include "openwhisk.label_boilerplate" . | indent 8 }}
+
+ {{- if .Values.metrics.prometheusEnabled }}
+ annotations:
+ prometheus.io/scrape: 'true'
+ prometheus.io/port: '{{ .Values.scheduler.endpoints.port }}'
+ {{- end }}
+
+ spec:
+ serviceAccountName: {{ .Release.Name }}-core
+ restartPolicy: {{ .Values.scheduler.restartPolicy }}
+
+ {{- if .Values.affinity.enabled }}
+ affinity:
+{{ include "openwhisk.affinity.core" . | indent 8 }}
+{{ include "openwhisk.affinity.selfAntiAffinity" ( printf "%s-scheduler"
.Release.Name ) | indent 8 }}
+ {{- end }}
+
+ {{- if .Values.toleration.enabled }}
+ tolerations:
+{{ include "openwhisk.toleration.core" . | indent 8 }}
+ {{- end }}
+
+ initContainers:
+ # The scheduler must wait for etcd, kafka, and/or couchdb to be ready
before it starts
+{{ include "openwhisk.readiness.waitForEtcd" . | indent 6 }}
+{{ include "openwhisk.readiness.waitForKafka" . | indent 6 }}
+{{ include "openwhisk.readiness.waitForCouchDB" . | indent 6 }}
+{{ include "openwhisk.docker.imagePullSecrets" . | indent 6 }}
+
+ containers:
+ - name: scheduler
+ imagePullPolicy: {{ .Values.scheduler.imagePullPolicy | quote }}
+ image: "{{- .Values.docker.registry.name -}}{{-
.Values.scheduler.imageName -}}:{{- .Values.scheduler.imageTag -}}"
+ command: ["/bin/bash", "-c", "/init.sh `hostname | awk -F '-' '{print
$NF}'`"]
+ ports:
+ - name: scheduler
+ containerPort: {{ .Values.scheduler.endpoints.port }}
+ - name: rpc
+ containerPort: {{ .Values.scheduler.endpoints.rpcPort }}
+ - name: akka-remoting
+ containerPort: {{ .Values.scheduler.endpoints.akkaPort }}
+ - name: akka-mgmt-http
+ containerPort: 19999
+ livenessProbe:
+ httpGet:
+ path: "/ping"
+ port: {{ .Values.scheduler.endpoints.port }}
+ scheme: "HTTP"
+ initialDelaySeconds: {{
.Values.probes.scheduler.livenessProbe.initialDelaySeconds }}
+ periodSeconds: {{
.Values.probes.scheduler.livenessProbe.periodSeconds }}
+ timeoutSeconds: {{
.Values.probes.scheduler.livenessProbe.timeoutSeconds }}
+ readinessProbe:
+ httpGet:
+ path: "/ping"
+ port: {{ .Values.scheduler.endpoints.port }}
+ scheme: "HTTP"
+ initialDelaySeconds: {{
.Values.probes.scheduler.readinessProbe.initialDelaySeconds }}
+ periodSeconds: {{
.Values.probes.scheduler.readinessProbe.periodSeconds }}
+ timeoutSeconds: {{
.Values.probes.scheduler.readinessProbe.timeoutSeconds }}
+ env:
+ - name: "PORT"
+ value: {{ .Values.scheduler.endpoints.port | quote }}
+
+ - name: "POD_IP"
+ valueFrom:
+ fieldRef:
+ fieldPath: status.podIP
+
+ - name: "WHISK_SCHEDULER_ENDPOINTS_RPCPORT"
+ value: {{ .Values.scheduler.endpoints.rpcPort | quote }}
+
+ - name: "WHISK_SCHEDULER_ENDPOINTS_AKKAPORT"
+ value: {{ .Values.scheduler.endpoints.akkaPort | quote }}
+
+ - name: "CONFIG_akka_remote_artery_canonical_port"
+ value: {{ .Values.scheduler.endpoints.akkaPort | quote }}
+
+ - name: "TZ"
+ value: {{ .Values.docker.timezone | quote }}
+
+ - name: "CONFIG_whisk_info_date"
+ valueFrom:
+ configMapKeyRef:
+ name: {{ .Release.Name }}-whisk.config
+ key: whisk_info_date
+ - name: "CONFIG_whisk_info_buildNo"
+ valueFrom:
+ configMapKeyRef:
+ name: {{ .Release.Name }}-whisk.config
+ key: whisk_info_buildNo
+
+ # Java options
+ - name: "JAVA_OPTS"
+ value: "-Xmx{{- .Values.scheduler.jvmHeapMB -}}M {{
.Values.scheduler.jvmOptions }}"
+
+ # specific scheduler arguments
+ - name: "SCHEDULER_OPTS"
+ value: "{{ .Values.scheduler.options }}"
+
+ # Action limits
+{{ include "openwhisk.limitsEnvVars" . | indent 8 }}
+
+ - name: "CONFIG_whisk_durationChecker_timeWindow"
+ value: {{ .Values.whisk.durationChecker.timeWindow }}
+
+ - name: "CONFIG_whisk_loadbalancer_blackboxFraction"
+ value: {{ .Values.whisk.loadbalancer.blackboxFraction | quote }}
+
+ - name: "CONFIG_whisk_loadbalancer_timeoutFactor"
+ value: {{ .Values.whisk.loadbalancer.timeoutFactor | quote }}
+
+ # Kafka properties
+ - name: "KAFKA_HOSTS"
+ value: "{{ include "openwhisk.kafka_connect" . }}"
+{{ include "openwhisk.kafkaConfigEnvVars" . | indent 8 }}
+
+ # etcd properties
+{{ include "openwhisk.etcdConfigEnvVars" . | indent 8 }}
+
+ # properties for zookeeper connection
+ - name: "ZOOKEEPER_HOSTS"
+ value: "{{ include "openwhisk.zookeeper_connect" . }}"
+
+ # action runtimes
+ - name: "RUNTIMES_MANIFEST"
+ value: {{ template "openwhisk.runtimes_manifest" . }}
+
+ # properties for DB connection
+{{ include "openwhisk.dbEnvVars" . | indent 8 }}
+
+ - name: "WHISK_SCHEDULER_ENDPOINTS_HOST"
+ valueFrom:
+ fieldRef:
+ fieldPath: status.podIP
+
+ - name: "CONFIG_whisk_scheduler_dataManagementService_retryInterval"
+ value: {{ .Values.scheduler.dataManagementService.retryInterval |
quote }}
+
+ - name: "CONFIG_whisk_scheduler_queueManager_maxSchedulingTime"
+ value: {{ .Values.scheduler.queueManager.maxSchedulingTime | quote }}
+
+ - name: "CONFIG_whisk_scheduler_queueManager_maxRetriesToGetQueue"
+ value: {{ .Values.scheduler.queueManager.maxRetriesToGetQueue |
quote }}
+
+ - name: "CONFIG_whisk_scheduler_queue_idleGrace"
+ value: {{ .Values.scheduler.queue.idleGrace | quote }}
+
+ - name: "CONFIG_whisk_scheduler_queue_stopGrace"
+ value: {{ .Values.scheduler.queue.stopGrace | quote }}
+
+ - name: "CONFIG_whisk_scheduler_queue_flushGrace"
+ value: {{ .Values.scheduler.queue.flushGrace | quote }}
+
+ - name: "CONFIG_whisk_scheduler_queue_gracefulShutdownTimeout"
+ value: {{ .Values.scheduler.queue.gracefulShutdownTimeout | quote }}
+
+ - name: "CONFIG_whisk_scheduler_queue_maxRetentionSize"
+ value: {{ .Values.scheduler.queue.maxRetentionSize | quote }}
+
+ - name: "CONFIG_whisk_scheduler_queue_maxRetentionMs"
+ value: {{ .Values.scheduler.queue.maxRetentionMs | quote }}
+
+ - name: "CONFIG_whisk_scheduler_queue_maxBlackboxRetentionMs"
+ value: {{ .Values.scheduler.queue.maxBlackboxRetentionMs | quote }}
+
+ - name: "CONFIG_whisk_scheduler_queue_throttlingFraction"
+ value: {{ .Values.scheduler.queue.throttlingFraction | quote }}
+
+ - name: "CONFIG_whisk_scheduler_queue_durationBufferSize"
+ value: {{ .Values.scheduler.queue.durationBufferSize | quote }}
+
+ - name: "CONFIG_whisk_scheduler_protocol"
+ value: {{ .Values.scheduler.protocol | quote }}
+
+ - name: "CONFIG_whisk_scheduler_maxPeek"
+ value: {{ .Values.scheduler.maxPeek | quote }}
+
+ - name: "CONFIG_whisk_scheduler_inProgressJobRetention"
+ value: {{ .Values.scheduler.inProgressJobRetention | quote }}
+
+ - name: "CONFIG_whisk_scheduler_blackboxMultiple"
+ value: {{ .Values.scheduler.blackboxMultiple | quote }}
+
+ - name: "CONFIG_whisk_scheduler_scheduling_staleThreshold"
+ value: {{ .Values.scheduler.scheduling.staleThreshold | quote }}
+
+ - name: "CONFIG_whisk_scheduler_scheduling_checkInterval"
+ value: {{ .Values.scheduler.scheduling.checkInterval | quote }}
+
+ - name: "CONFIG_whisk_scheduler_scheduling_dropInterval"
+ value: {{ .Values.scheduler.scheduling.dropInterval | quote }}
+
+ - name: "SCHEDULER_INSTANCES"
+ value: {{ .Values.scheduler.replicaCount | quote }}
+
+{{- if gt (int .Values.scheduler.replicaCount) 1 }}
+ - name: "CONFIG_whisk_cluster_useClusterBootstrap"
+ value: "true"
+ - name: "CONFIG_akka_actor_provider"
+ value: "cluster"
+ - name:
"CONFIG_akka_management_cluster_bootstrap_contactPointDiscovery_discoveryMethod"
+ value: "kubernetes-api"
+ - name: "CONFIG_akka_coordinatedShutdown_exitJvm"
+ value: "on"
+ - name: "CONFIG_akka_discovery_kubernetesApi_podNamespace"
+ value: {{ .Release.Namespace | quote }}
+ - name: "CONFIG_akka_discovery_kubernetesApi_podLabelSelector"
+ value: "name={{ .Release.Name }}-scheduler"
+{{- else }}
+ - name: "CONFIG_akka_cluster_seedNodes_0"
+ value: "akka://scheduler-actor-system@$(POD_IP):{{
.Values.scheduler.endpoints.akkaPort }}"
+{{- end }}
+
+{{- if .Values.metrics.prometheusEnabled }}
+ - name: "OPENWHISK_ENCODED_CONFIG"
+ value: {{ template "openwhisk.whiskconfig" . }}
+{{- end }}
+{{ if or .Values.metrics.kamonEnabled .Values.metrics.prometheusEnabled }}
+ - name: "METRICS_KAMON"
+ value: "true"
+{{ end }}
+{{ if .Values.metrics.kamonTags }}
+ - name: "METRICS_KAMON_TAGS"
+ value: "true"
+{{ end }}
+{{ if .Values.metrics.userMetricsEnabled }}
+ - name: "CONFIG_whisk_userEvents_enabled"
+ value: "true"
+{{ end }}
+ - name: "CONFIG_logback_log_level"
+ value: "{{ .Values.scheduler.loglevel }}"
+{{- if eq .Values.activationStoreBackend "ElasticSearch" }}
+ - name: "CONFIG_whisk_activationStore_elasticsearch_protocol"
+ value: "{{ .Values.elasticsearch.protocol }}"
+ - name: "CONFIG_whisk_activationStore_elasticsearch_hosts"
+ value: {{ template "openwhisk.elasticsearch_connect" . }}
+ - name: "CONFIG_whisk_activationStore_elasticsearch_indexPattern"
+ value: {{ .Values.elasticsearch.indexPattern }}
+ - name: "CONFIG_whisk_activationStore_elasticsearch_username"
+ value: "{{ .Values.elasticsearch.username }}"
+ - name: "CONFIG_whisk_activationStore_elasticsearch_password"
+ value: "{{ .Values.elasticsearch.password }}"
+ - name: "CONFIG_whisk_spi_ActivationStoreProvider"
+ value:
"org.apache.openwhisk.core.database.elasticsearch.ElasticSearchActivationStoreProvider"
+{{- end }}
+{{- end }}
diff --git a/helm/openwhisk/templates/scheduler-svc.yaml
b/helm/openwhisk/templates/scheduler-svc.yaml
new file mode 100644
index 0000000..fe9fd4b
--- /dev/null
+++ b/helm/openwhisk/templates/scheduler-svc.yaml
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{- if .Values.scheduler.enabled }}
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ .Release.Name }}-scheduler
+ labels:
+ name: {{ .Release.Name }}-scheduler
+{{ include "openwhisk.label_boilerplate" . | indent 4 }}
+spec:
+ selector:
+ name: {{ .Release.Name }}-scheduler
+ ports:
+ - port: {{ .Values.scheduler.endpoints.port }}
+ name: http
+ - port: {{ .Values.scheduler.endpoints.rpcPort }}
+ name: grpc
+ - port: {{ .Values.scheduler.endpoints.akkaPort }}
+ name: akka
+{{- end }}
diff --git a/helm/openwhisk/values.schema.json
b/helm/openwhisk/values.schema.json
index 9820768..e36ffd4 100644
--- a/helm/openwhisk/values.schema.json
+++ b/helm/openwhisk/values.schema.json
@@ -150,8 +150,7 @@
"time": { "$ref": "#/definitions/actionLimitSpecString" },
"memory": { "$ref": "#/definitions/actionLimitSpecString" },
"concurrency": { "$ref":
"#/definitions/actionLimitSpecInteger" },
- "log": { "$ref": "#/definitions/actionLimitSpecString" },
- "time": { "$ref": "#/definitions/actionLimitSpecString" }
+ "log": { "$ref": "#/definitions/actionLimitSpecString" }
}
},
"activation": {
@@ -182,11 +181,14 @@
"topics": {
"type": "object",
"properties": {
+ "prefix": { "type": "string" },
"cacheInvalidation": { "$ref": "#/definitions/kafkaTopicConfig"
},
"completed": { "$ref": "#/definitions/kafkaTopicConfig" },
"events": { "$ref": "#/definitions/kafkaTopicConfig" },
"health": { "$ref": "#/definitions/kafkaTopicConfig" },
- "invoker": { "$ref": "#/definitions/kafkaTopicConfig" }
+ "invoker": { "$ref": "#/definitions/kafkaTopicConfig" },
+ "scheduler": { "$ref": "#/definitions/kafkaTopicConfig" },
+ "creationAck": { "$ref": "#/definitions/kafkaTopicConfig" }
}
}
}
@@ -198,6 +200,10 @@
}
},
"runtimes": { "type": "string" },
+ "durationChecker": {
+ "type": "object",
+ "timeWindow": { "type": "string" }
+ },
"testing": {
"type": "object",
"properties": {
@@ -368,6 +374,73 @@
]
},
+ "scheduler": {
+ "properties": { "enabled": { "type": "boolean" } },
+ "required": ["enabled"],
+ "if": { "properties": { "enabled": { "const": true } } },
+ "then": {
+ "allOf": [
+ { "$ref": "#/definitions/podspec" },
+ { "properties": {
+ "endpoints": {
+ "properties": {
+ "port": { "type": "integer", "minimum": 0 },
+ "rpcPort": { "type": "integer", "minimum": 0 },
+ "akkaPort": { "type": "integer", "minimum": 0 } },
+ "required": ["port", "rpcPort", "akkaPort"] },
+ "options": { "type": "string" },
+ "jvmHeapMB": { "type": "string" },
+ "jvmOptions": { "type": "string" },
+ "logLevel": { "type": "string" },
+ "protocol": { "type": "string" },
+ "maxPeek": { "type": "integer", "minimum": 0 },
+ "inProgressJobRetention": { "type": "string" },
+ "blackboxMultiple": { "type": "integer" },
+ "dataManagementService": { "retryInterval": { "type": "string" } },
+ "queue": {
+ "idleGrace": { "type": "string" },
+ "stopGrace": { "type": "string" },
+ "flushGrace": { "type": "string" },
+ "gracefulShutdownTimeout": { "type": "string" },
+ "maxRetentionSize": { "type": "integer", "minimum": 0 },
+ "maxRetentionMs": { "type": "integer", "minimum": 0 },
+ "maxBlackboxRetentionMs": { "type": "integer", "minimum": 0 },
+ "throttlingFraction": { "type": "number", "minimum": 0,
"maximum": 1.0 },
+ "durationBufferSize": { "type": "integer", "minimum": 0 } },
+ "queueManager": {
+ "maxSchedulingTime": { "type": "string" },
+ "maxRetriesToGetQueue": { "type": "integer", "minimum": 0 } },
+ "scheduling": {
+ "staleThreshold": { "type": "string" },
+ "checkInterval": { "type": "string" },
+ "dropInterval": { "type": "string" } }
+ },
+ "required": ["jvmHeapMB"]
+ }
+ ]
+ }
+ },
+
+ "etcd": {
+ "type": "object",
+ "properties": {
+ "external": { "type": "boolean" },
+ "clusterName": { "type": "string" } },
+ "required": ["external"],
+ "if": { "properties": { "external": { "const": false } } },
+ "then": {
+ "allOf": [
+ { "$ref": "#/definitions/podspec" },
+ { "properties": { "port": { "type": "integer", "minimum": 0 } },
"required": ["port"] },
+ { "properties": { "replicaCount": { "const": 1 } } },
+ { "properties": { "leaseTimeout": { "type": "integer" } } },
+ { "properties": { "poolThreads": { "type": "integer", "minimum": 1 }
} },
+ { "properties": { "persistence": { "size": { "type": "string" } } }
},
+ { "$ref": "#/definitions/optionalExternalHostCheck" }
+ ]
+ }
+ },
+
"invoker": {
"allOf": [
{ "$ref": "#/definitions/daemonset" },
@@ -514,7 +587,9 @@
"type": "object",
"properties": {
"zookeeper": { "$ref": "#/definitions/standardPodProbes" },
- "controller": { "$ref": "#/definitions/standardPodProbes" }
+ "kafka": { "$ref": "#/definitions/standardPodProbes" },
+ "controller": { "$ref": "#/definitions/standardPodProbes" },
+ "scheduler": { "$ref": "#/definitions/standardPodProbes" }
}
}
},
diff --git a/helm/openwhisk/values.yaml b/helm/openwhisk/values.yaml
index 787e6e7..d6c19da 100644
--- a/helm/openwhisk/values.yaml
+++ b/helm/openwhisk/values.yaml
@@ -100,6 +100,7 @@ whisk:
kafka:
replicationFactor: ""
topics:
+ prefix: ""
cacheInvalidation:
segmentBytes: ""
retentionBytes: ""
@@ -120,17 +121,27 @@ whisk:
segmentBytes: ""
retentionBytes: ""
retentionMs: ""
+ scheduler:
+ segmentBytes: ""
+ retentionBytes: ""
+ retentionMs: ""
+ creationAck:
+ segmentBytes: ""
+ retentionBytes: ""
+ retentionMs: ""
containerPool:
userMemory: "2048m"
runtimes: "runtimes.json"
+ durationChecker:
+ timeWindow: "1 d"
testing:
includeTests: true
includeSystemTests: false
versions:
openwhisk:
- buildDate: "2022-06-23-06:18:00Z"
- buildNo: "20220623a"
- gitTag: "c5970a657a3070e9964cb14715b9df31819d3b75"
+ buildDate: "2022-09-13-02:40:10Z"
+ buildNo: "20220912"
+ gitTag: "a1639f0e4d7270c9a230190ac26acb61413b6bbb"
openwhiskCli:
tag: "1.1.0"
openwhiskCatalog:
@@ -151,7 +162,7 @@ k8s:
# Images used to run auxillary tasks/jobs
utility:
imageName: "openwhisk/ow-utils"
- imageTag: "c5970a6"
+ imageTag: "a1639f0"
imagePullPolicy: "IfNotPresent"
# Docker registry
@@ -249,7 +260,7 @@ nginx:
# Controller configurations
controller:
imageName: "openwhisk/controller"
- imageTag: "c5970a6"
+ imageTag: "a1639f0"
imagePullPolicy: "IfNotPresent"
replicaCount: 1
restartPolicy: "Always"
@@ -259,10 +270,68 @@ controller:
jvmOptions: ""
loglevel: "INFO"
+# Scheduler configurations
+scheduler:
+ enabled: false
+ imageName: "openwhisk/scheduler"
+ imageTag: "a1639f0"
+ imagePullPolicy: "IfNotPresent"
+ replicaCount: 1
+ restartPolicy: "Always"
+ endpoints:
+ akkaPort: 25520
+ port: 8080
+ rpcPort: 13001
+ options: ""
+ jvmHeapMB: "1024"
+ jvmOptions: ""
+ loglevel: "INFO"
+ protocol: "http"
+ maxPeek: 128
+ # Sometimes the kubernetes client takes a long time for pod creation
+ inProgressJobRetention: "20 seconds"
+ blackboxMultiple: 15
+ dataManagementService:
+ retryInterval: "1 second"
+ queueManager:
+ maxSchedulingTime: "20 seconds"
+ maxRetriesToGetQueue: "13"
+ queue:
+ idleGrace: "20 seconds"
+ stopGrace: "20 seconds"
+ flushGrace: "60 seconds"
+ gracefulShutdownTimeout: "5 seconds"
+ maxRetentionSize: 10000
+ maxRetentionMs: 60000
+ maxBlackboxRetentionMs: 300000
+ throttlingFraction: 0.9
+ durationBufferSize: 10
+ scheduling:
+ staleThreshold: "100ms"
+ checkInterval: "100ms"
+ dropInterval: "10 minutes"
+
+# etcd (used by scheduler and controller if scheduler is enabled)
+etcd:
+ # NOTE: external etcd is not supported yet
+ external: false
+ clusterName: ""
+ imageName: "quay.io/coreos/etcd"
+ imageTag: "v3.4.0"
+ imagePullPolicy: "IfNotPresent"
+ # NOTE: setting replicaCount > 1 will not work; need to add etcd cluster
configuration
+ replicaCount: 1
+ restartPolicy: "Always"
+ port: 2379
+ leaseTimeout: 1
+ poolThreads: 10
+ persistence:
+ size: 256Mi
+
# Invoker configurations
invoker:
imageName: "openwhisk/invoker"
- imageTag: "c5970a6"
+ imageTag: "a1639f0"
imagePullPolicy: "IfNotPresent"
restartPolicy: "Always"
runtimeDeleteTimeout: "30 seconds"
@@ -317,7 +386,7 @@ redis:
# User-events configuration
user_events:
imageName: "openwhisk/user-events"
- imageTag: "c5970a6"
+ imageTag: "a1639f0"
imagePullPolicy: "IfNotPresent"
replicaCount: 1
restartPolicy: "Always"
@@ -462,6 +531,15 @@ probes:
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
+ scheduler:
+ livenessProbe:
+ initialDelaySeconds: 10
+ periodSeconds: 10
+ timeoutSeconds: 1
+ readinessProbe:
+ initialDelaySeconds: 10
+ periodSeconds: 10
+ timeoutSeconds: 1
# Pod Disruption Budget allows Pods to survive Voluntary and Involuntary
Disruptions.
# for more information refer -
https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
diff --git a/tools/travis/collect-logs.sh b/tools/travis/collect-logs.sh
index 5725497..f6c9d77 100755
--- a/tools/travis/collect-logs.sh
+++ b/tools/travis/collect-logs.sh
@@ -32,6 +32,7 @@ kubectl -n openwhisk logs -lname=ow4travis-couchdb >&
logs/couchdb.log
kubectl -n openwhisk logs -lname=ow4travis-zookeeper >& logs/zookeeper.log
kubectl -n openwhisk logs -lname=ow4travis-kafka >& logs/kafka.log
kubectl -n openwhisk logs -lname=ow4travis-controller >& logs/controller.log
+kubectl -n openwhisk logs -lname=ow4travis-scheduler >& logs/scheduler.log
kubectl -n openwhisk logs -lname=ow4travis-invoker -c docker-pull-runtimes >&
logs/invoker-docker-pull.log
kubectl -n openwhisk logs -lname=ow4travis-invoker -c invoker >&
logs/invoker-invoker.log
kubectl -n openwhisk logs -lname=ow4travis-nginx >& logs/nginx.log
diff --git a/tools/travis/deploy-chart.sh b/tools/travis/deploy-chart.sh
index e4bcc58..b528ffe 100755
--- a/tools/travis/deploy-chart.sh
+++ b/tools/travis/deploy-chart.sh
@@ -171,6 +171,9 @@ OW_INCLUDE_SYSTEM_TESTS=${OW_INCLUDE_SYSTEM_TESTS:="false"}
# Default timeout limit to 60 steps
TIMEOUT_STEP_LIMIT=${TIMEOUT_STEP_LIMIT:=60}
+# Default is to not use the scheduler component
+OW_SCHEDULER_ENABLED=${OW_SCHEDULER_ENABLED:="false"}
+
# Create namespace
kubectl create namespace openwhisk
@@ -194,6 +197,9 @@ invoker:
controller:
lean: ${OW_LEAN_MODE:-false}
+scheduler:
+ enabled: $OW_SCHEDULER_ENABLED
+
metrics:
userMetricsEnabled: true
EOF
@@ -216,12 +222,19 @@ if [ "${OW_LEAN_MODE:-false}" == "false" ]; then
# Wait for the controller to confirm that it has at least one healthy invoker
verifyHealthyInvoker
+ if [ "${OW_SCHEDULER_ENABLED:-false}" == "true" ]; then
+ # Wait for scheduler to be up
+ statefulsetHealthCheck "ow4travis-scheduler"
+ fi
+
# Verify that the user-metrics components were deployed successfully
deploymentHealthCheck "ow4travis-user-events"
# deploymentHealthCheck "ow4travis-prometheus-server"
deploymentHealthCheck "ow4travis-grafana"
fi
+
+
# Wait for install-packages job to complete successfully
jobHealthCheck "ow4travis-install-packages"
