I have an kubernetes cluster of 2 nodes (1 master and 1 worker) and was trying
to deploy openwhisk following documentation steps. But controller and invoker
pods are stuck in PodInitializing phase. I am using latest clone.
Please find details as below:
-> kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
server-f2-02.ipc3dev.com Ready master 33d v1.11.0
beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=server-f2-02.ipc3dev.com,node-role.kubernetes.io/master=master
server-f2-0f.ipc3dev.com Ready worker 33d v1.11.0
beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=server-f2-0f.ipc3dev.com,node-role.kubernetes.io/worker=worker,openwhisk-role=invoker
-> kubectl describe nodes | grep InternalIP
nternalIP: 10.71.3.7
InternalIP: 10.71.3.8
-> cat mycluster.yaml
whisk:
ingress:
type: NodePort
api_host_name: 10.71.3.7
api_host_port: 31001
nginx:
httpsNodePort: 31001
-> kubectl -n openwhisk get pods
NAME READY STATUS RESTARTS AGE
apigateway-64b5fb47df-h8v5f 1/1 Running 0 2h
controller-0 0/1 Init:1/2 0 2h
couchdb-6db7d989dd-886mc 1/1 Running 0 2h
invoker-b5mpt 0/1 Init:1/2 0 2h
kafka-0 1/1 Running 0 2h
nginx-55d45778cb-5vgd6 1/1 Running 0 2h
redis-6788bb4875-lcwgw 1/1 Running 0 2h
zookeeper-0 1/1 Running 0 2h
-> kubectl -n openwhisk describe pod controller-0
Name: controller-0
Namespace: openwhisk
Node: server-f2-0f.ipc3dev.com/10.71.3.8
Start Time: Mon, 08 Oct 2018 04:42:25 -0400
Labels: controller-revision-hash=controller-5bc6758f68
name=controller
statefulset.kubernetes.io/pod-name=controller-0
Annotations: <none>
Status: Pending
IP: 10.244.1.156
Controlled By: StatefulSet/controller
Init Containers:
wait-for-kafka:
Container ID:
docker://6d870e1e10ab5913fd8f48c00f7ee342c7e8e9e0af25a727a15d2229ccb4d00a
Image: busybox
Image ID:
docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1
zookeeper-0.zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok"
]; then result=0; echo "zookeeper returned imok!"; else echo waiting for
zookeeper to be ready; sleep 1; fi done; echo "Zookeeper is up; will wait for
10 seconds to give kafka time to initialize"; sleep 10;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 08 Oct 2018 04:42:37 -0400
Finished: Mon, 08 Oct 2018 04:43:17 -0400
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9qfsx
(ro)
wait-for-couchdb:
Container ID:
docker://08391dc6b3eb76127050c35f0948006a0e398d7b761dc208f83c2b621f4a5c64
Image: busybox
Image ID:
docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd
Port: <none>
Host Port: <none>
Command:
sh
-c
while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider
$READINESS_URL --header="Authorization: Basic
d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then
echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping
3 seconds before retry'; sleep 3; done;
State: Running
Started: Mon, 08 Oct 2018 04:43:19 -0400
Ready: False
Restart Count: 0
Environment:
READINESS_URL:
http://couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9qfsx
(ro)
Containers:
controller:
Container ID:
Image: openwhisk/controller:latest
Image ID:
Ports: 8080/TCP, 2552/TCP, 19999/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/bash
-c
/init.sh `hostname | cut -d'-' -f2`
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: http-get http://:8080/ping delay=5s timeout=1s period=10s
#success=1 #failure=3
Environment:
PORT: 8080
CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config
map 'whisk.config'> Optional: false
CONFIG_whisk_info_buildNo: <set to the key 'whisk_info_buildNo' of
config map 'whisk.config'> Optional: false
JAVA_OPTS: -Xmx1024M
CONTROLLER_OPTS:
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs",
"image": {
"prefix": "openwhisk",
"name": "nodejsaction",
"tag": "latest"
},
"deprecated": true
},
{
"kind": "nodejs:6",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "nodejs6action",
"tag": "latest"
},
"deprecated": false,
"stemCells": [{
"count": 2,
"memory": "256 MB"
}]
},
{
"kind": "nodejs:8",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v8",
"tag": "latest"
},
"deprecated": false
}
],
"python": [
{
"kind": "python",
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:3",
"image": {
"prefix": "openwhisk",
"name": "python3action",
"tag": "latest"
},
"deprecated": false
}
],
"swift": [
{
"kind": "swift:3.1.1",
"image": {
"prefix": "openwhisk",
"name": "action-swift-v3.1.1",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "swift:4.1",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.1",
"tag": "latest"
},
"deprecated": false
}
],
"java": [
{
"kind": "java",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "latest"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"sentinelledLogs": false,
"requireMain": true
}
],
"php": [
{
"kind": "php:7.1",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.1",
"tag": "latest"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "latest"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "latest"
}
]
}
CONFIG_whisk_loadbalancer_invokerUserMemory: 2048m
KAFKA_HOSTS:
kafka.openwhisk.svc.cluster.local:9092
KAFKA_HOST_PORT: 9092
CONFIG_whisk_couchdb_username: <set to the key
'db_username' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_password: <set to the key
'db_password' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_port: <set to the key
'db_port' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_protocol: <set to the key
'db_protocol' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_host:
couchdb.openwhisk.svc.cluster.local
CONFIG_whisk_couchdb_provider: <set to the key
'db_provider' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key
'db_whisk_activations' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key
'db_whisk_actions' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key
'db_whisk_auths' of config map 'db.config'> Optional: false
LIMITS_ACTIONS_SEQUENCE_MAXLENGTH: 50
LIMITS_TRIGGERS_FIRES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_CONCURRENT: 30
CONTROLLER_INSTANCES: 1
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9qfsx
(ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
ow-core-token-9qfsx:
Type: Secret (a volume populated by a Secret)
SecretName: ow-core-token-9qfsx
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
-> kubectl logs controller-0 -n openwhisk
Error from server (BadRequest): container "controller" in pod "controller-0" is
waiting to start: PodInitializing
-> kubectl -n openwhisk describe pod invoker-b5mpt
Name: invoker-b5mpt
Namespace: openwhisk
Node: server-f2-0f.ipc3dev.com/10.71.3.8
Start Time: Mon, 08 Oct 2018 04:42:25 -0400
Labels: controller-revision-hash=137522596
name=invoker
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 10.244.1.154
Controlled By: DaemonSet/invoker
Init Containers:
docker-pull-runtimes:
Container ID:
docker://83cffcd527aea83fcf4f9f1849cea44c9bc3fe017e9eb73d932032dc77e27a25
Image: openwhisk/ansible-runner:latest
Image ID:
docker-pullable://docker.io/openwhisk/ansible-runner@sha256:5c9a2354b872a86ac08cba8fee79ba7452c55377f26761bc156247b9026d20d6
Port: <none>
Host Port: <none>
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 08 Oct 2018 04:47:05 -0400
Finished: Mon, 08 Oct 2018 05:07:13 -0400
Ready: True
Restart Count: 0
Environment:
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs",
"image": {
"prefix": "openwhisk",
"name": "nodejsaction",
"tag": "latest"
},
"deprecated": true
},
{
"kind": "nodejs:6",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "nodejs6action",
"tag": "latest"
},
"deprecated": false,
"stemCells": [{
"count": 2,
"memory": "256 MB"
}]
},
{
"kind": "nodejs:8",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v8",
"tag": "latest"
},
"deprecated": false
}
],
"python": [
{
"kind": "python",
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:3",
"image": {
"prefix": "openwhisk",
"name": "python3action",
"tag": "latest"
},
"deprecated": false
}
],
"swift": [
{
"kind": "swift:3.1.1",
"image": {
"prefix": "openwhisk",
"name": "action-swift-v3.1.1",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "swift:4.1",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.1",
"tag": "latest"
},
"deprecated": false
}
],
"java": [
{
"kind": "java",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "latest"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"sentinelledLogs": false,
"requireMain": true
}
],
"php": [
{
"kind": "php:7.1",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.1",
"tag": "latest"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "latest"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "latest"
}
]
}
Mounts:
/task/playbook.yml from task-dir (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-m4d8z
(ro)
wait-for-controller:
Container ID:
docker://b05867e5c3d1632bbbfa79339f379e43fadc6619e690d24d34fd269fa5de922f
Image: busybox
Image ID:
docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do echo 'Checking controller
readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1; done; echo
'Success: controller is ready'
State: Running
Started: Mon, 08 Oct 2018 05:07:14 -0400
Ready: False
Restart Count: 0
Environment:
READINESS_URL: http://controller.openwhisk.svc.cluster.local:8080/ping
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-m4d8z
(ro)
Containers:
invoker:
Container ID:
Image: openwhisk/invoker:latest
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
/init.sh --uniqueName $INVOKER_NAME
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
PORT: 8080
WHISK_API_HOST_PROTO: <set to the key
'whisk_api_host_proto' of config map 'whisk.config'> Optional: false
WHISK_API_HOST_PORT: <set to the key
'whisk_api_host_port' of config map 'whisk.config'> Optional: false
WHISK_API_HOST_NAME: <set to the key
'whisk_api_host_name' of config map 'whisk.config'> Optional: false
INVOKER_CONTAINER_NETWORK: bridge
CONFIG_whisk_docker_containerFactory_useRunc: false
DOCKER_IMAGE_PREFIX: openwhisk
DOCKER_IMAGE_TAG: latest
DOCKER_REGISTRY:
CONFIG_whisk_loadbalancer_invokerUserMemory: 2048m
INVOKER_NAME: (v1:spec.nodeName)
JAVA_OPTS: -Xmx512M
INVOKER_OPTS:
-Dwhisk.spi.ContainerFactoryProvider=whisk.core.containerpool.docker.DockerContainerFactoryProvider
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs",
"image": {
"prefix": "openwhisk",
"name": "nodejsaction",
"tag": "latest"
},
"deprecated": true
},
{
"kind": "nodejs:6",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "nodejs6action",
"tag": "latest"
},
"deprecated": false,
"stemCells": [{
"count": 2,
"memory": "256 MB"
}]
},
{
"kind": "nodejs:8",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v8",
"tag": "latest"
},
"deprecated": false
}
],
"python": [
{
"kind": "python",
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:3",
"image": {
"prefix": "openwhisk",
"name": "python3action",
"tag": "latest"
},
"deprecated": false
}
],
"swift": [
{
"kind": "swift:3.1.1",
"image": {
"prefix": "openwhisk",
"name": "action-swift-v3.1.1",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "swift:4.1",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.1",
"tag": "latest"
},
"deprecated": false
}
],
"java": [
{
"kind": "java",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "latest"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"sentinelledLogs": false,
"requireMain": true
}
],
"php": [
{
"kind": "php:7.1",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.1",
"tag": "latest"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "latest"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "latest"
}
]
}
WHISK_LOGS_DIR:
CONFIG_whisk_info_date: <set to the key
'whisk_info_date' of config map 'whisk.config'> Optional: false
CONFIG_whisk_couchdb_username: <set to the key
'db_username' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_password: <set to the key
'db_password' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_port: <set to the key
'db_port' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_protocol: <set to the key
'db_protocol' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_host:
couchdb.openwhisk.svc.cluster.local
CONFIG_whisk_couchdb_provider: <set to the key
'db_provider' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key
'db_whisk_activations' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key
'db_whisk_actions' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key
'db_whisk_auths' of config map 'db.config'> Optional: false
KAFKA_HOSTS:
kafka.openwhisk.svc.cluster.local:9092
ZOOKEEPER_HOSTS:
zookeeper-0.zookeeper.openwhisk.svc.cluster.local:2181
Mounts:
/containers from dockerrootdir (rw)
/run/runc from runc (rw)
/sys/fs/cgroup from cgroup (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-m4d8z
(ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
cgroup:
Type: HostPath (bare host directory volume)
Path: /sys/fs/cgroup
HostPathType:
runc:
Type: HostPath (bare host directory volume)
Path: /run/runc
HostPathType:
dockerrootdir:
Type: HostPath (bare host directory volume)
Path: /var/lib/docker/containers
HostPathType:
dockersock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
task-dir:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: docker-pull-runtimes
Optional: false
default-token-m4d8z:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-m4d8z
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events: <none>
-> kubectl logs invoker-b5mpt -n openwhisk
Error from server (BadRequest): container "invoker" in pod "invoker-b5mpt" is
waiting to start: PodInitializing
Please suggest if I am doing something wrong.
[ Full content available at:
https://github.com/apache/incubator-openwhisk-deploy-kube/issues/311 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
