saggymac commented on issue #272: Controller crash loop URL: https://github.com/apache/incubator-openwhisk-deploy-kube/issues/272#issuecomment-422366086 Sure thing ... ``` whisk: ingress: type: NodePort api_host_name: 172.22.101.111 api_host_port: 31001 nginx: httpsNodePort: 31001 ``` And here's the describe ... ``` $ kubectl describe pod controller-0 --namespace openwhisk Name: controller-0 Namespace: openwhisk Node: node-01/10.0.2.15 Start Time: Mon, 17 Sep 2018 21:57:53 -0400 Labels: controller-revision-hash=controller-5bc6758f68 name=controller statefulset.kubernetes.io/pod-name=controller-0 Annotations: <none> Status: Running IP: 10.42.0.10 Controlled By: StatefulSet/controller Init Containers: wait-for-kafka: Container ID: docker://ccc8317f9cb90f2a4be04122bcd4021d7239c7614814372ddfd360363cd51b49 Image: busybox Image ID: docker-pullable://busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd Port: <none> Host Port: <none> Command: sh -c result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 zookeeper-0.zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi done; echo "Zookeeper is up; will wait for 10 seconds to give kafka time to initialize"; sleep 10; State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 17 Sep 2018 21:58:13 -0400 Finished: Mon, 17 Sep 2018 22:00:08 -0400 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9p9jh (ro) wait-for-couchdb: Container ID: docker://e4cb726c5a380b3a4b8aefbfc9ba614c0969d3e4b4f8fa28e99da4c600f37717 Image: busybox Image ID: docker-pullable://busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd Port: <none> Host Port: <none> Command: sh -c while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done; State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 17 Sep 2018 22:00:09 -0400 Finished: Mon, 17 Sep 2018 22:01:38 -0400 Ready: True Restart Count: 0 Environment: READINESS_URL: http://couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker Mounts: /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9p9jh (ro) Containers: controller: Container ID: docker://8cb3b012e79d2858864d0728e578946a8e60ffb55e551c08dea68435f955290c Image: openwhisk/controller:latest Image ID: docker-pullable://openwhisk/controller@sha256:011f80a25b38e464faf618afbe3cd48e57ad5da5be6bc8a21fea58c09879387e Ports: 8080/TCP, 2552/TCP, 19999/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Command: /bin/bash -c /init.sh `hostname | cut -d'-' -f2` State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 143 Started: Mon, 17 Sep 2018 22:50:00 -0400 Finished: Mon, 17 Sep 2018 22:50:20 -0400 Ready: False Restart Count: 21 Liveness: http-get http://:8080/ping delay=5s timeout=1s period=10s #success=1 #failure=3 Environment: PORT: 8080 CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'whisk.config'> Optional: false CONFIG_whisk_info_buildNo: <set to the key 'whisk_info_buildNo' of config map 'whisk.config'> Optional: false JAVA_OPTS: -Xmx1024M CONTROLLER_OPTS: RUNTIMES_MANIFEST: { "runtimes": { "nodejs": [ { "kind": "nodejs", "image": { "prefix": "openwhisk", "name": "nodejsaction", "tag": "latest" }, "deprecated": true }, { "kind": "nodejs:6", "default": true, "image": { "prefix": "openwhisk", "name": "nodejs6action", "tag": "latest" }, "deprecated": false, "stemCells": [{ "count": 2, "memory": "256 MB" }] }, { "kind": "nodejs:8", "default": false, "image": { "prefix": "openwhisk", "name": "action-nodejs-v8", "tag": "latest" }, "deprecated": false } ], "python": [ { "kind": "python", "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:2", "default": true, "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:3", "image": { "prefix": "openwhisk", "name": "python3action", "tag": "latest" }, "deprecated": false } ], "swift": [ { "kind": "swift:3.1.1", "image": { "prefix": "openwhisk", "name": "action-swift-v3.1.1", "tag": "latest" }, "deprecated": false }, { "kind": "swift:4.1", "default": true, "image": { "prefix": "openwhisk", "name": "action-swift-v4.1", "tag": "latest" }, "deprecated": false } ], "java": [ { "kind": "java", "default": true, "image": { "prefix": "openwhisk", "name": "java8action", "tag": "latest" }, "deprecated": false, "attached": { "attachmentName": "jarfile", "attachmentType": "application/java-archive" }, "sentinelledLogs": false, "requireMain": true } ], "php": [ { "kind": "php:7.1", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-php-v7.1", "tag": "latest" } } ], "ruby": [ { "kind": "ruby:2.5", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-ruby-v2.5", "tag": "latest" } } ] }, "blackboxes": [ { "prefix": "openwhisk", "name": "dockerskeleton", "tag": "latest" } ] } CONFIG_whisk_loadbalancer_invokerUserMemory: 2048m KAFKA_HOSTS: kafka.openwhisk.svc.cluster.local:9092 KAFKA_HOST_PORT: 9092 CONFIG_whisk_couchdb_username: <set to the key 'db_username' in secret 'db.auth'> Optional: false CONFIG_whisk_couchdb_password: <set to the key 'db_password' in secret 'db.auth'> Optional: false CONFIG_whisk_couchdb_port: <set to the key 'db_port' of config map 'db.config'> Optional: false CONFIG_whisk_couchdb_protocol: <set to the key 'db_protocol' of config map 'db.config'> Optional: false CONFIG_whisk_couchdb_host: couchdb.openwhisk.svc.cluster.local CONFIG_whisk_couchdb_provider: <set to the key 'db_provider' of config map 'db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key 'db_whisk_activations' of config map 'db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key 'db_whisk_actions' of config map 'db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key 'db_whisk_auths' of config map 'db.config'> Optional: false LIMITS_ACTIONS_SEQUENCE_MAXLENGTH: 50 LIMITS_TRIGGERS_FIRES_PERMINUTE: 60 LIMITS_ACTIONS_INVOKES_PERMINUTE: 60 LIMITS_ACTIONS_INVOKES_CONCURRENT: 30 CONTROLLER_INSTANCES: 1 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9p9jh (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: ow-core-token-9p9jh: Type: Secret (a volume populated by a Secret) SecretName: ow-core-token-9p9jh Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 10h default-scheduler Successfully assigned controller-0 to node-01 Normal SuccessfulMountVolume 10h kubelet, node-01 MountVolume.SetUp succeeded for volume "ow-core-token-9p9jh" Normal Pulling 10h kubelet, node-01 pulling image "busybox" Normal Pulled 10h kubelet, node-01 Successfully pulled image "busybox" Normal Created 10h kubelet, node-01 Created container Normal Started 10h kubelet, node-01 Started container Normal Pulled 10h kubelet, node-01 Container image "busybox" already present on machine Normal Created 10h kubelet, node-01 Created container Normal Started 10h kubelet, node-01 Started container Normal Created 10h (x2 over 10h) kubelet, node-01 Created container Normal Started 10h (x2 over 10h) kubelet, node-01 Started container Normal Pulled 10h (x3 over 10h) kubelet, node-01 Successfully pulled image "openwhisk/controller:latest" Normal Killing 10h (x2 over 10h) kubelet, node-01 Killing container with id docker://controller:Container failed liveness probe.. Container will be killed and recreated. Warning Unhealthy 10h (x6 over 10h) kubelet, node-01 Liveness probe failed: Get http://10.42.0.10:8080/ping: dial tcp 10.42.0.10:8080: getsockopt: connection refused Normal Pulling 9h (x17 over 10h) kubelet, node-01 pulling image "openwhisk/controller:latest" Warning BackOff 9h (x206 over 9h) kubelet, node-01 Back-off restarting failed container ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
