nikolavukovic opened a new issue #577: Kafka, Controller and Invoker pods stuck in Init phase, CouchDB not initialized properly URL: https://github.com/apache/openwhisk-deploy-kube/issues/577 I have set up a 2 nodes cluster (master and one worker) using Vagrant following technical requirements written here. I am trying to install Openwhisk using Helm 3. Kafka, Controller and Invoker pods are stuck in Init phase and I can't find the problem using the logs. I will put some information that could be useful for you. ``` kubectl get all -n openwhisk -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/owdev-alarmprovider-686b9796f7-f9hp9 0/1 Init:0/1 0 12m 10.32.0.5 worker <none> <none> pod/owdev-apigateway-6b5686f4c9-dwbl4 1/1 Running 0 12m 10.32.0.14 worker <none> <none> pod/owdev-cloudantprovider-7445484996-tj2z5 0/1 Init:0/1 0 12m 10.32.0.4 worker <none> <none> pod/owdev-controller-0 0/1 Init:0/2 0 12m 10.32.0.17 worker <none> <none> pod/owdev-couchdb-59cc777bbc-jb6gd 1/1 Running 0 12m 10.32.0.13 worker <none> <none> pod/owdev-gen-certs-4rzzl 0/1 Completed 0 12m 10.32.0.9 worker <none> <none> pod/owdev-init-couchdb-xlbch 1/1 Running 0 12m 10.32.0.10 worker <none> <none> pod/owdev-install-packages-7kz5v 0/1 Init:0/1 0 12m 10.32.0.15 worker <none> <none> pod/owdev-invoker-0 0/1 Init:0/1 0 12m 10.32.0.16 worker <none> <none> pod/owdev-kafka-0 0/1 Init:0/1 0 12m 10.32.0.6 worker <none> <none> pod/owdev-kafkaprovider-bf858d897-g4vqt 0/1 Init:0/1 0 12m 10.32.0.8 worker <none> <none> pod/owdev-nginx-8495fdcb9c-htbb2 1/1 Running 0 12m 10.32.0.18 worker <none> <none> pod/owdev-redis-75674fd487-nzns7 1/1 Running 0 12m 10.32.0.11 worker <none> <none> pod/owdev-wskadmin 1/1 Running 0 12m 10.32.0.3 worker <none> <none> pod/owdev-zookeeper-0 1/1 Running 0 12m 10.32.0.12 worker <none> <none> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/owdev-apigateway ClusterIP 10.106.233.102 <none> 8080/TCP,9000/TCP 12m name=owdev-apigateway service/owdev-controller ClusterIP 10.98.99.59 <none> 8080/TCP 12m name=owdev-controller service/owdev-couchdb ClusterIP 10.108.235.142 <none> 5984/TCP 12m name=owdev-couchdb service/owdev-kafka ClusterIP None <none> 9092/TCP 12m name=owdev-kafka service/owdev-nginx NodePort 10.106.198.121 <none> 80:32511/TCP,443:31001/TCP 12m name=owdev-nginx service/owdev-redis ClusterIP 10.100.249.168 <none> 6379/TCP 12m name=owdev-redis service/owdev-zookeeper ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 12m name=owdev-zookeeper NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/owdev-alarmprovider 0/1 1 0 12m alarmprovider openwhisk/alarmprovider:2.0.0-incubating name=owdev-alarmprovider deployment.apps/owdev-apigateway 1/1 1 1 12m apigateway openwhisk/apigateway:0.11.0 name=owdev-apigateway deployment.apps/owdev-cloudantprovider 0/1 1 0 12m cloudantprovider openwhisk/cloudantprovider:2.0.0-incubating name=owdev-cloudantprovider deployment.apps/owdev-couchdb 1/1 1 1 12m couchdb apache/couchdb:2.3 name=owdev-couchdb deployment.apps/owdev-kafkaprovider 0/1 1 0 12m kafkaprovider openwhisk/kafkaprovider:2.0.0-incubating name=owdev-kafkaprovider deployment.apps/owdev-nginx 1/1 1 1 12m nginx nginx:1.17 name=owdev-nginx deployment.apps/owdev-redis 1/1 1 1 12m redis redis:4.0 name=owdev-redis NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/owdev-alarmprovider-686b9796f7 1 1 0 12m alarmprovider openwhisk/alarmprovider:2.0.0-incubating name=owdev-alarmprovider,pod-template-hash=686b9796f7replicaset.apps/owdev-apigateway-6b5686f4c9 1 1 1 12m apigateway openwhisk/apigateway:0.11.0 name=owdev-apigateway,pod-template-hash=6b5686f4c9 replicaset.apps/owdev-cloudantprovider-7445484996 1 1 0 12m cloudantprovider openwhisk/cloudantprovider:2.0.0-incubating name=owdev-cloudantprovider,pod-template-hash=7445484996 replicaset.apps/owdev-couchdb-59cc777bbc 1 1 1 12m couchdb apache/couchdb:2.3 name=owdev-couchdb,pod-template-hash=59cc777bbc replicaset.apps/owdev-kafkaprovider-bf858d897 1 1 0 12m kafkaprovider openwhisk/kafkaprovider:2.0.0-incubating name=owdev-kafkaprovider,pod-template-hash=bf858d897 replicaset.apps/owdev-nginx-8495fdcb9c 1 1 1 12m nginx nginx:1.17 name=owdev-nginx,pod-template-hash=8495fdcb9c replicaset.apps/owdev-redis-75674fd487 1 1 1 12m redis redis:4.0 name=owdev-redis,pod-template-hash=75674fd487 NAME READY AGE CONTAINERS IMAGES statefulset.apps/owdev-controller 0/1 12m controller openwhisk/controller:8eb922f statefulset.apps/owdev-invoker 0/1 12m invoker openwhisk/invoker:8eb922f statefulset.apps/owdev-kafka 0/1 12m kafka wurstmeister/kafka:0.11.0.1 statefulset.apps/owdev-zookeeper 1/1 12m zookeeper zookeeper:3.4 NAME COMPLETIONS DURATION AGE CONTAINERS IMAGES SELECTOR job.batch/owdev-gen-certs 1/1 12s 12m gen-certs openwhisk/ow-utils:8eb922f controller-uid=d09ee120-728a-4e9a-8922-904639f7f1dd job.batch/owdev-init-couchdb 0/1 12m 12m init-couchdb openwhisk/ow-utils:8eb922f controller-uid=27c47d17-6701-4254-b4b6-dd0b90acbaba job.batch/owdev-install-packages 0/1 12m 12m install-packages openwhisk/ow-utils:8eb922f controller-uid=58aaddb9-2016-4499-9706-7ec4ff7922ad ``` Shortened version of invoker ``` kubectl describe pod -n openwhisk owdev-invoker-0 Name: owdev-invoker-0 Namespace: openwhisk Priority: 0 Node: worker/172.42.42.101 Start Time: Fri, 24 Jan 2020 10:59:50 +0000 Labels: app=owdev-openwhisk chart=openwhisk-0.1.10 controller-revision-hash=owdev-invoker-6c8967fdb5 heritage=Helm name=owdev-invoker release=owdev statefulset.kubernetes.io/pod-name=owdev-invoker-0 Annotations: <none> Status: Pending IP: 10.32.0.16 IPs: IP: 10.32.0.16 Controlled By: StatefulSet/owdev-invoker Init Containers: wait-for-controller: Container ID: docker://72979b7d13ea6e2e34dea3d33c79165e1e9d328bc1b9ecb321afba99ff2bf8b8 Image: busybox Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a Port: <none> Host Port: <none> Command: sh -c result=1; until [ $result -eq 0 ]; do echo 'Checking controller readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1; done; echo 'Success: controller is ready' State: Running Started: Fri, 24 Jan 2020 10:59:57 +0000 Ready: False Restart Count: 0 Environment: READINESS_URL: http://owdev-controller.openwhisk.svc.kubernetes.default.svc.cluster.local:8080/ping Mounts: /var/run/secrets/kubernetes.io/serviceaccount from owdev-invoker-token-kjv8m (ro) Containers: invoker: Container ID: Image: openwhisk/invoker:8eb922f Image ID: Port: 8080/TCP Host Port: 0/TCP Command: /bin/bash -c /init.sh --uniqueName $INVOKER_NAME State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: PORT: 8080 WHISK_API_HOST_PROTO: <set to the key 'whisk_api_host_proto' of config map 'owdev-whisk.config'> Optional: false WHISK_API_HOST_PORT: <set to the key 'whisk_api_host_port' of config map 'owdev-whisk.config'> Optional: false WHISK_API_HOST_NAME: <set to the key 'whisk_api_host_name' of config map 'owdev-whisk.config'> Optional: false CONFIG_whisk_docker_containerFactory_useRunc: false CONFIG_whisk_containerPool_userMemory: 2048m CONFIG_whisk_containerFactory_containerArgs_network: bridge CONFIG_whisk_containerFactory_containerArgs_extraEnvVars_0: __OW_ALLOW_CONCURRENT=false CONFIG_whisk_containerFactory_runtimesRegistry_url: INVOKER_NAME: owdev-invoker-0 (v1:metadata.name) JAVA_OPTS: -Xmx512M INVOKER_OPTS: -Dkubernetes.master=https://$KUBERNETES_SERVICE_HOST -Dwhisk.spi.ContainerFactoryProvider=org.apache.openwhisk.core.containerpool.kubernetes.KubernetesContainerFactoryProvider RUNTIMES_MANIFEST: { .................................... ZOOKEEPER_HOSTS: owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local:2181 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from owdev-invoker-token-kjv8m (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: owdev-invoker-token-kjv8m: Type: Secret (a volume populated by a Secret) SecretName: owdev-invoker-token-kjv8m Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s openwhisk-role=invoker:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 23m default-scheduler Successfully assigned openwhisk/owdev-invoker-0 to worker Normal Pulled 23m kubelet, worker Container image "busybox" already present on machine Normal Created 23m kubelet, worker Created container wait-for-controller Normal Started 23m kubelet, worker Started container wait-for-controller ``` Shortened version of controller ``` kubectl describe pod -n openwhisk owdev-controller-0 Name: owdev-controller-0 Namespace: openwhisk Priority: 0 Node: worker/172.42.42.101 Start Time: Fri, 24 Jan 2020 10:59:50 +0000 Labels: app=owdev-openwhisk chart=openwhisk-0.1.10 controller-revision-hash=owdev-controller-65c995d96c heritage=Helm name=owdev-controller release=owdev statefulset.kubernetes.io/pod-name=owdev-controller-0 Annotations: <none> Status: Pending IP: 10.32.0.17 IPs: IP: 10.32.0.17 Controlled By: StatefulSet/owdev-controller Init Containers: wait-for-kafka: Container ID: docker://a1bde85e6c8d402203964f9e65be57ccd5c1714c8b1c5e26022552e252b91442 Image: busybox Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a Port: <none> Host Port: <none> Command: sh -c result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi done; echo "Zookeeper is up; will wait for 10 seconds to give kafka time to initialize"; sleep 10; State: Running Started: Fri, 24 Jan 2020 10:59:57 +0000 Ready: False Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-2qh55 (ro) wait-for-couchdb: Container ID: Image: busybox Image ID: Port: <none> Host Port: <none> Command: sh -c while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done; State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: READINESS_URL: http://owdev-couchdb.openwhisk.svc.kubernetes.default.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker Mounts: /var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-2qh55 (ro) Containers: controller: Container ID: Image: openwhisk/controller:8eb922f Image ID: Ports: 8080/TCP, 2552/TCP, 19999/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Command: /bin/bash -c /init.sh `hostname | awk -F '-' '{print $NF}'` State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Liveness: http-get http://:8080/ping delay=10s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/ping delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: PORT: 8080 TZ: UTC CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'owdev-whisk.config'> Optional: false CONFIG_whisk_info_buildNo: <set to the key 'whisk_info_buildNo' of config map 'owdev-whisk.config'> Optional: false JAVA_OPTS: -Xmx1024M CONTROLLER_OPTS: RUNTIMES_MANIFEST: { ........................................................ CONFIG_whisk_kafka_topics_events_segmentBytes: CONFIG_whisk_kafka_topics_health_retentionBytes: CONFIG_whisk_kafka_topics_health_retentionMs: CONFIG_whisk_kafka_topics_health_segmentBytes: CONFIG_whisk_kafka_topics_invoker_retentionBytes: CONFIG_whisk_kafka_topics_invoker_retentionMs: CONFIG_whisk_kafka_topics_invoker_segmentBytes: CONFIG_whisk_couchdb_username: <set to the key 'db_username' in secret 'owdev-db.auth'> Optional: false CONFIG_whisk_couchdb_password: <set to the key 'db_password' in secret 'owdev-db.auth'> Optional: false CONFIG_whisk_couchdb_port: <set to the key 'db_port' of config map 'owdev-db.config'> Optional: false CONFIG_whisk_couchdb_protocol: <set to the key 'db_protocol' of config map 'owdev-db.config'> Optional: false CONFIG_whisk_couchdb_host: owdev-couchdb.openwhisk.svc.kubernetes.default.svc.cluster.local CONFIG_whisk_couchdb_provider: <set to the key 'db_provider' of config map 'owdev-db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key 'db_whisk_activations' of config map 'owdev-db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key 'db_whisk_actions' of config map 'owdev-db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key 'db_whisk_auths' of config map 'owdev-db.config'> Optional: false CONTROLLER_INSTANCES: 1 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-2qh55 (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: owdev-core-token-2qh55: Type: Secret (a volume populated by a Secret) SecretName: owdev-core-token-2qh55 Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s openwhisk-role=core:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 26m default-scheduler Successfully assigned openwhisk/owdev-controller-0 to worker Warning FailedMount 26m kubelet, worker MountVolume.SetUp failed for volume "owdev-core-token-2qh55" : failed to sync secret cache: timed out waiting for the condition Normal Pulled 26m kubelet, worker Container image "busybox" already present on machine Normal Created 26m kubelet, worker Created container wait-for-kafka Normal Started 26m kubelet, worker Started container wait-for-kafka ``` ``` kubectl describe pod -n openwhisk owdev-kafka-0 Name: owdev-kafka-0 Namespace: openwhisk Priority: 0 Node: worker/172.42.42.101 Start Time: Fri, 24 Jan 2020 10:59:50 +0000 Labels: app=owdev-openwhisk chart=openwhisk-0.1.10 controller-revision-hash=owdev-kafka-847cd555bf heritage=Helm name=owdev-kafka release=owdev statefulset.kubernetes.io/pod-name=owdev-kafka-0 Annotations: <none> Status: Pending IP: 10.32.0.6 IPs: IP: 10.32.0.6 Controlled By: StatefulSet/owdev-kafka Init Containers: wait-for-zookeeper: Container ID: docker://4b9db4e302ddf91200721d537b150c4b28b5942c4a4a71ea9adfe5eba2162881 Image: busybox Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a Port: <none> Host Port: <none> Command: sh -c result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi; done; echo "Success: zookeeper is up" State: Running Started: Fri, 24 Jan 2020 10:59:56 +0000 Ready: False Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro) Containers: kafka: Container ID: Image: wurstmeister/kafka:0.11.0.1 Image ID: Port: 9092/TCP Host Port: 0/TCP Command: /bin/bash -c export KAFKA_BROKER_ID=`hostname | awk -F '-' '{print $NF}'`; export KAFKA_ADVERTISED_HOST_NAME=`hostname -f`; /usr/bin/start-kafka.sh State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: KAFKA_ADVERTISED_PORT: 9092 KAFKA_ZOOKEEPER_CONNECT: owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local:2181 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: default-token-bjt6q: Type: Secret (a volume populated by a Secret) SecretName: default-token-bjt6q Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s openwhisk-role=core:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 28m default-scheduler Successfully assigned openwhisk/owdev-kafka-0 to worker Normal Pulled 27m kubelet, worker Container image "busybox" already present on machine Normal Created 27m kubelet, worker Created container wait-for-zookeeper Normal Started 27m kubelet, worker Started container wait-for-zookeeper ``` ``` kubectl describe pod -n openwhisk owdev-kafkaprovider-bf858d897-g4vqt Name: owdev-kafkaprovider-bf858d897-g4vqt Namespace: openwhisk Priority: 0 Node: worker/172.42.42.101 Start Time: Fri, 24 Jan 2020 10:59:50 +0000 Labels: app=owdev-openwhisk chart=openwhisk-0.1.10 heritage=Helm name=owdev-kafkaprovider pod-template-hash=bf858d897 release=owdev Annotations: <none> Status: Pending IP: 10.32.0.8 IPs: IP: 10.32.0.8 Controlled By: ReplicaSet/owdev-kafkaprovider-bf858d897 Init Containers: wait-for-controller: Container ID: docker://6139e487e23f903f1275f60e9296fc42c763918c2cb14d48d1ddc713f8a4d38c Image: busybox Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a Port: <none> Host Port: <none> Command: sh -c result=1; until [ $result -eq 0 ]; do echo 'Checking controller readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1; done; echo 'Success: controller is ready' State: Running Started: Fri, 24 Jan 2020 10:59:56 +0000 Ready: False Restart Count: 0 Environment: READINESS_URL: http://owdev-controller.openwhisk.svc.kubernetes.default.svc.cluster.local:8080/ping Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro) Containers: kafkaprovider: Container ID: Image: openwhisk/kafkaprovider:2.0.0-incubating Image ID: Port: 8080/TCP Host Port: 0/TCP State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: DB_URL: <set to the key 'db_url' of config map 'owdev-db.config'> Optional: false DB_USER: <set to the key 'db_username' in secret 'owdev-db.auth'> Optional: false DB_PASS: <set to the key 'db_password' in secret 'owdev-db.auth'> Optional: false DB_PREFIX: kp LOCAL_DEV: true ROUTER_HOST: <set to the key 'whisk_api_host_nameAndPort' of config map 'owdev-whisk.config'> Optional: false ENDPOINT_AUTH: <set to the key 'whisk_api_host_nameAndPort' of config map 'owdev-whisk.config'> Optional: false Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: default-token-bjt6q: Type: Secret (a volume populated by a Secret) SecretName: default-token-bjt6q Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 28m default-scheduler Successfully assigned openwhisk/owdev-kafkaprovider-bf858d897-g4vqt to worker Normal Pulled 28m kubelet, worker Container image "busybox" already present on machine Normal Created 28m kubelet, worker Created container wait-for-controller Normal Started 28m kubelet, worker Started container wait-for-controller ``` ``` kubectl describe pod -n openwhisk owdev-zookeeper-0 Name: owdev-zookeeper-0 Namespace: openwhisk Priority: 0 Node: worker/172.42.42.101 Start Time: Fri, 24 Jan 2020 10:59:50 +0000 Labels: app=owdev-openwhisk chart=openwhisk-0.1.10 controller-revision-hash=owdev-zookeeper-6bc7d5cf48 heritage=Helm name=owdev-zookeeper release=owdev statefulset.kubernetes.io/pod-name=owdev-zookeeper-0 Annotations: <none> Status: Running IP: 10.32.0.12 IPs: IP: 10.32.0.12 Controlled By: StatefulSet/owdev-zookeeper Containers: zookeeper: Container ID: docker://93ee6f00abab5ec6bc16a78c6c9808a87d1a33f5ffb9b9f1bcca393236432c49 Image: zookeeper:3.4 Image ID: docker-pullable://zookeeper@sha256:491427fc9f788c168e096422afe620fdb269a5d604efd11f953682919101c658 Ports: 2181/TCP, 2888/TCP, 3888/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Command: /bin/bash -c hostname -s | awk -F '-' '{print $NF}'> /data/myid; cat /data/myid; cat /conf/zoo.cfg; zkServer.sh start-foreground State: Running Started: Fri, 24 Jan 2020 10:59:57 +0000 Ready: True Restart Count: 0 Liveness: tcp-socket :2181 delay=5s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [/bin/bash -c echo ruok | nc -w 1 localhost 2181 | grep imok] delay=5s timeout=1s period=10s #success=1 #failure=3 Environment: <none> Mounts: /conf from zk-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: zk-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: owdev-zookeeper Optional: false default-token-bjt6q: Type: Secret (a volume populated by a Secret) SecretName: default-token-bjt6q Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s openwhisk-role=core:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 29m default-scheduler Successfully assigned openwhisk/owdev-zookeeper-0 to worker Normal Pulled 29m kubelet, worker Container image "zookeeper:3.4" already present on machine Normal Created 29m kubelet, worker Created container zookeeper Normal Started 29m kubelet, worker Started container zookeeper ``` What I understand from this is that invoker is waiting for controller, controller is waiting for kafka, kafka is waiting for zookeeper. There are docker containers running on worker executing commands printed here. There is another problem with couchdb pods. They are running but the DB is not correctly initialized. Following are the logs ``` kubectl logs -n openwhisk owdev-init-couchdb-xlbch Cloning into '/openwhisk'... /openwhisk / Note: checking out '8eb922f568d7843cdb1068252d29ba5723b30283'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b <new-branch-name> HEAD is now at 8eb922f5 Separate test cases for the logs and results APIs of activation (#4777) / /openwhisk/ansible / [WARNING]: Unable to parse /openwhisk/ansible/environments/local as an inventory source [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* Friday 24 January 2020 11:00:44 +0000 (0:00:00.275) 0:00:00.275 ******** ok: [localhost] TASK [gen hosts if 'local' env is used] **************************************** Friday 24 January 2020 11:00:46 +0000 (0:00:01.673) 0:00:01.950 ******** changed: [localhost -> localhost] TASK [find the ip of docker-machine] ******************************************* Friday 24 January 2020 11:00:46 +0000 (0:00:00.865) 0:00:02.816 ******** skipping: [localhost] TASK [get the docker-machine ip] *********************************************** Friday 24 January 2020 11:00:47 +0000 (0:00:00.043) 0:00:02.860 ******** skipping: [localhost] TASK [gen hosts for docker-machine] ******************************************** Friday 24 January 2020 11:00:47 +0000 (0:00:00.060) 0:00:02.921 ******** skipping: [localhost] TASK [gen hosts for Jenkins] *************************************************** Friday 24 January 2020 11:00:47 +0000 (0:00:00.059) 0:00:02.980 ******** skipping: [localhost] TASK [check if db_local.ini exists?] ******************************************* Friday 24 January 2020 11:00:47 +0000 (0:00:00.054) 0:00:03.035 ******** ok: [localhost] TASK [prepare db_local.ini] **************************************************** Friday 24 January 2020 11:00:47 +0000 (0:00:00.335) 0:00:03.371 ******** changed: [localhost -> localhost] TASK [gen untrusted server certificate for host] ******************************* Friday 24 January 2020 11:00:48 +0000 (0:00:00.512) 0:00:03.884 ******** changed: [localhost -> localhost] TASK [gen untrusted client certificate for host] ******************************* Friday 24 January 2020 11:00:49 +0000 (0:00:00.957) 0:00:04.842 ******** changed: [localhost -> localhost] TASK [clean up old kafka keystore] ********************************************* Friday 24 January 2020 11:00:49 +0000 (0:00:00.965) 0:00:05.808 ******** skipping: [localhost] TASK [ensure kafka files directory exists] ************************************* Friday 24 January 2020 11:00:50 +0000 (0:00:00.052) 0:00:05.861 ******** skipping: [localhost] TASK [generate kafka certificates] ********************************************* Friday 24 January 2020 11:00:50 +0000 (0:00:00.078) 0:00:05.940 ******** skipping: [localhost] TASK [ensure controller files directory exists] ******************************** Friday 24 January 2020 11:00:50 +0000 (0:00:00.047) 0:00:05.987 ******** changed: [localhost] TASK [generate controller certificates] **************************************** Friday 24 January 2020 11:00:51 +0000 (0:00:00.975) 0:00:06.963 ******** changed: [localhost -> localhost] TASK [ensure invoker files directory exists] *********************************** Friday 24 January 2020 11:00:52 +0000 (0:00:01.161) 0:00:08.125 ******** changed: [localhost] TASK [generate invoker certificates] ******************************************* Friday 24 January 2020 11:00:52 +0000 (0:00:00.692) 0:00:08.818 ******** changed: [localhost -> localhost] PLAY RECAP ********************************************************************* localhost : ok=10 changed=8 unreachable=0 failed=0 Friday 24 January 2020 11:00:54 +0000 (0:00:01.083) 0:00:09.901 ******** =============================================================================== Gathering Facts --------------------------------------------------------- 1.68s generate controller certificates ---------------------------------------- 1.16s generate invoker certificates ------------------------------------------- 1.08s ensure controller files directory exists -------------------------------- 0.98s gen untrusted client certificate for host ------------------------------- 0.97s gen untrusted server certificate for host ------------------------------- 0.96s gen hosts if 'local' env is used ---------------------------------------- 0.87s ensure invoker files directory exists ----------------------------------- 0.69s prepare db_local.ini ---------------------------------------------------- 0.51s check if db_local.ini exists? ------------------------------------------- 0.34s ensure kafka files directory exists ------------------------------------- 0.08s get the docker-machine ip ----------------------------------------------- 0.06s gen hosts for docker-machine -------------------------------------------- 0.06s gen hosts for Jenkins --------------------------------------------------- 0.06s clean up old kafka keystore --------------------------------------------- 0.05s generate kafka certificates --------------------------------------------- 0.05s find the ip of docker-machine ------------------------------------------- 0.04s PLAY [db] ********************************************************************** TASK [Gathering Facts] ********************************************************* Friday 24 January 2020 11:00:56 +0000 (0:00:00.269) 0:00:00.269 ******** ok: [172.17.0.1] PLAY RECAP ********************************************************************* 172.17.0.1 : ok=1 changed=0 unreachable=0 failed=0 Friday 24 January 2020 11:00:58 +0000 (0:00:01.591) 0:00:01.861 ******** =============================================================================== Gathering Facts --------------------------------------------------------- 1.59s / waiting for CouchDB to be available waiting for CouchDB to be available waiting for CouchDB to be available waiting for CouchDB to be available waiting for CouchDB to be available waiting for CouchDB to be available waiting for CouchDB to be available ... ``` ``` kubectl logs -n openwhisk owdev-couchdb-59cc777bbc-jb6gd [info] 2020-01-24T11:00:00.198305Z couchdb@couchdb0 <0.9.0> -------- Application couch_log started on node couchdb@couchdb0 [info] 2020-01-24T11:00:00.213370Z couchdb@couchdb0 <0.9.0> -------- Application folsom started on node couchdb@couchdb0 [info] 2020-01-24T11:00:00.388557Z couchdb@couchdb0 <0.9.0> -------- Application couch_stats started on node couchdb@couchdb0 [info] 2020-01-24T11:00:00.389234Z couchdb@couchdb0 <0.9.0> -------- Application khash started on node couchdb@couchdb0 [info] 2020-01-24T11:00:00.412502Z couchdb@couchdb0 <0.9.0> -------- Application couch_event started on node couchdb@couchdb0 [info] 2020-01-24T11:00:00.413146Z couchdb@couchdb0 <0.9.0> -------- Application hyper started on node couchdb@couchdb0 [info] 2020-01-24T11:00:00.430790Z couchdb@couchdb0 <0.9.0> -------- Application ibrowse started on node couchdb@couchdb0 [info] 2020-01-24T11:00:00.453061Z couchdb@couchdb0 <0.9.0> -------- Application ioq started on node couchdb@couchdb0 [info] 2020-01-24T11:00:00.454319Z couchdb@couchdb0 <0.9.0> -------- Application mochiweb started on node couchdb@couchdb0 [info] 2020-01-24T11:00:00.491742Z couchdb@couchdb0 <0.212.0> -------- Apache CouchDB 2.3.1 is starting. [info] 2020-01-24T11:00:00.491926Z couchdb@couchdb0 <0.213.0> -------- Starting couch_sup [notice] 2020-01-24T11:00:00.541999Z couchdb@couchdb0 <0.96.0> -------- config: [features] pluggable-storage-engines set to true for reason nil [notice] 2020-01-24T11:00:00.566597Z couchdb@couchdb0 <0.96.0> -------- config: [admins] whisk_admin set to -pbkdf2-3a1c30b3e9965585f6d0cd9ff907ede232f0847f,2f06a1ea99722b3696d260d440f791c4,10 for reason nil [notice] 2020-01-24T11:00:00.654765Z couchdb@couchdb0 <0.96.0> -------- config: [couchdb] uuid set to 3a9c7e0f5a9b50a69616239d1619cc6d for reason nil [info] 2020-01-24T11:00:00.779773Z couchdb@couchdb0 <0.218.0> -------- open_result error {not_found,no_db_file} for _users [info] 2020-01-24T11:00:00.933696Z couchdb@couchdb0 <0.212.0> -------- Apache CouchDB has started. Time to relax. [info] 2020-01-24T11:00:00.933868Z couchdb@couchdb0 <0.212.0> -------- Apache CouchDB has started on http://any:5986/ [info] 2020-01-24T11:00:00.934031Z couchdb@couchdb0 <0.9.0> -------- Application couch started on node couchdb@couchdb0 [info] 2020-01-24T11:00:00.934156Z couchdb@couchdb0 <0.9.0> -------- Application ets_lru started on node couchdb@couchdb0 [notice] 2020-01-24T11:00:01.028698Z couchdb@couchdb0 <0.285.0> -------- rexi_server : started servers [notice] 2020-01-24T11:00:01.039627Z couchdb@couchdb0 <0.289.0> -------- rexi_buffer : started servers [info] 2020-01-24T11:00:01.040398Z couchdb@couchdb0 <0.9.0> -------- Application rexi started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.060380Z couchdb@couchdb0 <0.218.0> -------- open_result error {not_found,no_db_file} for _nodes [warning] 2020-01-24T11:00:01.060443Z couchdb@couchdb0 <0.297.0> -------- creating missing database: _nodes [info] 2020-01-24T11:00:01.156452Z couchdb@couchdb0 <0.218.0> -------- open_result error {not_found,no_db_file} for _dbs [warning] 2020-01-24T11:00:01.156534Z couchdb@couchdb0 <0.311.0> -------- creating missing database: _dbs [info] 2020-01-24T11:00:01.165833Z couchdb@couchdb0 <0.9.0> -------- Application mem3 started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.166078Z couchdb@couchdb0 <0.9.0> -------- Application fabric started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.202113Z couchdb@couchdb0 <0.9.0> -------- Application chttpd started on node couchdb@couchdb0 [error] 2020-01-24T11:00:01.226620Z couchdb@couchdb0 emulator -------- Error in process <0.351.0> on node couchdb@couchdb0 with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]} [notice] 2020-01-24T11:00:01.226735Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142) [info] 2020-01-24T11:00:01.238657Z couchdb@couchdb0 <0.9.0> -------- Application couch_index started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.238715Z couchdb@couchdb0 <0.9.0> -------- Application couch_mrview started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.238775Z couchdb@couchdb0 <0.9.0> -------- Application couch_plugins started on node couchdb@couchdb0 [notice] 2020-01-24T11:00:01.287124Z couchdb@couchdb0 <0.96.0> -------- config: [features] scheduler set to true for reason nil [info] 2020-01-24T11:00:01.316012Z couchdb@couchdb0 <0.218.0> -------- open_result error {not_found,no_db_file} for _replicator [notice] 2020-01-24T11:00:01.319811Z couchdb@couchdb0 <0.368.0> -------- creating replicator ddoc <<"_replicator">> [info] 2020-01-24T11:00:01.331007Z couchdb@couchdb0 <0.9.0> -------- Application couch_replicator started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.350148Z couchdb@couchdb0 <0.9.0> -------- Application couch_peruser started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.374999Z couchdb@couchdb0 <0.9.0> -------- Application ddoc_cache started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.397739Z couchdb@couchdb0 <0.9.0> -------- Application global_changes started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.397811Z couchdb@couchdb0 <0.9.0> -------- Application jiffy started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.407443Z couchdb@couchdb0 <0.9.0> -------- Application mango started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.416278Z couchdb@couchdb0 <0.9.0> -------- Application setup started on node couchdb@couchdb0 [info] 2020-01-24T11:00:01.416325Z couchdb@couchdb0 <0.9.0> -------- Application snappy started on node couchdb@couchdb0 [notice] 2020-01-24T11:00:06.227704Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142) [error] 2020-01-24T11:00:06.228333Z couchdb@couchdb0 emulator -------- Error in process <0.490.0> on node couchdb@couchdb0 with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]} [notice] 2020-01-24T11:00:06.254012Z couchdb@couchdb0 <0.372.0> -------- couch_replicator_clustering : cluster stable [notice] 2020-01-24T11:00:06.261084Z couchdb@couchdb0 <0.389.0> -------- Started replicator db changes listener <0.491.0> [error] 2020-01-24T11:00:11.228985Z couchdb@couchdb0 emulator -------- Error in process <0.590.0> on node couchdb@couchdb0 with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]} [notice] 2020-01-24T11:00:11.229059Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142) [notice] 2020-01-24T11:00:16.029104Z couchdb@couchdb0 <0.289.0> -------- rexi_buffer : cluster stable [notice] 2020-01-24T11:00:16.029228Z couchdb@couchdb0 <0.285.0> -------- rexi_server : cluster stable [error] 2020-01-24T11:00:16.231300Z couchdb@couchdb0 emulator -------- Error in process <0.655.0> on node couchdb@couchdb0 with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]} [notice] 2020-01-24T11:00:16.231504Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142) [error] 2020-01-24T11:00:21.232975Z couchdb@couchdb0 emulator -------- Error in process <0.752.0> on node couchdb@couchdb0 with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]} [notice] 2020-01-24T11:00:21.233053Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142) [notice] 2020-01-24T11:00:26.235354Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142) [error] 2020-01-24T11:00:26.235514Z couchdb@couchdb0 emulator -------- Error in process <0.817.0> on node couchdb@couchdb0 with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]} ... ``` I tried setting an external CouchDB and putting new values in mycluster.yaml to redirect Openwhisk to it. CouchDB pods were not initialized and I guess that is a sign of normal execution. Anyway I would like to solve this problem with DB and with other pods. Can you please help? If you need any other logs just tell.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
