saurabh1284 commented on issue #2264: Unable to connect to the controller docker: Controller docker not starting URL: https://github.com/apache/incubator-openwhisk/issues/2264#issuecomment-313165119 @rabbah . Tried again redeployment, but still getting the same issue. Steps performed -->> > Cleaned all running container (except registry and nginx) as below == -> docker ps --format "{{.Names}} - {{.Image}} - {{.Ports}}" nginx - nginx:1.11 - 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8443->8443/tcp whisk_docker_registry - registry:2.3.1 - 0.0.0.0:5000->5000/tcp > Ran following ansible playbooks == -> ansible-playbook -i environments/distributed couchdb.yml -> ansible-playbook -i environments/distributed initdb.yml -> ansible-playbook -i environments/distributed wipe.yml << All above playbook ran without any Error >> -> ansible-playbook -i environments/distributed openwhisk.yml It failed with Error >> TASK [controller : wait until the Controller in this host is up and running] ********************************* Wednesday 05 July 2017 15:59:56 +0000 (0:01:04.168) 0:34:19.560 ******** FAILED - RETRYING: wait until the Controller in this host is up and running (12 retries left). FAILED - RETRYING: wait until the Controller in this host is up and running (11 retries left). FAILED - RETRYING: wait until the Controller in this host is up and running (10 retries left). FAILED - RETRYING: wait until the Controller in this host is up and running (9 retries left). FAILED - RETRYING: wait until the Controller in this host is up and running (8 retries left). FAILED - RETRYING: wait until the Controller in this host is up and running (7 retries left). FAILED - RETRYING: wait until the Controller in this host is up and running (6 retries left). FAILED - RETRYING: wait until the Controller in this host is up and running (5 retries left). FAILED - RETRYING: wait until the Controller in this host is up and running (4 retries left). FAILED - RETRYING: wait until the Controller in this host is up and running (3 retries left). FAILED - RETRYING: wait until the Controller in this host is up and running (2 retries left). FAILED - RETRYING: wait until the Controller in this host is up and running (1 retries left). fatal: [10.11.12.24]: FAILED! => {"attempts": 12, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://10.11.12.24:10001/ping"} PLAY RECAP *************************************************************************************************** 10.11.12.24 : ok=25 changed=20 unreachable=0 failed=1 10.11.12.25 : ok=4 changed=2 unreachable=0 failed=0 ansible : ok=2 changed=1 unreachable=0 failed=0 Wednesday 05 July 2017 16:07:32 +0000 (0:07:35.596) 0:41:55.233 ******** =============================================================================== controller : wait until the Controller in this host is up and running - 455.69s kafka : wait until the kafka server started up ------------------------ 316.73s kafka : create the health topic --------------------------------------- 260.57s kafka : create the active-ack topics ---------------------------------- 196.70s kafka : create the invoker topics ------------------------------------- 193.12s kafka : wait until the Zookeeper in this host is up and running ------- 122.22s consul : fill consul kv ----------------------------------------------- 104.41s Gathering Facts -------------------------------------------------------- 94.08s Gathering Facts -------------------------------------------------------- 84.56s kafka : (re)start zookeeper -------------------------------------------- 66.54s controller : (re)start controller -------------------------------------- 64.17s consul : pull the consul:0.7.0 image ----------------------------------- 59.16s Gathering Facts -------------------------------------------------------- 52.96s consul : (re)start consul server/agent --------------------------------- 46.37s consul : copy template from local to remote in consul config directory -- 43.64s kafka : (re)start kafka ------------------------------------------------ 37.65s consul : start registrator using docker cli ---------------------------- 36.61s controller : pull the latest image of controller ----------------------- 32.88s kafka : pull the zookeeper:3.4 image ----------------------------------- 32.54s Gathering Facts -------------------------------------------------------- 30.18s ============================== ## Additional information you deem important: -> docker ps --format "{{.Names}} - {{.Image}} - {{.Ports}}" controller0 - 10.11.12.24:5000/whisk/controller:latest - 0.0.0.0:10001->8080/tcp kafka - ches/kafka:0.10.2.1 - 7203/tcp, 0.0.0.0:9092->9092/tcp zookeeper - zookeeper:3.4 - 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp registrator - gliderlabs/registrator - consul - consul:0.7.0 - 0.0.0.0:8300-8302->8300-8302/tcp, 0.0.0.0:8400->8400/tcp, 0.0.0.0:8301-8302->8301-8302/udp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->8600/udp, 8600/tcp couchdb - couchdb:1.6 - 0.0.0.0:5984->5984/tcp nginx - nginx:1.11 - 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8443->8443/tcp whisk_docker_registry - registry:2.3.1 - 0.0.0.0:5000->5000/tcp -> ps -eaf | grep docker root 1091 1 5 14:58 ? 00:06:00 /usr/bin/dockerd -H tcp://10.11.12.24:4243 -H unix:///var/run docker.sock --storage-driver=aufs --insecure-registry 10.11.12.24:5000 --raw-logs root 1159 1091 2 14:58 ? 00:02:38 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run docker/libcontainerd/containerd --runtime docker-runc . . -> telnet 10.11.12.24 4243 Trying 10.11.12.24... Connected to 10.11.12.24. Escape character is '^]'. ^C Connection closed by foreign host. -> curl http://10.11.12.24:4243/info {"ID":"VK7F:GMEU:K6W4:3K5Y:ELVI:AIAG:LRW3:2N5T:K7PN:W5OR:MN5J:5KBA","Containers":8,"ContainersRunning":7,"ContainersPaused":0,"ContainersStopped":1,"Images":134,"Driver":"aufs","DriverStatus":[["Root Dir","/var/lib/docker/aufs"],["Backing Filesystem","extfs"],["Dirs","176"],["Dirperm1 Supported","true"]],"SystemStatus":null,"Plugins":{"Volume":["local"],"Network":["bridge","overlay","host","null"],"Authorization":null},"MemoryLimit":true,"SwapLimit":false,"KernelMemory":true,"CpuCfsPeriod":true,"CpuCfsQuota":true,"CPUShares":true,"CPUSet":true,"IPv4Forwarding":true,"BridgeNfIptables":true,"BridgeNfIp6tables":true,"Debug":false,"NFd":45,"OomKillDisable":true,"NGoroutines":72,"SystemTime":"2017-07-05T16:52:19.604149476Z","ExecutionDriver":"","LoggingDriver":"json-file","CgroupDriver":"cgroupfs","NEventsListener":1,"KernelVersion":"4.4.0-31-generic","OperatingSystem":"Ubuntu 14.04.5 LTS","OSType":"linux","Architecture":"x86_64","IndexServerAddress":"https://index.docker.io/v1/","Regi stryConfig":{"InsecureRegistryCIDRs":["127.0.0.0/8"],"IndexConfigs":{"10.11.12.24:5000":{"Name":"10.11.12.24:5000","Mirrors":[],"Secure":false,"Official":false},"docker.io":{"Name":"docker.io","Mirrors":null,"Secure":true,"Official":true}},"Mirrors":null},"NCPU":4,"MemTotal":8371515392,"DockerRootDir":"/var/lib/docker","HttpProxy":"","HttpsProxy":"","NoProxy":"","Name":"skg-faas-infra","Labels":null,"ExperimentalBuild":false,"ServerVersion":"1.12.0","ClusterStore":"","ClusterAdvertise":"","SecurityOptions":["apparmor"],"Runtimes":{"runc":{"path":"docker-runc"}},"DefaultRuntime":"runc","Swarm":{"NodeID":"","NodeAddr":"","LocalNodeState":"inactive","ControlAvailable":false,"Error":"","RemoteManagers":null,"Nodes":0,"Managers":0,"Cluster":{"ID":"","Version":{},"CreatedAt":"0001-01-01T00:00:00Z","UpdatedAt":"0001-01-01T00:00:00Z","Spec":{"Orchestration":{},"Raft":{},"Dispatcher":{},"CAConfig":{},"TaskDefaults":{}}}},"LiveRestoreEnabled":false} -> netstat -aptn Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 10.11.12.24:4243 0.0.0.0:* LISTEN 1091/dockerd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1372/sshd tcp 0 0 10.11.12.24:55964 10.11.12.24:4243 TIME_WAIT - tcp 0 0 172.17.0.1:48284 172.17.0.8:9092 ESTABLISHED 7134/docker-proxy tcp 0 0 10.11.12.24:22 10.20.2.4:58544 ESTABLISHED 1815/sshd: ubuntu [ tcp 0 0 10.11.12.24:22 10.20.2.4:58592 ESTABLISHED 1837/sshd: ubuntu [ tcp 0 0 172.17.0.1:48332 172.17.0.8:9092 ESTABLISHED 7134/docker-proxy tcp 0 0 172.17.0.1:48336 172.17.0.8:9092 ESTABLISHED 7134/docker-proxy tcp 0 0 10.11.12.24:22 10.20.2.4:33246 ESTABLISHED 2690/sshd: ubuntu [ tcp 0 0 172.17.0.1:48428 172.17.0.8:9092 ESTABLISHED 7134/docker-proxy tcp 0 0 172.17.0.1:48280 172.17.0.8:9092 ESTABLISHED 7134/docker-proxy tcp 0 0 172.17.0.1:49428 172.17.0.5:8500 ESTABLISHED 5897/docker-proxy tcp6 0 0 :::10001 :::* LISTEN 9648/docker-proxy tcp6 0 0 :::8500 :::* LISTEN 5897/docker-proxy tcp6 0 0 :::22 :::* LISTEN 1372/sshd tcp6 0 0 :::443 :::* LISTEN 1720/docker-proxy tcp6 0 0 :::8443 :::* LISTEN 1710/docker-proxy tcp6 0 0 :::5984 :::* LISTEN 3121/docker-proxy tcp6 0 0 :::9092 :::* LISTEN 7134/docker-proxy tcp6 0 0 :::2181 :::* LISTEN 6799/docker-proxy tcp6 0 0 :::5000 :::* LISTEN 1739/docker-proxy tcp6 0 0 :::8300 :::* LISTEN 5955/docker-proxy tcp6 0 0 :::8301 :::* LISTEN 5935/docker-proxy tcp6 0 0 :::8302 :::* LISTEN 5917/docker-proxy tcp6 0 0 :::8400 :::* LISTEN 5908/docker-proxy tcp6 0 0 :::80 :::* LISTEN 1729/docker-proxy tcp6 0 0 10.11.12.24:9092 172.17.0.9:60928 ESTABLISHED 7134/docker-proxy tcp6 0 0 10.11.12.24:9092 172.17.0.9:60980 ESTABLISHED 7134/docker-proxy tcp6 0 0 10.11.12.24:10001 172.17.0.5:41858 TIME_WAIT - tcp6 0 0 10.11.12.24:9092 172.17.0.9:60976 ESTABLISHED 7134/docker-proxy tcp6 0 0 10.11.12.24:8500 172.17.0.6:49122 ESTABLISHED 5897/docker-proxy tcp6 0 0 10.11.12.24:9092 172.17.0.9:60924 ESTABLISHED 7134/docker-proxy tcp6 0 0 10.11.12.24:9092 172.17.0.9:32840 ESTABLISHED 7134/docker-proxy -> curl http://10.11.12.24:10001/ping curl: (7) Failed to connect to 10.11.12.24 port 10001: Connection refused -> cat /tmp/wsklogs/controller0/controller0_logs.log . . [2017-07-05T16:07:23.645Z] [INFO] Slf4jLogger started [2017-07-05T16:08:27.321Z] [INFO] Slf4jLogger started [2017-07-05T16:08:47.422Z] [INFO] [??] [Config] environment set value for consulserver.host [2017-07-05T16:08:47.596Z] [INFO] [??] [Config] environment set value for whisk.version.date [2017-07-05T16:08:47.608Z] [INFO] [??] [Config] environment set value for whisk.version.buildno [2017-07-05T16:08:47.620Z] [INFO] [??] [Config] environment set value for consul.host.port4 [2017-07-05T16:08:47.628Z] [INFO] [??] [Config] environment set value for port [2017-07-05T16:08:47.652Z] [INFO] [??] [WhiskConfig] reading properties from consul at 10.11.12.24:8500 [2017-07-05T16:10:29.082Z] [WARN] [??] [WhiskConfig] failed to read properties from consul: Futures timed out after [1 minute] [2017-07-05T16:10:30.297Z] [ERROR] [??] **[Config] required property db.port still not set** [2017-07-05T16:10:30.441Z] [ERROR] [??] **[Controller] Bad configuration, cannot start.** [2017-07-05T16:11:31.111Z] [INFO] Slf4jLogger started [2017-07-05T16:11:47.210Z] [INFO] [??] [Config] environment set value for consulserver.host [2017-07-05T16:11:47.518Z] [INFO] [??] [Config] environment set value for whisk.version.date [2017-07-05T16:11:47.611Z] [INFO] [??] [Config] environment set value for whisk.version.buildno [2017-07-05T16:11:47.616Z] [INFO] [??] [Config] environment set value for consul.host.port4 [2017-07-05T16:11:47.764Z] [INFO] [??] [Config] environment set value for port [2017-07-05T16:11:47.769Z] [INFO] [??] [WhiskConfig] reading properties from consul at 10.11.12.24:8500 [2017-07-05T16:13:21.270Z] [WARN] [??] [WhiskConfig] failed to read properties from consul: Futures timed out after [1 minute] [2017-07-05T16:13:22.227Z] **[ERROR] [??] [Config] required property db.port still not set** [2017-07-05T16:13:22.307Z] **[ERROR] [??] [Controller] Bad configuration, cannot start.** [2017-07-05T16:14:29.321Z] [INFO] Slf4jLogger started [2017-07-05T16:14:47.801Z] [INFO] [??] [Config] environment set value for consulserver.host [2017-07-05T16:14:47.988Z] [INFO] [??] [Config] environment set value for whisk.version.date [2017-07-05T16:14:48.006Z] [INFO] [??] [Config] environment set value for whisk.version.buildno [2017-07-05T16:14:48.016Z] [INFO] [??] [Config] environment set value for consul.host.port4 [2017-07-05T16:14:48.020Z] [INFO] [??] [Config] environment set value for port [2017-07-05T16:14:48.052Z] [INFO] [??] [WhiskConfig] reading properties from consul at 10.11.12.24:8500 [2017-07-05T16:16:37.235Z] [INFO] [#sid_130] [Controller] starting controller instance 0 [marker:controller_startup0_count:110630] [2017-07-05T16:16:39.753Z] [INFO] [??] [LoadBalancerService] blackboxFraction = 0.1 [2017-07-05T16:17:03.289Z] [INFO] [#sid_120] [WhiskAction] [GET] serving from datastore: id: whisk.system/invokerHealthTestAction [marker:database_cacheMiss_count:137033] . . ======= Please suggest. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
With regards, Apache Git Services
