But OpenShift uses it's own DNS inside isn't it? So has this something to do with my cluster. How do I have to debug/fix this issue? Some more logs
Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.923436 13766 proxier.go:421] Adding new service "default/kubernetes:https" at 172.30.0.1:443/TCP Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.923524 13766 proxier.go:421] Adding new service "default/kubernetes:dns" at 172.30.0.1:53/UDP Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.923552 13766 proxier.go:421] Adding new service "default/kubernetes:dns-tcp" at 172.30.0.1:53/TCP Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.923576 13766 proxier.go:421] Adding new service "default/router:80-tcp" at 172.30.44.225:80/TCP Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.923602 13766 proxier.go:421] Adding new service "default/router:443-tcp" at 172.30.44.225:443/TCP Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.923625 13766 proxier.go:421] Adding new service "default/router:1936-tcp" at 172.30.44.225:1936/TCP Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.923647 13766 proxier.go:421] Adding new service "test/mysql:mysql" at 172.30.222.94:3306/TCP Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.923685 13766 proxier.go:421] Adding new service "test/postgresql-94-centos7:5432-tcp" at 172.30.200.135:5432/TCP Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.923713 13766 proxier.go:421] Adding new service "default/docker-registry:5000-tcp" at 172.30.44.216:5000/TCP Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.923750 13766 proxier.go:565] Not syncing iptables until Services and Endpoints have been received from master Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.926911 13766 proxier.go:484] Setting endpoints for "default/router:80-tcp" to [10.0.0.5:80] Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.926947 13766 proxier.go:484] Setting endpoints for "default/router:443-tcp" to [10.0.0.5:443] Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.926957 13766 proxier.go:484] Setting endpoints for "default/router:1936-tcp" to [10.0.0.5:1936] Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.926968 13766 proxier.go:484] Setting endpoints for "default/docker-registry:5000-tcp" to [10.1.0.3:5000] Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.926978 13766 proxier.go:484] Setting endpoints for "default/kubernetes:dns-tcp" to [178.xx.xx.xx:8053] Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.926986 13766 proxier.go:484] Setting endpoints for "default/kubernetes:https" to [178.xx.xx.xx:8443] Jul 6 19:46:23 node01 origin-node: I0706 19:46:23.926995 13766 proxier.go:484] Setting endpoints for "default/kubernetes:dns" to [178.xx.xx.xx:8053] My public master-server IP: 178.xx.xx.xx My internal node-server IP: 10.0.0.5 Service IP of router: 172.30.44.225 Service IP of registry: 172.30.44.216 From: [email protected] Date: Wed, 6 Jul 2016 13:45:31 -0400 Subject: Re: Unable to connect with service using mysql-ephemeral template To: [email protected]; [email protected] CC: [email protected] On Wed, Jul 6, 2016 at 1:12 PM, Den Cowboy <[email protected]> wrote: I have an older environment on amazon (older images + 1.1.6v) and there it works for mysql: I perform a lookup of my service IP: nslookup 172.30.177.4 Server: 172.30.0.1 Address: 172.30.0.1#53 Non-authoritative answer: 4.177.30.172.in-addr.arpa name = mysql.dev-activiti.svc.cluster.local. Authoritative answers can be found from: But when I perform the same on my environment on OVH (newer version ofcourse) sh-4.2$ nslookup 172.30.222.94 Server: 213.186.33.xx Address: 213.186.33.xx#53 ** server can't find 94.222.30.172.in-addr.arpa.: NXDOMAIN Than it's pointing to a wrong server (not the 172.30... of OpenShift) it sounds like your cluster DNS server/service is not functioning correctly. I didn't saw an issue pending the installation and also /var/log/messages are telling nothing. logs: Jul 6 19:11:17 node01 origin-node: I0706 19:11:17.989899 4926 manager.go:1024] Using docker native exec to run cmd [/bin/sh -i -c MYSQL_PWD="$MYSQL_PASSWORD" mysql -h 127.0.0.1 -u $MYSQL_USER -D $MYSQL_DATABASE -e 'SELECT 1'] inside container {docker b4958c468b643b7ec7dc239569f73e2ea8568b6c6d7e4151cffd621c58db5778} Jul 6 19:11:17 node01 journal: time="2016-07-06T19:11:17.990407769+02:00" level=info msg="{Action=exec, ID=b4958c468b643b7ec7dc239569f73e2ea8568b6c6d7e4151cffd621c58db5778, LoginUID=4294967295, PID=4926}" Jul 6 19:11:17 node01 journal: time="2016-07-06T19:11:17.991021609+02:00" level=info msg="{Action=start, LoginUID=4294967295, PID=4926}" Jul 6 19:11:18 node01 origin-node: I0706 19:11:18.042239 4926 proxier.go:484] Setting endpoints for "test/mysql:mysql" to [10.1.0.2:3306] It keeps showing this log every 10 seconds: sing docker native exec to run cmd [/bin/sh -i -c MYSQL_PWD="$MYSQL_PASSWORD" mysql -h 127.0.0.1 -u $MYSQL_USER -D $MYSQL_DATABASE -e 'SELECT 1'] inside container ... From: [email protected] To: [email protected] Subject: RE: Unable to connect with service using mysql-ephemeral template Date: Wed, 6 Jul 2016 17:03:56 +0000 CC: [email protected] I seem to have the same issue for my postgresdb: nslookup 172.30.200.135 ** server can't find 135.200.30.172.in-addr.arpa.: NXDOMAIN From: [email protected] Date: Wed, 6 Jul 2016 12:53:06 -0400 Subject: Re: Unable to connect with service using mysql-ephemeral template To: [email protected] CC: [email protected] On Wed, Jul 6, 2016 at 12:44 PM, Den Cowboy <[email protected]> wrote: I don't know the best way to check: I was wondering if you had other apps deployed in your cluster that were accessing this, or other services by service hostname. I see this error in my events after the deploy: Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device sh: no job control in this shell ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) I also saw this: https://github.com/openshift/origin/blob/master/docs/debugging-openshift.md I putted also 8.8.8.8 as ns in my /etc/resolv.conf and rebooted but didn't work. Also not after scaling down and up the pod From: [email protected] Date: Wed, 6 Jul 2016 12:31:14 -0400 Subject: Re: Unable to connect with service using mysql-ephemeral template To: [email protected] CC: [email protected] is service hostname resolution otherwise working in your cluster? On Wed, Jul 6, 2016 at 12:20 PM, Den Cowboy <[email protected]> wrote: ping mysql: unknown host mysql nslookup mysql: Server: 213.186.33.xx Address: 213.186.33.xx#53 ** server can't find mysql: NXDOMAIN dig: answer 0 content of /etc/resolv.conf: search test.svc.cluster.local svc.cluster.local cluster.local ovh.net nameserver 178.32.27.xx nameserver 213.186.33.xx options ndots:5 This works fine (IP = service IP): mysql -utest -ptest -h172.30.222.94 ping 172.30.222.94 PING 172.30.222.94 (172.30.222.94) 56(84) bytes of data. From 10.1.0.1 icmp_seq=1 Destination Host Unreachable From 10.1.0.1 icmp_seq=2 Destination Host Unreachable From 10.1.0.1 icmp_seq=3 Destination Host Unreachable From 10.1.0.1 icmp_seq=4 Destination Host Unreachable ~ From: [email protected] Date: Wed, 6 Jul 2016 12:02:20 -0400 Subject: Re: Unable to connect with service using mysql-ephemeral template To: [email protected] CC: [email protected] Is service DNS resolution otherwise working in your cluster? if you just enter the container w/o starting the mysql shell are you able to dig/nslookup/ping the mysql hostname? can you check the /etc/resolv.conf settings within the container to ensure the cluster DNS server is listed? On Wed, Jul 6, 2016 at 11:49 AM, Den Cowboy <[email protected]> wrote: I'm on: oc v1.2.0 kubernetes v1.2.0-36-g4a3f9c5 I've deployed the mysql-template which went fine: Now I've a running mysql container. I go to the terminal inside my webconsole to authenticate on my mysql container: mysql -utest -ptest -h127.0.0.1 > mysql Fine, but when I try my service as my host: mysql -utest -ptest -hmysql Error 2005 (HY000): Unknown MySQL server host 'mysql' (0) My service above my container is called 'mysql' Can someone explain this issue? _______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users -- Ben Parees | OpenShift -- Ben Parees | OpenShift -- Ben Parees | OpenShift _______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users -- Ben Parees | OpenShift
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
