BTW, 172.172.10.0/24 is host subnet. pod network is 172.10.0.0/16. Inside Pod: [root@tulip-saas-xnode-3216045024-ctctp /]# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 172.10.2.1 0.0.0.0 UG 0 0 0 eth0 10.244.0.0 172.10.2.1 255.255.0.0 UG 0 0 0 eth0 172.10.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
On Monday, April 24, 2017 at 9:29:53 PM UTC+8, Roger Song wrote: > > Hi all, > > Kindly help me review this issue. Thanks! > > *[ Description ]* > > *I am newbie to k8s, recently setup k8s cluster on top of CentOS 7.3 with > kubeadm 1.6.1. * > > *Master: k8s* > *Minions: host01, host02, host03* > > *In one of pods(po/tulip-saas-xnode), I tried to make connection from pod > to external rds service(172.172.10.16:3306 <http://172.172.10.16:3306>). > That's the reason I setup service&endpoint "tulip-saas-db2" manually, as > follows:* > =========== > # kubectl get service tulip-saas-db2 -o yaml > apiVersion: v1 > kind: Service > metadata: > creationTimestamp: 2017-04-24T07:46:10Z > name: tulip-saas-db2 > namespace: default > resourceVersion: "905529" > selfLink: /api/v1/namespaces/default/services/tulip-saas-db2 > uid: 153b3520-28c2-11e7-a272-000c29235036 > spec: > clusterIP: 10.111.128.117 > ports: > - port: 3306 > protocol: TCP > targetPort: 3306 > sessionAffinity: None > type: ClusterIP > status: > loadBalancer: {} > > # kubectl get endpoints tulip-saas-db2 -o yaml > apiVersion: v1 > kind: Endpoints > metadata: > creationTimestamp: 2017-04-24T07:46:10Z > name: tulip-saas-db2 > namespace: default > resourceVersion: "905533" > selfLink: /api/v1/namespaces/default/endpoints/tulip-saas-db2 > uid: 15552d0d-28c2-11e7-a272-000c29235036 > subsets: > - addresses: > - ip: 172.172.10.16 > ports: > - port: 3306 > protocol: TCP > > ========== > > > *I'm able to connect to the port in host02 OS via cluster IP. * > ========== > [root@host02 .kube]# iptables-save | grep tulip-saas-db2 > -A KUBE-SEP-TS2EMOGZXA7V27BD -s 172.172.10.16/32 -m comment --comment > "default/tulip-saas-db2:" -j KUBE-MARK-MASQ > -A KUBE-SEP-TS2EMOGZXA7V27BD -p tcp -m comment --comment > "default/tulip-saas-db2:" -m tcp -j DNAT --to-destination > 172.172.10.16:3306 > -A KUBE-SERVICES ! -s 172.10.0.0/16 -d 10.111.128.117/32 -p tcp -m > comment --comment "default/tulip-saas-db2: cluster IP" -m tcp --dport 3306 > -j KUBE-MARK-MASQ > -A KUBE-SERVICES -d 10.111.128.117/32 -p tcp -m comment --comment > "default/tulip-saas-db2: cluster IP" -m tcp --dport 3306 -j > KUBE-SVC-ASAFJW2B6372ZEVA > -A KUBE-SVC-ASAFJW2B6372ZEVA -m comment --comment > "default/tulip-saas-db2:" -j KUBE-SEP-TS2EMOGZXA7V27BD > [root@host02 .kube]# telnet 10.111.128.117 3306 > Trying 10.111.128.117... > Connected to 10.111.128.117. > Escape character is '^]'. > ===== > > *The pod is running on host02* > ==== > # kubectl get pod --all-namespaces -o wide | grep tulip > default tulip-saas-xnode-3216045024-ctctp 1/1 > Running 1 8h 172.10.2.22 host02.corp.mooit.net > ==== > > *Inside the pod, service name can be resolved. However, I'm not able to > connect to the port. * > === > [root@tulip-saas-xnode-3216045024-ctctp /]# nslookup tulip-saas-db2 > Server: 10.96.0.10 > Address: 10.96.0.10#53 > > Name: tulip-saas-db2.default.svc.cluster.local > Address: 10.111.128.117 > > [root@tulip-saas-xnode-3216045024-ctctp /]# telnet tulip-saas-db2 3306 > Trying 10.111.128.117... > ^C > === > > *kube-proxy logs in host02 doesn't give any message. I tried to delete the > pod, and let DS create it again, no lucky. No remarkable messages in > /var/log/message.* > ====== > [root@k8s manifests]# kubectl logs -f po/kube-proxy-p279k -n kube-system > I0424 12:37:24.220402 1 server.go:225] Using iptables Proxier. > I0424 12:37:24.301205 1 server.go:249] Tearing down userspace rules. > I0424 12:37:24.433983 1 conntrack.go:81] Set sysctl > 'net/netfilter/nf_conntrack_max' to 131072 > I0424 12:37:24.435683 1 conntrack.go:66] Setting conntrack hashsize > to 32768 > I0424 12:37:24.436164 1 conntrack.go:81] Set sysctl > 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400 > I0424 12:37:24.436217 1 conntrack.go:81] Set sysctl > 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600 > ======= > > *I tried to > follow > https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ > <https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/>, > but failed to understand in which layer the problem takes place. firewalld > is disabled in all nodes. * > > *Had tried some RBAC stuff for kube-proxy account referring > to https://github.com/uruddarraju/kubernetes-rbac-policies > <https://github.com/uruddarraju/kubernetes-rbac-policies>* > > > *[ env ] * > # kubeadm version > kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", > GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", > BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", > Platform:"linux/amd64"} > > # kubectl get all --all-namespaces > NAMESPACE NAME READY > STATUS RESTARTS AGE > default po/busybox 1/1 > Running 1 3h > default po/http-svc-zzj1q 1/1 > Running 1 4d > default po/nginx-deployment-4234284026-04wb3 1/1 > Running 2 3d > default po/nginx-deployment-4234284026-pdvml 1/1 > Running 1 3d > default po/tulip-saas-xnode-3216045024-ctctp 1/1 > Running 1 8h > kube-system po/default-http-backend-2198840601-7wdbk 1/1 > Running 2 4d > kube-system po/etcd-k8s.corp.mooit.net 1/1 > Running 11 7d > kube-system po/kube-apiserver-k8s.corp.mooit.net 1/1 > Running 6 6d > kube-system po/kube-controller-manager-k8s.corp.mooit.net 1/1 > Running 13 7d > kube-system po/kube-dns-3913472980-mtml5 3/3 > Running 96 7d > kube-system po/kube-flannel-ds-57crg 2/2 > Running 7 7d > kube-system po/kube-flannel-ds-bn66x 2/2 > Running 4 7d > kube-system po/kube-flannel-ds-wxj4d 2/2 > Running 3 7d > kube-system po/kube-flannel-ds-xk9wh 2/2 > Running 56 7d > kube-system po/kube-proxy-mp6xr 1/1 > Running 10 7d > kube-system po/kube-proxy-p279k 1/1 > Running 0 7m > kube-system po/kube-proxy-qqdvd 1/1 > Running 2 7d > kube-system po/kube-proxy-vjmnw 1/1 > Running 1 7d > kube-system po/kube-scheduler-k8s.corp.mooit.net 1/1 > Running 13 7d > kube-system po/kubernetes-dashboard-915795657-wf3fp 1/1 > Running 2 6d > kube-system po/nginx-ingress-lb-0q6n8 1/1 > Running 1 2d > kube-system po/nginx-ingress-lb-20km8 1/1 > Running 2 2d > kube-system po/nginx-ingress-lb-fk7nd 1/1 > Running 1 2d > kube-system po/nginx-ingress-lb-q0z4c 1/1 > Running 1 2d > > NAMESPACE NAME DESIRED CURRENT READY AGE > default rc/http-svc 1 1 1 4d > > NAMESPACE NAME CLUSTER-IP EXTERNAL-IP > PORT(S) AGE > default svc/http-svc 10.109.111.193 <nodes> > 80:30301/TCP 4d > default svc/kubernetes 10.96.0.1 <none> > 443/TCP 7d > default svc/nginx-svc 10.105.48.156 <nodes> > 80:30302/TCP 3d > default svc/tulip-saas-db2 10.111.128.117 <none> > 3306/TCP 5h > default svc/tulip-saas-xnode 10.106.241.164 <nodes> > 80:30189/TCP 1d > kube-system svc/default-http-backend 10.98.17.92 <none> > 80/TCP 4d > kube-system svc/kube-dns 10.96.0.10 <none> > 53/UDP,53/TCP 7d > kube-system svc/kubernetes-dashboard 10.106.75.115 <nodes> > 80:32416/TCP 7d > > NAMESPACE NAME DESIRED CURRENT UP-TO-DATE > AVAILABLE AGE > default deploy/nginx-deployment 2 2 2 > 2 3d > default deploy/tulip-saas-xnode 1 1 1 > 1 8h > kube-system deploy/default-http-backend 1 1 1 > 1 4d > kube-system deploy/kube-dns 1 1 1 > 1 7d > kube-system deploy/kubernetes-dashboard 1 1 1 > 1 7d > > NAMESPACE NAME DESIRED CURRENT > READY AGE > default rs/nginx-deployment-4234284026 2 2 2 > 3d > default rs/tulip-saas-xnode-3216045024 1 1 1 > 8h > kube-system rs/default-http-backend-2198840601 1 1 1 > 4d > kube-system rs/kube-dns-3913472980 1 1 1 > 7d > kube-system rs/kubernetes-dashboard-915795657 1 1 1 > 6d > [root@k8s manifests]# kubectl get ds --all-namespaces > NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE > AVAILABLE NODE-SELECTOR AGE > kube-system kube-flannel-ds 4 4 4 4 > 4 beta.kubernetes.io/arch=amd64 7d > kube-system kube-proxy 4 4 4 4 > 4 <none> 7d > kube-system nginx-ingress-lb 4 4 4 4 > 4 <none> 2d > > # kubectl get ing --all-namespaces > NAMESPACE NAME HOSTS ADDRESS > PORTS AGE > default ng k8s.corp.mooit.net 172.172.10.23... > 80, 443 3d > default test-http k8s.corp.mooit.net 172.172.10.23... > 80, 443 3d > default tulip-saas-xnode xnode.svr.mooit.net 172.172.10.23... > 80, 443 1d > kube-system dashboard k8s.corp.mooit.net 172.172.10.23... > 80, 443 3d > > > > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
