Thanks for the feedback. I see I didn't quite understand k8s networking properly (and had my cluster misconfigured as a result).

I now have it configured as:

--cluster-cidr=10.240.0.0/12
--service-cluster-ip-range=10.128.0.0/16

And I'm deducing that the /12 in the cluster-cidr is what would then allow this cluster to go beyond 256 nodes.


One other point about the networking I'm a little confused about that I'd like to clarify: it seems that IP's in the cluster-cidr range (i.e., service endpoints) are reachable from any host that is on the flannel network, while IP's in the service-cluster-ip-range (i.e., services) are only reachable from the worker nodes in the cluster.

So, for example, I have a k8s setup with 4 machines: a master, 2 worker nodes, and a "driver" machine. All 4 machines are on the flannel network. I have a nginx service defined like so:

$ kubectl get svc nginx; kubectl get ep nginx
NAME      CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
nginx     10.128.105.78   <nodes>       80:30207/TCP   2d
NAME      ENDPOINTS                       AGE
nginx     10.240.14.5:80,10.240.27.2:80   2d


Now "curl 10.128.105.78" only succeeds on the 2 worker node machines, while "curl 10.240.14.5" succeeds on all 4.

I'm guessing this is expected / makes sense, since 10.240.0.0/12 addresses are accessible to any machine on the flannel network, whereas 10.128.0.0/16 addresses can only be reached via iptables rules - i.e., only accessible on machines running kube-proxy, aka the worker nodes.

Again I guess this is makes sense in retrospect. But I was a bit surprised when I first saw this, as I had thought that services' cluster IP's would be reachable from all machines. (Or at least from the master too.)

Perhaps you could confirm that I'm understanding this all correctly. (And have my cluster configured correctly?)

Thanks,

DR

On 2017-08-11 11:26 am, Matthias Rampke wrote:
Oh hold on. the _service cluster IP range_ is not for pod IPs at all.
It's for the ClusterIP of services, so you can have up to 64k services
in a cluster at the default setting. The range for pods is the
--cluster-cidr flag on kube-controller-manager.

On Fri, Aug 11, 2017 at 3:05 PM David Rosenstrauch <dar...@darose.net>
wrote:

Actually, that begs another question.  The docs also specify that
k8s
can support up to 5000 nodes.  But I'm not clear on how the
networking
can support that.

So let's go back to that service-cluster-ip-range with the /16 CIDR.
That only supports a maximum of 256 nodes.

Now the maximum size for the service-cluster-ip-range appears to be
/12
- e.g., --service-cluster-ip-range=10.240.0.0/12 [1]  (Beyond that
you get a
"Specified service-cluster-ip-range is too large" error.)  So that
means
12 bits for the high part of the address, and with each node taking
the
lower 8 bits for the IP address of individual pods, that leaves 12
remaining bits worth of unique IP address ranges.  12 bits = 4095
possible IP addresses for nodes.  How then could anyone scale up to
5000
nodes?

DR

On 2017-08-11 10:47 am, David Rosenstrauch wrote:
Ah.  That makes a bit more sense.

Thanks!

DR

On 2017-08-11 10:41 am, Ben Kochie wrote:
Kuberentes will be giving a /24 to each node, not each pod.  Each
node
will give one IP out of that /24 to a pod it controls.  This
default
means you can have 253 pods-per-node.  This of course can be
adjust
depending on the size of your pods and nodes.

This means that you can fully utilize the /16 for pods (minus
per-node
network, broadcast, gateway)

On Fri, Aug 11, 2017 at 4:36 PM, David Rosenstrauch
<dar...@darose.net> wrote:

According to the docs, k8s can support systems of up to 150000
pods.
(See https://kubernetes.io/docs/admin/cluster-large/ [1])  But
given k8s' networking model, I'm a bit puzzled on how that would
work.

It seems like a typical setup is to assign a
service-cluster-ip-range with a /16 CIDR.  (Say 10.254.0.0/16
[2] [2])
However, I notice that my cluster assigns a full /24 IP range to
each pod that it creates.  (E.g., pod1 gets 10.254.1.*, pod2
gets
10.254.2.*, etc.)  Given this networking setup, it would seem
that
Kubernetes would only be capable of launching a maximum of 256
pods.

Am I misunderstanding how k8s works in this r?  Or is it that
the
networking would need to be configured differently to support
more
than 256 pods?

Thanks,

DR

--
You received this message because you are subscribed to the
Google
Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from
it,
send an email to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to
kubernetes-users@googlegroups.com.
Visit this group at
https://groups.google.com/group/kubernetes-users
[3].
For more options, visit https://groups.google.com/d/optout [4].

--
You received this message because you are subscribed to the
Google
Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to
kubernetes-users@googlegroups.com.
Visit this group at
https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.


Links:
------
[1] https://kubernetes.io/docs/admin/cluster-large/
[2] http://10.254.0.0/16
[3] https://groups.google.com/group/kubernetes-users
[4] https://groups.google.com/d/optout

--
You received this message because you are subscribed to the Google
Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to
kubernetes-users@googlegroups.com.
Visit this group at
https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

 --
You received this message because you are subscribed to the Google
Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to
kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.


Links:
------
[1] http://10.240.0.0/12
[2] http://10.254.0.0/16

--
You received this message because you are subscribed to the Google Groups "Kubernetes 
user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

Reply via email to