So as others have said, this is not a problem on GCE, shouldn't be a problem with some of the alternative network providers on AWS, and is really only an issue with the default VPC provider on AWS.
I am personally looking at a number of options that may be a half-way house between VPC routing configuration (where AWS imposes the 50 node limit) and the full network providers. For example, setting up a GRE mesh is simple and performs relatively well (though doesn't work on GCE I believe). I also did some experiments and found that IPSEC over UDP performed surprisingly well. IPSEC over UDP is interesting because it's potentially universal: I don't know of any network that doesn't support UDP, and encryption means you aren't assuming a secure network either. I'd love to talk to you about which networking options make the most sense to someone at your scale, Garo. And that goes for anyone else that might have more insight into which networking options make most sense! Also, I investigated further, and AWS only supports layer 2 in a single VPC Subnet. So on AWS we can have Layer 2 networking, or HA clusters, but not both at the same time. Relatedly (if you're not aware of it) Kubernetes added the first pieces of federation in 1.3, and you might consider whether you actually want a few clusters of 256 nodes each (for example). Advantages are that you could span datacenters / providers, and that you're better able to tolerate control plane failures. Justin ( @justinsb on the k8s slack ) On Thursday, July 21, 2016 at 12:46:42 PM UTC-4, Tim Hockin wrote: > > Flannel should work on AWS at that scale. > > Justin (Mr. k8s on AWS) mentioned he was exploring an alternate > solution to the AWS static routes. VPC has an L2 domain doesn't it? > If so, something like Calico should work (no overlay). > > On Thu, Jul 21, 2016 at 6:53 AM, Rodrigo Campos <[email protected] > <javascript:>> wrote: > > I think on gce or gke, you can do this easily. It doesn't use flannel, > etc > > (you can, but is not the default). It uses the Google equivalent of aws > vpc, > > so I guess it doesn't have those limits aws has. In fact, a 1000/2000 > vms > > cluster is used for several blog postson gke and it works just fine. > > > > The aws vpc has the limit, but I'm sure flannel will be an issue. The > coreos > > guys use that, so I'dbe really surprised if it was an issue on a 1000 > vms > > cluster. > > > > So, on gce or gke it should just work. And in aws, it probably should > just > > work if you use coreos, at least. And you can easily install coreos with > > kube-aws, a tool coreos created. > > > > On Thursday, July 21, 2016, Juho Mäkinen <[email protected] > <javascript:>> wrote: > >> > >> I'm evaluating Kubernetes and I'm struggling on finding out any good > >> examples and solutions how Kubernetes can be deployed into AWS so that > the > >> cluster has at least 1000 virtual machines. > >> > >> I have been reading on pretty much all of the suggested networking > layers: > >> flanner, weave, calico and a few others, but they all have some > limitations > >> which I'm worried about: Either their performance is sub-optimal, they > >> suggest using AWS RouteTables (limits the instance count to 50-100), or > they > >> have some other limitations which feels are too restrictive when I'm > aiming > >> for over 1000 virtual machines. > >> > >> I'd like to hear some success stories from other users how they have > built > >> big Kubernetes installations. > >> > >> - Garo > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Containers at Google" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to [email protected] <javascript:>. > >> To post to this group, send email to [email protected] > <javascript:>. > >> Visit this group at https://groups.google.com/group/google-containers. > >> For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Containers at Google" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > To post to this group, send email to [email protected] > <javascript:>. > > Visit this group at https://groups.google.com/group/google-containers. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Containers at Google" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/google-containers. For more options, visit https://groups.google.com/d/optout.
