Re: [kubernetes-users] Generally speaking, separate apps = separate clusters, right?
Yeah, exactly, co-locating for us means the pods that sit on the same nodes have network access to each other. We aren't using any dynamic overlay networks. We are excited about Envoy/Istio and its service mesh concepts, so that access between pods is uniform. EJ On Sun, Aug 13, 2017 at 1:02 PM wrote: > EJ, > > Awesome info and I believe I follow the breakdown. For the record, we're > are nowhere near the scale you're working with - machines or employees :-). > > Just one last clarifying question about what you mean by co-locating. I > assume that means all the workloads (pods) within a BU's cluster are > allowed network-level access to each other. Service access is restricted > others ways (e.g. athenz), if necessary. Is that right? > > Best, > Terence > > On Saturday, August 12, 2017 at 4:21:52 PM UTC-7, EJ Campbell wrote: > > We have an internal auth mechanism that fronts our services and has also > been integrated with RBAC into K8s, which protects almost all our services: > https://github.com/yahoo/athenz > > > > > > Network / Host level ACL's (automated iptables) are used to prevent one > cluster from hitting another for our most sensitive systems (and for legacy > workloads). > > > > > > To your point below, we did have to setup more course grained acl's > (which is the minority of systems that don't support athenz) when moving to > kuberetes from static data center deployments where workloads don't move. > Kubernetes labeling is used to ensure workloads only run on nodes we know > have the proper network acl's. > > > > > > Our business units are pretty broad. We have about 100+ separate > services from about 30 teams deployed in one K8s cluster in one data > center. The rough rule has been if we give our central ops team in the BU > sudo across workloads, we are okay with those workloads co-locating. > > > > > > I do think the more fine grained isolation the public cloud can support > (especially with cluster auto-scaling) provides some advantages over the > model we settled on inside our data center. For us, it is nice to really > let K8s and our operators bin-pack our workloads optimally by having a > larger number of pods and nodes to work with. > > > > > > EJ > > > > > > On Sat, Aug 12, 2017 at 12:42 PM wrote: > > EJ, > > > > > > > > Thanks for the insight into how you're handling your clusters! That's > very helpful. To your question, we are running in one of the public clouds > (AWS). > > > > > > > > Since you group clusters by business unit, I have a question about how > you handle your network security. Is it the case that: > > > > > > > > (1) All applications deployed by a business unit can share a common set > of network egress rules for traffic going outside the cluster, and all pods > deployed by a business unit are allowed network access to each other. > > > > > > > > (2) Applications deployed by a business unit have varying security > requirements and not all containers deployed by a business unit can have > network access to all the others. > > > > > > > > If (1) is true, then I would assume that each business unit deploys what > I would consider an "application" in our world and you're more or less > doing what we're hoping to set up. > > > > > > > > If (2) is true, then I'm curious how you're handling the security part. > Are you using a cluster network solution, like calico, for ingress/egress > rules? Are you using a traditional network solution, then ensuring certain > pods are only scheduled on nodes that will be subject to those rules? Are > you doing something else entirely? > > > > > > > > Thanks in advance! > > > > Terence > > > > > > > > > > > > On Friday, August 11, 2017 at 8:12:56 PM UTC-7, EJ Campbell wrote: > > > > > Curious if you are running your setup within a physical data center or > in one of the public clouds? > > > > > > > > > > > > > > > Where I am, we group clusters by large-scale business unit (many > applications deployed on behalf of hundreds of engineers) and run three > clusters per physical data center: canary, "corp" (special network access) > and prod. RBAC is used for authorization among those hundreds of engineers, > with cluster admin access for production engineers within that business > unit. > > > > > > > > > > > > > > > The separate clusters per data center and canary cluster within data > center keeps the "blast radius" small during deployments and chef keeps > things manageable between clusters. > > > > > > > > > > > > > > > EJ > > > > > > > > > > > > > > > On Fri, Aug 11, 2017 at 6:42 PM wrote: > > > > > Tim, > > > > > > > > > > > > > > > > > > > > This is a hugely appreciated answer. Sorry the question was so generic! > > > > > > > > > > > > > > > > > > > > For now, we'll chose separate clusters - understanding the sub-optimal > side effects of that decision. We'll revisit the single-or-multiple cluster > question in a year or so, once we've had a reasonable amount of experience > running everything on it. I'll bet the work required
Re: [kubernetes-users] Generally speaking, separate apps = separate clusters, right?
On Sunday, August 13, 2017 at 12:32:24 AM UTC-7, Timo Reimann wrote: > Somewhat of a middle ground solution might be to create "sub-clusters" using > taints and tolerations -- that is, have dedicated nodes for dedicated > application classes or workloads inside a given cluster. It may reduce the > general overhead inherent to managing separate clusters while still being > able to leverage a lot of the built-in Kubernetes tooling. Timo, I'm glad you brought that up, that's a varation of what we were debating originally. However we would accomplish it, taints + tolerations, node selectors, or node affinities, we're basically forcing k8s to only run certain pods on certain nodes. Extactly as you mentioned, this would be a middle ground approach instead of creating separate clusters. It would neatly solve the problem I was concerned about with applying different egress rules to different pods. It would even provide some fencing, so really bad misbehaviors caused by an app (e.g. network saturation) can only effect so many nodes. Controlling which nodes a pod can be deployed on wouldn't solve our cluster network security needs (pod-to-pod security). But, of course, I've seen there are new-ish NetworkPolicies and freely available providers (e.g. calico) to account for that. The sticky part for me, and the reason for the quesiton here, was "what's the best idea in k8s today?". I can absolutely see handling deployments in either way and the most common way I've seen people describe their installations was using a cluster per application. Being new to k8s, I'm keen on following as many well-worn paths as possible for our initial deployments :-). -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
Re: [kubernetes-users] Generally speaking, separate apps = separate clusters, right?
EJ, Awesome info and I believe I follow the breakdown. For the record, we're are nowhere near the scale you're working with - machines or employees :-). Just one last clarifying question about what you mean by co-locating. I assume that means all the workloads (pods) within a BU's cluster are allowed network-level access to each other. Service access is restricted others ways (e.g. athenz), if necessary. Is that right? Best, Terence On Saturday, August 12, 2017 at 4:21:52 PM UTC-7, EJ Campbell wrote: > We have an internal auth mechanism that fronts our services and has also been > integrated with RBAC into K8s, which protects almost all our services: > https://github.com/yahoo/athenz > > > Network / Host level ACL's (automated iptables) are used to prevent one > cluster from hitting another for our most sensitive systems (and for legacy > workloads). > > > To your point below, we did have to setup more course grained acl's (which is > the minority of systems that don't support athenz) when moving to kuberetes > from static data center deployments where workloads don't move. Kubernetes > labeling is used to ensure workloads only run on nodes we know have the > proper network acl's. > > > Our business units are pretty broad. We have about 100+ separate services > from about 30 teams deployed in one K8s cluster in one data center. The rough > rule has been if we give our central ops team in the BU sudo across > workloads, we are okay with those workloads co-locating. > > > I do think the more fine grained isolation the public cloud can support > (especially with cluster auto-scaling) provides some advantages over the > model we settled on inside our data center. For us, it is nice to really let > K8s and our operators bin-pack our workloads optimally by having a larger > number of pods and nodes to work with. > > > EJ > > > On Sat, Aug 12, 2017 at 12:42 PM wrote: > EJ, > > > > Thanks for the insight into how you're handling your clusters! That's very > helpful. To your question, we are running in one of the public clouds (AWS). > > > > Since you group clusters by business unit, I have a question about how you > handle your network security. Is it the case that: > > > > (1) All applications deployed by a business unit can share a common set of > network egress rules for traffic going outside the cluster, and all pods > deployed by a business unit are allowed network access to each other. > > > > (2) Applications deployed by a business unit have varying security > requirements and not all containers deployed by a business unit can have > network access to all the others. > > > > If (1) is true, then I would assume that each business unit deploys what I > would consider an "application" in our world and you're more or less doing > what we're hoping to set up. > > > > If (2) is true, then I'm curious how you're handling the security part. Are > you using a cluster network solution, like calico, for ingress/egress rules? > Are you using a traditional network solution, then ensuring certain pods are > only scheduled on nodes that will be subject to those rules? Are you doing > something else entirely? > > > > Thanks in advance! > > Terence > > > > > > On Friday, August 11, 2017 at 8:12:56 PM UTC-7, EJ Campbell wrote: > > > Curious if you are running your setup within a physical data center or in > > one of the public clouds? > > > > > > > > > Where I am, we group clusters by large-scale business unit (many > > applications deployed on behalf of hundreds of engineers) and run three > > clusters per physical data center: canary, "corp" (special network access) > > and prod. RBAC is used for authorization among those hundreds of engineers, > > with cluster admin access for production engineers within that business > > unit. > > > > > > > > > The separate clusters per data center and canary cluster within data center > > keeps the "blast radius" small during deployments and chef keeps things > > manageable between clusters. > > > > > > > > > EJ > > > > > > > > > On Fri, Aug 11, 2017 at 6:42 PM wrote: > > > Tim, > > > > > > > > > > > > This is a hugely appreciated answer. Sorry the question was so generic! > > > > > > > > > > > > For now, we'll chose separate clusters - understanding the sub-optimal side > > effects of that decision. We'll revisit the single-or-multiple cluster > > question in a year or so, once we've had a reasonable amount of experience > > running everything on it. I'll bet the work required to run our > > infrastructure in a single cluster won't seem so daunting then :-). > > > > > > > > > > > > FWIW, we're huge fans of k8s so far and needing to run a few separate > > clusters is a pittance compared to the gains. > > > > > > > > > > > > Best, > > > > > > Terence > > > > > > > > > > > > On Friday, August 11, 2017 at 4:55:34 PM UTC-7, Tim Hockin wrote: > > > > > > > This is no
Re: [kubernetes-users] Generally speaking, separate apps = separate clusters, right?
Somewhat of a middle ground solution might be to create "sub-clusters" using taints and tolerations -- that is, have dedicated nodes for dedicated application classes or workloads inside a given cluster. It may reduce the general overhead inherent to managing separate clusters while still being able to leverage a lot of the built-in Kubernetes tooling. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
Re: [kubernetes-users] Generally speaking, separate apps = separate clusters, right?
We have an internal auth mechanism that fronts our services and has also been integrated with RBAC into K8s, which protects almost all our services: https://github.com/yahoo/athenz Network / Host level ACL's (automated iptables) are used to prevent one cluster from hitting another for our most sensitive systems (and for legacy workloads). To your point below, we did have to setup more course grained acl's (which is the minority of systems that don't support athenz) when moving to kuberetes from static data center deployments where workloads don't move. Kubernetes labeling is used to ensure workloads only run on nodes we know have the proper network acl's. Our business units are pretty broad. We have about 100+ separate services from about 30 teams deployed in one K8s cluster in one data center. The rough rule has been if we give our central ops team in the BU sudo across workloads, we are okay with those workloads co-locating. I do think the more fine grained isolation the public cloud can support (especially with cluster auto-scaling) provides some advantages over the model we settled on inside our data center. For us, it is nice to really let K8s and our operators bin-pack our workloads optimally by having a larger number of pods and nodes to work with. EJ On Sat, Aug 12, 2017 at 12:42 PM wrote: > EJ, > > Thanks for the insight into how you're handling your clusters! That's very > helpful. To your question, we are running in one of the public clouds (AWS). > > Since you group clusters by business unit, I have a question about how you > handle your network security. Is it the case that: > > (1) All applications deployed by a business unit can share a common set of > network egress rules for traffic going outside the cluster, and all pods > deployed by a business unit are allowed network access to each other. > > (2) Applications deployed by a business unit have varying security > requirements and not all containers deployed by a business unit can have > network access to all the others. > > If (1) is true, then I would assume that each business unit deploys what I > would consider an "application" in our world and you're more or less doing > what we're hoping to set up. > > If (2) is true, then I'm curious how you're handling the security part. > Are you using a cluster network solution, like calico, for ingress/egress > rules? Are you using a traditional network solution, then ensuring certain > pods are only scheduled on nodes that will be subject to those rules? Are > you doing something else entirely? > > Thanks in advance! > Terence > > > On Friday, August 11, 2017 at 8:12:56 PM UTC-7, EJ Campbell wrote: > > Curious if you are running your setup within a physical data center or > in one of the public clouds? > > > > > > Where I am, we group clusters by large-scale business unit (many > applications deployed on behalf of hundreds of engineers) and run three > clusters per physical data center: canary, "corp" (special network access) > and prod. RBAC is used for authorization among those hundreds of engineers, > with cluster admin access for production engineers within that business > unit. > > > > > > The separate clusters per data center and canary cluster within data > center keeps the "blast radius" small during deployments and chef keeps > things manageable between clusters. > > > > > > EJ > > > > > > On Fri, Aug 11, 2017 at 6:42 PM wrote: > > Tim, > > > > > > > > This is a hugely appreciated answer. Sorry the question was so generic! > > > > > > > > For now, we'll chose separate clusters - understanding the sub-optimal > side effects of that decision. We'll revisit the single-or-multiple cluster > question in a year or so, once we've had a reasonable amount of experience > running everything on it. I'll bet the work required to run our > infrastructure in a single cluster won't seem so daunting then :-). > > > > > > > > FWIW, we're huge fans of k8s so far and needing to run a few separate > clusters is a pittance compared to the gains. > > > > > > > > Best, > > > > Terence > > > > > > > > On Friday, August 11, 2017 at 4:55:34 PM UTC-7, Tim Hockin wrote: > > > > > This is not an easy question to answer without opining. I'll try. > > > > > > > > > > Kubernetes was designed to model Borg. This assumes a smaller number > > > > > of larger clusters, shared across applications and users and > > > > > environments. This design decision is visible in a number of places > > > > > in the system - Namespaces, ip-per-pod, Services, PersistentVolumes. > > > > > We really emphasized the idea that sharing is important, and the > > > > > consumption of global resources (such as ports on a node) is rare and > > > > > carefully guarded. The benefits of this are myriad: amortization of > > > > > overhead, good HA properties, efficient bin-packing, higher > > > > > utilization, and centralization of cluster administration, just to > > > > > name a few. > > > > > > > > > > What we see most often, though,
Re: [kubernetes-users] Generally speaking, separate apps = separate clusters, right?
EJ, Thanks for the insight into how you're handling your clusters! That's very helpful. To your question, we are running in one of the public clouds (AWS). Since you group clusters by business unit, I have a question about how you handle your network security. Is it the case that: (1) All applications deployed by a business unit can share a common set of network egress rules for traffic going outside the cluster, and all pods deployed by a business unit are allowed network access to each other. (2) Applications deployed by a business unit have varying security requirements and not all containers deployed by a business unit can have network access to all the others. If (1) is true, then I would assume that each business unit deploys what I would consider an "application" in our world and you're more or less doing what we're hoping to set up. If (2) is true, then I'm curious how you're handling the security part. Are you using a cluster network solution, like calico, for ingress/egress rules? Are you using a traditional network solution, then ensuring certain pods are only scheduled on nodes that will be subject to those rules? Are you doing something else entirely? Thanks in advance! Terence On Friday, August 11, 2017 at 8:12:56 PM UTC-7, EJ Campbell wrote: > Curious if you are running your setup within a physical data center or in one > of the public clouds? > > > Where I am, we group clusters by large-scale business unit (many applications > deployed on behalf of hundreds of engineers) and run three clusters per > physical data center: canary, "corp" (special network access) and prod. RBAC > is used for authorization among those hundreds of engineers, with cluster > admin access for production engineers within that business unit. > > > The separate clusters per data center and canary cluster within data center > keeps the "blast radius" small during deployments and chef keeps things > manageable between clusters. > > > EJ > > > On Fri, Aug 11, 2017 at 6:42 PM wrote: > Tim, > > > > This is a hugely appreciated answer. Sorry the question was so generic! > > > > For now, we'll chose separate clusters - understanding the sub-optimal side > effects of that decision. We'll revisit the single-or-multiple cluster > question in a year or so, once we've had a reasonable amount of experience > running everything on it. I'll bet the work required to run our > infrastructure in a single cluster won't seem so daunting then :-). > > > > FWIW, we're huge fans of k8s so far and needing to run a few separate > clusters is a pittance compared to the gains. > > > > Best, > > Terence > > > > On Friday, August 11, 2017 at 4:55:34 PM UTC-7, Tim Hockin wrote: > > > This is not an easy question to answer without opining. I'll try. > > > > > > Kubernetes was designed to model Borg. This assumes a smaller number > > > of larger clusters, shared across applications and users and > > > environments. This design decision is visible in a number of places > > > in the system - Namespaces, ip-per-pod, Services, PersistentVolumes. > > > We really emphasized the idea that sharing is important, and the > > > consumption of global resources (such as ports on a node) is rare and > > > carefully guarded. The benefits of this are myriad: amortization of > > > overhead, good HA properties, efficient bin-packing, higher > > > utilization, and centralization of cluster administration, just to > > > name a few. > > > > > > What we see most often, though, is people using one cluster for one > > > application (typically a number of Deployments, Services, etc. but one > > > logical app). This means that any user of non-trivial size is going > > > to have multiple clusters. This reduces the opportunities for > > > efficiency and overhead amortization, and increases the administrative > > > burden (and likely decreases the depth of understanding any one admin > > > can reach). > > > > > > So, why? > > > > > > First, Kubernetes is, frankly, missing a few things that make the Borg > > > model truly viable. Most clouds do not have sub-VM billing abilities. > > > Container security is not well trusted yet (though getting better). > > > Linux's isolation primitives are not perfect (Google has hundreds or > > > thousands of patches) but they are catching up. The story around > > > identity and policy and authorization and security are not where we > > > want them to be. > > > > > > Second, it's still pretty early in the overall life of this system. > > > Best practices are still being developed / discovered. Books are > > > still being written. > > > > > > Third, the system is still evolving rapidly. People are not sure how > > > things like multi-tenancy are going to look as they emerge. Siloing > > > is a hedge against uncertainty. > > > > > > Fourth, upgrades-in-place are not as easy or robust as we want them to > > > be. It's sometimes easier to
Re: [kubernetes-users] Generally speaking, separate apps = separate clusters, right?
Curious if you are running your setup within a physical data center or in one of the public clouds? Where I am, we group clusters by large-scale business unit (many applications deployed on behalf of hundreds of engineers) and run three clusters per physical data center: canary, "corp" (special network access) and prod. RBAC is used for authorization among those hundreds of engineers, with cluster admin access for production engineers within that business unit. The separate clusters per data center and canary cluster within data center keeps the "blast radius" small during deployments and chef keeps things manageable between clusters. EJ On Fri, Aug 11, 2017 at 6:42 PM wrote: > Tim, > > This is a hugely appreciated answer. Sorry the question was so generic! > > For now, we'll chose separate clusters - understanding the sub-optimal > side effects of that decision. We'll revisit the single-or-multiple cluster > question in a year or so, once we've had a reasonable amount of experience > running everything on it. I'll bet the work required to run our > infrastructure in a single cluster won't seem so daunting then :-). > > FWIW, we're huge fans of k8s so far and needing to run a few separate > clusters is a pittance compared to the gains. > > Best, > Terence > > On Friday, August 11, 2017 at 4:55:34 PM UTC-7, Tim Hockin wrote: > > This is not an easy question to answer without opining. I'll try. > > > > Kubernetes was designed to model Borg. This assumes a smaller number > > of larger clusters, shared across applications and users and > > environments. This design decision is visible in a number of places > > in the system - Namespaces, ip-per-pod, Services, PersistentVolumes. > > We really emphasized the idea that sharing is important, and the > > consumption of global resources (such as ports on a node) is rare and > > carefully guarded. The benefits of this are myriad: amortization of > > overhead, good HA properties, efficient bin-packing, higher > > utilization, and centralization of cluster administration, just to > > name a few. > > > > What we see most often, though, is people using one cluster for one > > application (typically a number of Deployments, Services, etc. but one > > logical app). This means that any user of non-trivial size is going > > to have multiple clusters. This reduces the opportunities for > > efficiency and overhead amortization, and increases the administrative > > burden (and likely decreases the depth of understanding any one admin > > can reach). > > > > So, why? > > > > First, Kubernetes is, frankly, missing a few things that make the Borg > > model truly viable. Most clouds do not have sub-VM billing abilities. > > Container security is not well trusted yet (though getting better). > > Linux's isolation primitives are not perfect (Google has hundreds or > > thousands of patches) but they are catching up. The story around > > identity and policy and authorization and security are not where we > > want them to be. > > > > Second, it's still pretty early in the overall life of this system. > > Best practices are still being developed / discovered. Books are > > still being written. > > > > Third, the system is still evolving rapidly. People are not sure how > > things like multi-tenancy are going to look as they emerge. Siloing > > is a hedge against uncertainty. > > > > Fourth, upgrades-in-place are not as easy or robust as we want them to > > be. It's sometimes easier to just bring up a new cluster and flip the > > workload over. That is easier when the workload is more contained. > > > > All that said, I still believe the right eventual model is shared. I > > fully understand why people are not doing that yet, but I think that > > in a couple years time we will look back on this era as Kubernetes' > > awkward adolescence. > > > > Tim > > > > On Fri, Aug 11, 2017 at 4:38 PM, wrote: > > > First, I'm sorry if this question has already been asked & answered. > My search-foo may have failed me. > > > > > > We're in the process of moving to k8s and I'm not confident about how > many clusters I should setup. I know there are many possible options, but > I'd really appreciate feedback from people running k8s throughout their > company. > > > > > > Nearly everything we run is containerized, and that includes our > company-wide internal services like FreeIPA, Gitlab, Jenkins, etc. We also > have multiple, completely separate, applications with varying > security/auditing needs. > > > > > > Today, we schedule all of our containers via salt which only allows > for containers to be mapped to systems in a fixed way (not great). We have > a group of systems for each application environment and one group for > internal services. Each group of systems may be subject to different > network restrictions, depending on what they're running. > > > > > > The seemingly-obvious answer to replace our setup with k8s clusters is > the following configuration: > > > > > > - Creat
Re: [kubernetes-users] Generally speaking, separate apps = separate clusters, right?
Tim, This is a hugely appreciated answer. Sorry the question was so generic! For now, we'll chose separate clusters - understanding the sub-optimal side effects of that decision. We'll revisit the single-or-multiple cluster question in a year or so, once we've had a reasonable amount of experience running everything on it. I'll bet the work required to run our infrastructure in a single cluster won't seem so daunting then :-). FWIW, we're huge fans of k8s so far and needing to run a few separate clusters is a pittance compared to the gains. Best, Terence On Friday, August 11, 2017 at 4:55:34 PM UTC-7, Tim Hockin wrote: > This is not an easy question to answer without opining. I'll try. > > Kubernetes was designed to model Borg. This assumes a smaller number > of larger clusters, shared across applications and users and > environments. This design decision is visible in a number of places > in the system - Namespaces, ip-per-pod, Services, PersistentVolumes. > We really emphasized the idea that sharing is important, and the > consumption of global resources (such as ports on a node) is rare and > carefully guarded. The benefits of this are myriad: amortization of > overhead, good HA properties, efficient bin-packing, higher > utilization, and centralization of cluster administration, just to > name a few. > > What we see most often, though, is people using one cluster for one > application (typically a number of Deployments, Services, etc. but one > logical app). This means that any user of non-trivial size is going > to have multiple clusters. This reduces the opportunities for > efficiency and overhead amortization, and increases the administrative > burden (and likely decreases the depth of understanding any one admin > can reach). > > So, why? > > First, Kubernetes is, frankly, missing a few things that make the Borg > model truly viable. Most clouds do not have sub-VM billing abilities. > Container security is not well trusted yet (though getting better). > Linux's isolation primitives are not perfect (Google has hundreds or > thousands of patches) but they are catching up. The story around > identity and policy and authorization and security are not where we > want them to be. > > Second, it's still pretty early in the overall life of this system. > Best practices are still being developed / discovered. Books are > still being written. > > Third, the system is still evolving rapidly. People are not sure how > things like multi-tenancy are going to look as they emerge. Siloing > is a hedge against uncertainty. > > Fourth, upgrades-in-place are not as easy or robust as we want them to > be. It's sometimes easier to just bring up a new cluster and flip the > workload over. That is easier when the workload is more contained. > > All that said, I still believe the right eventual model is shared. I > fully understand why people are not doing that yet, but I think that > in a couple years time we will look back on this era as Kubernetes' > awkward adolescence. > > Tim > > On Fri, Aug 11, 2017 at 4:38 PM, wrote: > > First, I'm sorry if this question has already been asked & answered. My > > search-foo may have failed me. > > > > We're in the process of moving to k8s and I'm not confident about how many > > clusters I should setup. I know there are many possible options, but I'd > > really appreciate feedback from people running k8s throughout their company. > > > > Nearly everything we run is containerized, and that includes our > > company-wide internal services like FreeIPA, Gitlab, Jenkins, etc. We also > > have multiple, completely separate, applications with varying > > security/auditing needs. > > > > Today, we schedule all of our containers via salt which only allows for > > containers to be mapped to systems in a fixed way (not great). We have a > > group of systems for each application environment and one group for > > internal services. Each group of systems may be subject to different > > network restrictions, depending on what they're running. > > > > The seemingly-obvious answer to replace our setup with k8s clusters is the > > following configuration: > > > > - Create one cluster for internal services > > - Create one cluster per application, with environments managed by > > namespaces whenever possible > > > > Great, that puts us with several clusters, but a smaller number of clusters > > than our previous "system groups". And, our network rules will mostly > > remain as-is. > > > > However, there is another option. It seems that a mix of calico > > ingress/egress rules, namespaces, RBAC, and carefully crafted pod resource > > definitions would allow us to have a single large cluster. Maybe it's just > > my inexperience, but that path seems daunting. > > > > So, all that background leads me to the simple question: In general, do you > > create one cluster per application? If not, do you have some other general > > rule that's not just "when
Re: [kubernetes-users] Generally speaking, separate apps = separate clusters, right?
I think Tim described the situation well. Multi-tenancy is something that we are giving increased attention to now. For many multi-tenant scenarios, the required features are already there, just without a great UX (it's "possible" but not "easy"), and without best-practices docs to pull together the instructions you need to follow to get secure multi-tenancy. For some multi-tenant scenarios, there are genuinely missing features. We're hoping to have a doc some time soon that someone with a reasonable understanding of their multi-tenancy requirements can use to determine whether we currently support what they want, and if so tell them how to configure the system to do it. But it won't be this month. On Fri, Aug 11, 2017 at 4:55 PM, 'Tim Hockin' via Kubernetes user discussion and Q&A wrote: > This is not an easy question to answer without opining. I'll try. > > Kubernetes was designed to model Borg. This assumes a smaller number > of larger clusters, shared across applications and users and > environments. This design decision is visible in a number of places > in the system - Namespaces, ip-per-pod, Services, PersistentVolumes. > We really emphasized the idea that sharing is important, and the > consumption of global resources (such as ports on a node) is rare and > carefully guarded. The benefits of this are myriad: amortization of > overhead, good HA properties, efficient bin-packing, higher > utilization, and centralization of cluster administration, just to > name a few. > > What we see most often, though, is people using one cluster for one > application (typically a number of Deployments, Services, etc. but one > logical app). This means that any user of non-trivial size is going > to have multiple clusters. This reduces the opportunities for > efficiency and overhead amortization, and increases the administrative > burden (and likely decreases the depth of understanding any one admin > can reach). > > So, why? > > First, Kubernetes is, frankly, missing a few things that make the Borg > model truly viable. Most clouds do not have sub-VM billing abilities. > Container security is not well trusted yet (though getting better). > Linux's isolation primitives are not perfect (Google has hundreds or > thousands of patches) but they are catching up. The story around > identity and policy and authorization and security are not where we > want them to be. > > Second, it's still pretty early in the overall life of this system. > Best practices are still being developed / discovered. Books are > still being written. > > Third, the system is still evolving rapidly. People are not sure how > things like multi-tenancy are going to look as they emerge. Siloing > is a hedge against uncertainty. > > Fourth, upgrades-in-place are not as easy or robust as we want them to > be. It's sometimes easier to just bring up a new cluster and flip the > workload over. That is easier when the workload is more contained. > > All that said, I still believe the right eventual model is shared. I > fully understand why people are not doing that yet, but I think that > in a couple years time we will look back on this era as Kubernetes' > awkward adolescence. > > Tim > > On Fri, Aug 11, 2017 at 4:38 PM, wrote: > > First, I'm sorry if this question has already been asked & answered. My > search-foo may have failed me. > > > > We're in the process of moving to k8s and I'm not confident about how > many clusters I should setup. I know there are many possible options, but > I'd really appreciate feedback from people running k8s throughout their > company. > > > > Nearly everything we run is containerized, and that includes our > company-wide internal services like FreeIPA, Gitlab, Jenkins, etc. We also > have multiple, completely separate, applications with varying > security/auditing needs. > > > > Today, we schedule all of our containers via salt which only allows for > containers to be mapped to systems in a fixed way (not great). We have a > group of systems for each application environment and one group for > internal services. Each group of systems may be subject to different > network restrictions, depending on what they're running. > > > > The seemingly-obvious answer to replace our setup with k8s clusters is > the following configuration: > > > > - Create one cluster for internal services > > - Create one cluster per application, with environments managed by > namespaces whenever possible > > > > Great, that puts us with several clusters, but a smaller number of > clusters than our previous "system groups". And, our network rules will > mostly remain as-is. > > > > However, there is another option. It seems that a mix of calico > ingress/egress rules, namespaces, RBAC, and carefully crafted pod resource > definitions would allow us to have a single large cluster. Maybe it's just > my inexperience, but that path seems daunting. > > > > So, all that background leads me to the simple question: In general, do >
Re: [kubernetes-users] Generally speaking, separate apps = separate clusters, right?
Great overview Tim! This should be an FAQ item somewhere. On Fri, Aug 11, 2017 at 4:55 PM 'Tim Hockin' via Kubernetes user discussion and Q&A wrote: > This is not an easy question to answer without opining. I'll try. > > Kubernetes was designed to model Borg. This assumes a smaller number > of larger clusters, shared across applications and users and > environments. This design decision is visible in a number of places > in the system - Namespaces, ip-per-pod, Services, PersistentVolumes. > We really emphasized the idea that sharing is important, and the > consumption of global resources (such as ports on a node) is rare and > carefully guarded. The benefits of this are myriad: amortization of > overhead, good HA properties, efficient bin-packing, higher > utilization, and centralization of cluster administration, just to > name a few. > > What we see most often, though, is people using one cluster for one > application (typically a number of Deployments, Services, etc. but one > logical app). This means that any user of non-trivial size is going > to have multiple clusters. This reduces the opportunities for > efficiency and overhead amortization, and increases the administrative > burden (and likely decreases the depth of understanding any one admin > can reach). > > So, why? > > First, Kubernetes is, frankly, missing a few things that make the Borg > model truly viable. Most clouds do not have sub-VM billing abilities. > Container security is not well trusted yet (though getting better). > Linux's isolation primitives are not perfect (Google has hundreds or > thousands of patches) but they are catching up. The story around > identity and policy and authorization and security are not where we > want them to be. > > Second, it's still pretty early in the overall life of this system. > Best practices are still being developed / discovered. Books are > still being written. > > Third, the system is still evolving rapidly. People are not sure how > things like multi-tenancy are going to look as they emerge. Siloing > is a hedge against uncertainty. > > Fourth, upgrades-in-place are not as easy or robust as we want them to > be. It's sometimes easier to just bring up a new cluster and flip the > workload over. That is easier when the workload is more contained. > > All that said, I still believe the right eventual model is shared. I > fully understand why people are not doing that yet, but I think that > in a couple years time we will look back on this era as Kubernetes' > awkward adolescence. > > Tim > > On Fri, Aug 11, 2017 at 4:38 PM, wrote: > > First, I'm sorry if this question has already been asked & answered. My > search-foo may have failed me. > > > > We're in the process of moving to k8s and I'm not confident about how > many clusters I should setup. I know there are many possible options, but > I'd really appreciate feedback from people running k8s throughout their > company. > > > > Nearly everything we run is containerized, and that includes our > company-wide internal services like FreeIPA, Gitlab, Jenkins, etc. We also > have multiple, completely separate, applications with varying > security/auditing needs. > > > > Today, we schedule all of our containers via salt which only allows for > containers to be mapped to systems in a fixed way (not great). We have a > group of systems for each application environment and one group for > internal services. Each group of systems may be subject to different > network restrictions, depending on what they're running. > > > > The seemingly-obvious answer to replace our setup with k8s clusters is > the following configuration: > > > > - Create one cluster for internal services > > - Create one cluster per application, with environments managed by > namespaces whenever possible > > > > Great, that puts us with several clusters, but a smaller number of > clusters than our previous "system groups". And, our network rules will > mostly remain as-is. > > > > However, there is another option. It seems that a mix of calico > ingress/egress rules, namespaces, RBAC, and carefully crafted pod resource > definitions would allow us to have a single large cluster. Maybe it's just > my inexperience, but that path seems daunting. > > > > So, all that background leads me to the simple question: In general, do > you create one cluster per application? If not, do you have some other > general rule that's not just "when latency or redudancy require it, make a > new cluster"? > > > > Thanks in advance! > > Terence > > > > -- > > You received this message because you are subscribed to the Google > Groups "Kubernetes user discussion and Q&A" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to kubernetes-users+unsubscr...@googlegroups.com. > > To post to this group, send email to kubernetes-users@googlegroups.com. > > Visit this group at https://groups.google.com/group/kubernetes-users. > > For more options, visit https
Re: [kubernetes-users] Generally speaking, separate apps = separate clusters, right?
This is not an easy question to answer without opining. I'll try. Kubernetes was designed to model Borg. This assumes a smaller number of larger clusters, shared across applications and users and environments. This design decision is visible in a number of places in the system - Namespaces, ip-per-pod, Services, PersistentVolumes. We really emphasized the idea that sharing is important, and the consumption of global resources (such as ports on a node) is rare and carefully guarded. The benefits of this are myriad: amortization of overhead, good HA properties, efficient bin-packing, higher utilization, and centralization of cluster administration, just to name a few. What we see most often, though, is people using one cluster for one application (typically a number of Deployments, Services, etc. but one logical app). This means that any user of non-trivial size is going to have multiple clusters. This reduces the opportunities for efficiency and overhead amortization, and increases the administrative burden (and likely decreases the depth of understanding any one admin can reach). So, why? First, Kubernetes is, frankly, missing a few things that make the Borg model truly viable. Most clouds do not have sub-VM billing abilities. Container security is not well trusted yet (though getting better). Linux's isolation primitives are not perfect (Google has hundreds or thousands of patches) but they are catching up. The story around identity and policy and authorization and security are not where we want them to be. Second, it's still pretty early in the overall life of this system. Best practices are still being developed / discovered. Books are still being written. Third, the system is still evolving rapidly. People are not sure how things like multi-tenancy are going to look as they emerge. Siloing is a hedge against uncertainty. Fourth, upgrades-in-place are not as easy or robust as we want them to be. It's sometimes easier to just bring up a new cluster and flip the workload over. That is easier when the workload is more contained. All that said, I still believe the right eventual model is shared. I fully understand why people are not doing that yet, but I think that in a couple years time we will look back on this era as Kubernetes' awkward adolescence. Tim On Fri, Aug 11, 2017 at 4:38 PM, wrote: > First, I'm sorry if this question has already been asked & answered. My > search-foo may have failed me. > > We're in the process of moving to k8s and I'm not confident about how many > clusters I should setup. I know there are many possible options, but I'd > really appreciate feedback from people running k8s throughout their company. > > Nearly everything we run is containerized, and that includes our company-wide > internal services like FreeIPA, Gitlab, Jenkins, etc. We also have multiple, > completely separate, applications with varying security/auditing needs. > > Today, we schedule all of our containers via salt which only allows for > containers to be mapped to systems in a fixed way (not great). We have a > group of systems for each application environment and one group for internal > services. Each group of systems may be subject to different network > restrictions, depending on what they're running. > > The seemingly-obvious answer to replace our setup with k8s clusters is the > following configuration: > > - Create one cluster for internal services > - Create one cluster per application, with environments managed by namespaces > whenever possible > > Great, that puts us with several clusters, but a smaller number of clusters > than our previous "system groups". And, our network rules will mostly remain > as-is. > > However, there is another option. It seems that a mix of calico > ingress/egress rules, namespaces, RBAC, and carefully crafted pod resource > definitions would allow us to have a single large cluster. Maybe it's just my > inexperience, but that path seems daunting. > > So, all that background leads me to the simple question: In general, do you > create one cluster per application? If not, do you have some other general > rule that's not just "when latency or redudancy require it, make a new > cluster"? > > Thanks in advance! > Terence > > -- > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.co
