Yeah. In the end we would like: 1. "Simple" service discovery per env. foo in dev talks to bar in dev. foo in staging talks to bar in staging, and ideally the code and config for foo would just reference "bar" instead of "bar-dev" (or "bar.dev.cluster.local"). 2. Dev and staging on the same cluster. 3. Quotas per app (i.e. foo can only have 20 cpu cores) or per app-env. 4. Access control per app (i.e. only foo owners can exec into foo pods).
Right now 2,3,4 are achievable by sacrificing #1. Or we could also achieve: 1,3,4 by sacrificing #2. Or if quotas and abac/rbac worked via selectors within a namespace we could achieve 1,2,3,4. Or if we had hierarchical namespaces we could also achieve 1,2,3,4. I was somewhat worried about sacrificing #1 in the face of federated clusters, but Sam has convinced me that the "foo-dev" naming, while ugly will work just fine in that context. If this is confusing or if anyone wants further clarification I'm happy to chat as @huggsboson on the kubernetes chat. On Tue, May 9, 2017 at 8:44 AM, 'Tim Hockin' via Kubernetes user discussion and Q&A <kubernetes-users@googlegroups.com> wrote: > If I read correctly, they want quota to apply to a subset of pods in a > Namespace (by selector) not the whole namespace (so multiple teams can > share a namespace), or else they need to pollute names with > env-specific decorations. > > On Tue, May 9, 2017 at 12:44 AM, 'David Oppenheimer' via Kubernetes > user discussion and Q&A <kubernetes-users@googlegroups.com> wrote: > >> we'd need to make functionality like access control, quotas and other > >> things work based on labels and selectors within the namespace > > > > So you want access control and quotas to be per service per env, not just > > per env? > > > > > > On Mon, May 8, 2017 at 10:20 AM, 'John Huffaker' via Kubernetes user > > discussion and Q&A <kubernetes-users@googlegroups.com> wrote: > >> > >> So that's our current setup (see sam's message): We have namespaces > >> following the convention "serviceName-envName". This gives us quotas > and > >> ABAC, but we lose "semantic naming", as in our conf generation needs to > pass > >> dev, staging or prod everywhere and services need to hit > >> "https://serviceName-envName/";. > >> > >> In my mind there are two ways to make this clean: > >> 1. Hierarchical namespaces like Tim suggested. Where top level > namespace > >> is env and sub-namespace is service. > >> 2. Or the env namespaces like you suggest. Where the namespace is the > >> env. Service discovery happens in the clean way you you describe, but > we'd > >> need to make functionality like access control, quotas and other things > work > >> based on labels and selectors within the namespace. We'd probably also > need > >> to be more careful about linking configMaps to their deployments via > >> references instead of the "just kill the namespace" approach we take > today. > >> > >> #2 feels both practical and right to me at this point, but it'd > obviously > >> require some work. > >> > >> > >> > >> On Sat, May 6, 2017 at 10:34 PM, 'David Oppenheimer' via Kubernetes user > >> discussion and Q&A <kubernetes-users@googlegroups.com> wrote: > >>> > >>> On Sat, May 6, 2017 at 7:45 PM, 'John Huffaker' via Kubernetes user > >>> discussion and Q&A <kubernetes-users@googlegroups.com> wrote: > >>>> > >>>> So our "dev" env would be composed of N-different services foo, bar > and > >>>> baz for example. 3 different teams maintain the 3 different services > and > >>>> their related deployments. We would like to limit operations like > apply, > >>>> delete, exec and logs to only people on those teams to their > respective > >>>> services and deployments. the only way we found to get ABAC working > in the > >>>> way we wanted in 1.2 was to put each service/deployment in their own > >>>> namespace (+ "-env"). Additionally for each service's deployment > we'd like > >>>> to set a quota on how many CPUs/ram they can reserve. As of right > now it > >>>> looks like that is per-namespace as well. > >>> > >>> > >>> Is there some reason you don't want the three services to be in three > >>> different namespaces? If you put them in three different namespaces, > you can > >>> do everything you described with RBAC and quota. > >>> > >>>> > >>>> > >>>> I've been worried about this conflict between service discovery and > >>>> abac/quota's interpretation of how namespaces should be used for a > while. > >>>> > >>>> On May 6, 2017 7:31 PM, "'David Oppenheimer' via Kubernetes user > >>>> discussion and Q&A" <kubernetes-users@googlegroups.com> wrote: > >>>>> > >>>>> > >>>>> > >>>>> On Sat, May 6, 2017 at 7:18 PM, 'John Huffaker' via Kubernetes user > >>>>> discussion and Q&A <kubernetes-users@googlegroups.com> wrote: > >>>>>> > >>>>>> At this point it's the lack of quotas and abac associated with > >>>>>> selectors instead of namespaces. > >>>>> > >>>>> > >>>>> Can you say more about what you mean? What are scenarios where you'd > >>>>> like to restrict use of selectors? (and on what objects?) > >>>>> > >>>>>> > >>>>>> I haven't looked closely enough at rbac to see if it gives us what > >>>>>> we need within a namespace-per-env setup. > >>>>>> > >>>>>> The other side benefit that we can tool around is that namespace > make > >>>>>> a good "packaging" mechanism for deployments and their related > >>>>>> configMaps/secrets. i.e. Want to delete a deployment just delete > it's > >>>>>> namespace. > >>>>>> > >>>>>> On May 6, 2017 6:53 PM, "'David Oppenheimer' via Kubernetes user > >>>>>> discussion and Q&A" <kubernetes-users@googlegroups.com> wrote: > >>>>>> > >>>>>> Re-reading this thread, I'm wondering why the existing service name > >>>>>> resolution procedure that Nikhil mentioned doesn't solve Sam's > problem > >>>>>> (without the need for hierarchical namespaces). Use one namespace > per > >>>>>> environment, and use the unqualified service name for lookup to > find the > >>>>>> desired service in the same environment. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Tue, Apr 18, 2017 at 4:14 PM, 'Sam Ghods' via Kubernetes user > >>>>>> discussion and Q&A <kubernetes-users@googlegroups.com> wrote: > >>>>>>> > >>>>>>> Hello, > >>>>>>> > >>>>>>> We're struggling internally at Box with a question that we were > >>>>>>> hoping the community could help shed some light on. > >>>>>>> > >>>>>>> At Box we have three main environments: development, staging, and > >>>>>>> production. The definition of 'environment' here is primarily a > logical > >>>>>>> service discovery domain - what instances of other services should > access me > >>>>>>> and what instances of other services should I access? (Some > configuration > >>>>>>> can vary as well.) The development environment is a collection of > instances > >>>>>>> where changes are pushed first and the most frequently changing > environment. > >>>>>>> Staging is where changes go right before they're released and > where any > >>>>>>> manual testing is done. Production is the set of instances serving > customer > >>>>>>> traffic. > >>>>>>> > >>>>>>> Currently, we have four bare-metal datacenters, one is for > >>>>>>> non-production workloads (let's call it NP), the three other are > for > >>>>>>> production workloads (let's call them A, B, C). Each one has a > single large > >>>>>>> Kubernetes cluster named after the datacenter it's in. Initially, > we > >>>>>>> considered equating namespaces to environments, and having the dev > and > >>>>>>> staging namespaces in the NP cluster and the production namespace > in the A, > >>>>>>> B and C clusters. But we could not get good isolation between > different > >>>>>>> teams and microservices for authentication, quota management, etc. > >>>>>>> > >>>>>>> So instead, for a service 'foo,' each cluster uses namespaces like > >>>>>>> 'foo-dev', 'foo-staging', and 'foo-production', with the first two > >>>>>>> namespaces only existing in the NP cluster, but the production > namespace > >>>>>>> only existing in clusters A, B and C. The foo team only has access > to the > >>>>>>> foo namespaces (through ABAC, soon RBAC) and the foo namespaces > can have > >>>>>>> quotas put on them to ensure they do not overrun a cluster or > environment. > >>>>>>> > >>>>>>> However, we've started to wonder whether colocating multiple > >>>>>>> environments in a single cluster like this is a good idea. The > first thing > >>>>>>> that gave us pause was federation and service discovery - as the > foo > >>>>>>> service, I'd love to be able to deploy to a cluster, then indicate > that I > >>>>>>> want to talk to the 'baz' service, and have it automatically find > the baz > >>>>>>> service in my cluster, and fall back to a secondary cluster if > it's not > >>>>>>> there. Having multiple environments in a single cluster means > every app in a > >>>>>>> cluster needs to not only know that it reaches a 'baz' service, > but it needs > >>>>>>> to know to specifically reach out to 'baz-dev|staging|prod' etc., > which > >>>>>>> pollutes everyone's configs. This is specifically because there's > no first > >>>>>>> class concept for "environment" in Kubernetes at the moment - only > what > >>>>>>> we've clobbered into namespaces, configs and service names. > (Something like > >>>>>>> hierarchical namespaces may be able to help with this.) > >>>>>>> > >>>>>>> The alternative we're considering is to have each cluster contain > >>>>>>> only a single environment. Having one environment per cluster > simplifies a > >>>>>>> lot of configuration and object definitions across the board, > because > >>>>>>> there's only one axis to worry about (cluster) instead of two > (cluster + > >>>>>>> environment). We can know implicitly that everything in a given > cluster > >>>>>>> belongs to a specific environment, potentially simplifying > configuration > >>>>>>> more broadly. It also feels like it might be a lot more natural of > a fit to > >>>>>>> Kubernetes' federation plans, but we haven't looked into this in > as much > >>>>>>> depth. > >>>>>>> > >>>>>>> But on the flip side, I've always understood Kubernetes' ultimate > >>>>>>> goal to be a lot more like Borg or an AWS availability zone or > region, where > >>>>>>> the software operates more at an infrastructure layer than the > application > >>>>>>> layer, because this dramatically improves hardware utilization and > >>>>>>> efficiency and minimizes the number of clusters to operate, scale, > etc. > >>>>>>> > >>>>>>> An extreme alternative we've heard is to actually bootstrap a > cluster > >>>>>>> per team, but that feels pretty far from the Kubernetes vision, > though we > >>>>>>> might be wrong about that as well. > >>>>>>> > >>>>>>> So, we'd love to hear opinions on not only what's recommended or > >>>>>>> possible today with Kubernetes, but what is the vision - should > Kubernetes > >>>>>>> clusters exist at an application/environment layer or at the > infrastructure > >>>>>>> layer? > >>>>>>> > >>>>>>> Thank you! > >>>>>>> Sam > >>>>>>> > >>>>>>> -- > >>>>>>> You received this message because you are subscribed to the Google > >>>>>>> Groups "Kubernetes user discussion and Q&A" group. > >>>>>>> To unsubscribe from this group and stop receiving emails from it, > >>>>>>> send an email to kubernetes-users+unsubscr...@googlegroups.com. > >>>>>>> > >>>>>>> To post to this group, send email to > >>>>>>> kubernetes-users@googlegroups.com. > >>>>>>> Visit this group at https://groups.google.com/ > group/kubernetes-users. > >>>>>>> For more options, visit https://groups.google.com/d/optout. > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> You received this message because you are subscribed to a topic in > the > >>>>>> Google Groups "Kubernetes user discussion and Q&A" group. > >>>>>> To unsubscribe from this topic, visit > >>>>>> https://groups.google.com/d/topic/kubernetes-users/ > GPaGOGxCDD8/unsubscribe. > >>>>>> To unsubscribe from this group and all its topics, send an email to > >>>>>> kubernetes-users+unsubscr...@googlegroups.com. > >>>>>> To post to this group, send email to > >>>>>> kubernetes-users@googlegroups.com. > >>>>>> Visit this group at https://groups.google.com/ > group/kubernetes-users. > >>>>>> For more options, visit https://groups.google.com/d/optout. > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> You received this message because you are subscribed to the Google > >>>>>> Groups "Kubernetes user discussion and Q&A" group. > >>>>>> To unsubscribe from this group and stop receiving emails from it, > send > >>>>>> an email to kubernetes-users+unsubscr...@googlegroups.com. > >>>>>> To post to this group, send email to > >>>>>> kubernetes-users@googlegroups.com. > >>>>>> Visit this group at https://groups.google.com/ > group/kubernetes-users. > >>>>>> For more options, visit https://groups.google.com/d/optout. > >>>>> > >>>>> > >>>>> -- > >>>>> You received this message because you are subscribed to a topic in > the > >>>>> Google Groups "Kubernetes user discussion and Q&A" group. > >>>>> To unsubscribe from this topic, visit > >>>>> https://groups.google.com/d/topic/kubernetes-users/ > GPaGOGxCDD8/unsubscribe. > >>>>> To unsubscribe from this group and all its topics, send an email to > >>>>> kubernetes-users+unsubscr...@googlegroups.com. > >>>>> To post to this group, send email to kubernetes-users@googlegroups. > com. > >>>>> Visit this group at https://groups.google.com/group/kubernetes-users > . > >>>>> For more options, visit https://groups.google.com/d/optout. > >>>> > >>>> -- > >>>> You received this message because you are subscribed to the Google > >>>> Groups "Kubernetes user discussion and Q&A" group. > >>>> To unsubscribe from this group and stop receiving emails from it, send > >>>> an email to kubernetes-users+unsubscr...@googlegroups.com. > >>>> To post to this group, send email to kubernetes-users@googlegroups. > com. > >>>> Visit this group at https://groups.google.com/group/kubernetes-users. > >>>> For more options, visit https://groups.google.com/d/optout. > >>> > >>> > >>> -- > >>> You received this message because you are subscribed to a topic in the > >>> Google Groups "Kubernetes user discussion and Q&A" group. > >>> To unsubscribe from this topic, visit > >>> https://groups.google.com/d/topic/kubernetes-users/ > GPaGOGxCDD8/unsubscribe. > >>> To unsubscribe from this group and all its topics, send an email to > >>> kubernetes-users+unsubscr...@googlegroups.com. > >>> To post to this group, send email to kubernetes-users@googlegroups.com > . > >>> Visit this group at https://groups.google.com/group/kubernetes-users. > >>> For more options, visit https://groups.google.com/d/optout. > >> > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Kubernetes user discussion and Q&A" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to kubernetes-users+unsubscr...@googlegroups.com. > >> To post to this group, send email to kubernetes-users@googlegroups.com. > >> Visit this group at https://groups.google.com/group/kubernetes-users. > >> For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Kubernetes user discussion and Q&A" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to kubernetes-users+unsubscr...@googlegroups.com. > > To post to this group, send email to kubernetes-users@googlegroups.com. > > Visit this group at https://groups.google.com/group/kubernetes-users. > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Kubernetes user discussion and Q&A" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/kubernetes-users/GPaGOGxCDD8/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
