On Tuesday, May 9, 2017 at 10:44:54 AM UTC-5, Tim Hockin wrote: > > If I read correctly, they want quota to apply to a subset of pods in a > Namespace (by selector) not the whole namespace (so multiple teams can > share a namespace), or else they need to pollute names with > env-specific decorations. > > Quota part sounds similar to DHQ- https://docs.google.com/document/d/1UEw4NASc3bIFV9q9E628ORlOiATA6HxTcBjkPKyND0I/edit?usp=sharing
> On Tue, May 9, 2017 at 12:44 AM, 'David Oppenheimer' via Kubernetes > user discussion and Q&A <kubernet...@googlegroups.com <javascript:>> > wrote: > >> we'd need to make functionality like access control, quotas and other > >> things work based on labels and selectors within the namespace > > > > So you want access control and quotas to be per service per env, not > just > > per env? > > > > > > On Mon, May 8, 2017 at 10:20 AM, 'John Huffaker' via Kubernetes user > > discussion and Q&A <kubernet...@googlegroups.com <javascript:>> wrote: > >> > >> So that's our current setup (see sam's message): We have namespaces > >> following the convention "serviceName-envName". This gives us quotas > and > >> ABAC, but we lose "semantic naming", as in our conf generation needs to > pass > >> dev, staging or prod everywhere and services need to hit > >> "https://serviceName-envName/";. > >> > >> In my mind there are two ways to make this clean: > >> 1. Hierarchical namespaces like Tim suggested. Where top level > namespace > >> is env and sub-namespace is service. > >> 2. Or the env namespaces like you suggest. Where the namespace is the > >> env. Service discovery happens in the clean way you you describe, but > we'd > >> need to make functionality like access control, quotas and other things > work > >> based on labels and selectors within the namespace. We'd probably also > need > >> to be more careful about linking configMaps to their deployments via > >> references instead of the "just kill the namespace" approach we take > today. > >> > >> #2 feels both practical and right to me at this point, but it'd > obviously > >> require some work. > >> > >> > >> > >> On Sat, May 6, 2017 at 10:34 PM, 'David Oppenheimer' via Kubernetes > user > >> discussion and Q&A <kubernet...@googlegroups.com <javascript:>> wrote: > >>> > >>> On Sat, May 6, 2017 at 7:45 PM, 'John Huffaker' via Kubernetes user > >>> discussion and Q&A <kubernet...@googlegroups.com <javascript:>> > wrote: > >>>> > >>>> So our "dev" env would be composed of N-different services foo, bar > and > >>>> baz for example. 3 different teams maintain the 3 different services > and > >>>> their related deployments. We would like to limit operations like > apply, > >>>> delete, exec and logs to only people on those teams to their > respective > >>>> services and deployments. the only way we found to get ABAC working > in the > >>>> way we wanted in 1.2 was to put each service/deployment in their own > >>>> namespace (+ "-env"). Additionally for each service's deployment > we'd like > >>>> to set a quota on how many CPUs/ram they can reserve. As of right > now it > >>>> looks like that is per-namespace as well. > >>> > >>> > >>> Is there some reason you don't want the three services to be in three > >>> different namespaces? If you put them in three different namespaces, > you can > >>> do everything you described with RBAC and quota. > >>> > >>>> > >>>> > >>>> I've been worried about this conflict between service discovery and > >>>> abac/quota's interpretation of how namespaces should be used for a > while. > >>>> > >>>> On May 6, 2017 7:31 PM, "'David Oppenheimer' via Kubernetes user > >>>> discussion and Q&A" <kubernet...@googlegroups.com <javascript:>> > wrote: > >>>>> > >>>>> > >>>>> > >>>>> On Sat, May 6, 2017 at 7:18 PM, 'John Huffaker' via Kubernetes user > >>>>> discussion and Q&A <kubernet...@googlegroups.com <javascript:>> > wrote: > >>>>>> > >>>>>> At this point it's the lack of quotas and abac associated with > >>>>>> selectors instead of namespaces. > >>>>> > >>>>> > >>>>> Can you say more about what you mean? What are scenarios where you'd > >>>>> like to restrict use of selectors? (and on what objects?) > >>>>> > >>>>>> > >>>>>> I haven't looked closely enough at rbac to see if it gives us > what > >>>>>> we need within a namespace-per-env setup. > >>>>>> > >>>>>> The other side benefit that we can tool around is that namespace > make > >>>>>> a good "packaging" mechanism for deployments and their related > >>>>>> configMaps/secrets. i.e. Want to delete a deployment just delete > it's > >>>>>> namespace. > >>>>>> > >>>>>> On May 6, 2017 6:53 PM, "'David Oppenheimer' via Kubernetes user > >>>>>> discussion and Q&A" <kubernet...@googlegroups.com <javascript:>> > wrote: > >>>>>> > >>>>>> Re-reading this thread, I'm wondering why the existing service name > >>>>>> resolution procedure that Nikhil mentioned doesn't solve Sam's > problem > >>>>>> (without the need for hierarchical namespaces). Use one namespace > per > >>>>>> environment, and use the unqualified service name for lookup to > find the > >>>>>> desired service in the same environment. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Tue, Apr 18, 2017 at 4:14 PM, 'Sam Ghods' via Kubernetes user > >>>>>> discussion and Q&A <kubernet...@googlegroups.com <javascript:>> > wrote: > >>>>>>> > >>>>>>> Hello, > >>>>>>> > >>>>>>> We're struggling internally at Box with a question that we were > >>>>>>> hoping the community could help shed some light on. > >>>>>>> > >>>>>>> At Box we have three main environments: development, staging, and > >>>>>>> production. The definition of 'environment' here is primarily a > logical > >>>>>>> service discovery domain - what instances of other services should > access me > >>>>>>> and what instances of other services should I access? (Some > configuration > >>>>>>> can vary as well.) The development environment is a collection of > instances > >>>>>>> where changes are pushed first and the most frequently changing > environment. > >>>>>>> Staging is where changes go right before they're released and > where any > >>>>>>> manual testing is done. Production is the set of instances serving > customer > >>>>>>> traffic. > >>>>>>> > >>>>>>> Currently, we have four bare-metal datacenters, one is for > >>>>>>> non-production workloads (let's call it NP), the three other are > for > >>>>>>> production workloads (let's call them A, B, C). Each one has a > single large > >>>>>>> Kubernetes cluster named after the datacenter it's in. Initially, > we > >>>>>>> considered equating namespaces to environments, and having the dev > and > >>>>>>> staging namespaces in the NP cluster and the production namespace > in the A, > >>>>>>> B and C clusters. But we could not get good isolation between > different > >>>>>>> teams and microservices for authentication, quota management, etc. > >>>>>>> > >>>>>>> So instead, for a service 'foo,' each cluster uses namespaces like > >>>>>>> 'foo-dev', 'foo-staging', and 'foo-production', with the first two > >>>>>>> namespaces only existing in the NP cluster, but the production > namespace > >>>>>>> only existing in clusters A, B and C. The foo team only has access > to the > >>>>>>> foo namespaces (through ABAC, soon RBAC) and the foo namespaces > can have > >>>>>>> quotas put on them to ensure they do not overrun a cluster or > environment. > >>>>>>> > >>>>>>> However, we've started to wonder whether colocating multiple > >>>>>>> environments in a single cluster like this is a good idea. The > first thing > >>>>>>> that gave us pause was federation and service discovery - as the > foo > >>>>>>> service, I'd love to be able to deploy to a cluster, then indicate > that I > >>>>>>> want to talk to the 'baz' service, and have it automatically find > the baz > >>>>>>> service in my cluster, and fall back to a secondary cluster if > it's not > >>>>>>> there. Having multiple environments in a single cluster means > every app in a > >>>>>>> cluster needs to not only know that it reaches a 'baz' service, > but it needs > >>>>>>> to know to specifically reach out to 'baz-dev|staging|prod' etc., > which > >>>>>>> pollutes everyone's configs. This is specifically because there's > no first > >>>>>>> class concept for "environment" in Kubernetes at the moment - only > what > >>>>>>> we've clobbered into namespaces, configs and service names. > (Something like > >>>>>>> hierarchical namespaces may be able to help with this.) > >>>>>>> > >>>>>>> The alternative we're considering is to have each cluster contain > >>>>>>> only a single environment. Having one environment per cluster > simplifies a > >>>>>>> lot of configuration and object definitions across the board, > because > >>>>>>> there's only one axis to worry about (cluster) instead of two > (cluster + > >>>>>>> environment). We can know implicitly that everything in a given > cluster > >>>>>>> belongs to a specific environment, potentially simplifying > configuration > >>>>>>> more broadly. It also feels like it might be a lot more natural of > a fit to > >>>>>>> Kubernetes' federation plans, but we haven't looked into this in > as much > >>>>>>> depth. > >>>>>>> > >>>>>>> But on the flip side, I've always understood Kubernetes' ultimate > >>>>>>> goal to be a lot more like Borg or an AWS availability zone or > region, where > >>>>>>> the software operates more at an infrastructure layer than the > application > >>>>>>> layer, because this dramatically improves hardware utilization and > >>>>>>> efficiency and minimizes the number of clusters to operate, scale, > etc. > >>>>>>> > >>>>>>> An extreme alternative we've heard is to actually bootstrap a > cluster > >>>>>>> per team, but that feels pretty far from the Kubernetes vision, > though we > >>>>>>> might be wrong about that as well. > >>>>>>> > >>>>>>> So, we'd love to hear opinions on not only what's recommended or > >>>>>>> possible today with Kubernetes, but what is the vision - should > Kubernetes > >>>>>>> clusters exist at an application/environment layer or at the > infrastructure > >>>>>>> layer? > >>>>>>> > >>>>>>> Thank you! > >>>>>>> Sam > >>>>>>> > >>>>>>> -- > >>>>>>> You received this message because you are subscribed to the Google > >>>>>>> Groups "Kubernetes user discussion and Q&A" group. > >>>>>>> To unsubscribe from this group and stop receiving emails from it, > >>>>>>> send an email to kubernetes-use...@googlegroups.com <javascript:>. > > >>>>>>> > >>>>>>> To post to this group, send email to > >>>>>>> kubernet...@googlegroups.com <javascript:>. > >>>>>>> Visit this group at > https://groups.google.com/group/kubernetes-users. > >>>>>>> For more options, visit https://groups.google.com/d/optout. > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> You received this message because you are subscribed to a topic in > the > >>>>>> Google Groups "Kubernetes user discussion and Q&A" group. > >>>>>> To unsubscribe from this topic, visit > >>>>>> > https://groups.google.com/d/topic/kubernetes-users/GPaGOGxCDD8/unsubscribe. > > >>>>>> To unsubscribe from this group and all its topics, send an email to > >>>>>> kubernetes-use...@googlegroups.com <javascript:>. > >>>>>> To post to this group, send email to > >>>>>> kubernet...@googlegroups.com <javascript:>. > >>>>>> Visit this group at > https://groups.google.com/group/kubernetes-users. > >>>>>> For more options, visit https://groups.google.com/d/optout. > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> You received this message because you are subscribed to the Google > >>>>>> Groups "Kubernetes user discussion and Q&A" group. > >>>>>> To unsubscribe from this group and stop receiving emails from it, > send > >>>>>> an email to kubernetes-use...@googlegroups.com <javascript:>. > >>>>>> To post to this group, send email to > >>>>>> kubernet...@googlegroups.com <javascript:>. > >>>>>> Visit this group at > https://groups.google.com/group/kubernetes-users. > >>>>>> For more options, visit https://groups.google.com/d/optout. > >>>>> > >>>>> > >>>>> -- > >>>>> You received this message because you are subscribed to a topic in > the > >>>>> Google Groups "Kubernetes user discussion and Q&A" group. > >>>>> To unsubscribe from this topic, visit > >>>>> > https://groups.google.com/d/topic/kubernetes-users/GPaGOGxCDD8/unsubscribe. > > >>>>> To unsubscribe from this group and all its topics, send an email to > >>>>> kubernetes-use...@googlegroups.com <javascript:>. > >>>>> To post to this group, send email to kubernet...@googlegroups.com > <javascript:>. > >>>>> Visit this group at https://groups.google.com/group/kubernetes-users. > > >>>>> For more options, visit https://groups.google.com/d/optout. > >>>> > >>>> -- > >>>> You received this message because you are subscribed to the Google > >>>> Groups "Kubernetes user discussion and Q&A" group. > >>>> To unsubscribe from this group and stop receiving emails from it, > send > >>>> an email to kubernetes-use...@googlegroups.com <javascript:>. > >>>> To post to this group, send email to kubernet...@googlegroups.com > <javascript:>. > >>>> Visit this group at https://groups.google.com/group/kubernetes-users. > > >>>> For more options, visit https://groups.google.com/d/optout. > >>> > >>> > >>> -- > >>> You received this message because you are subscribed to a topic in the > >>> Google Groups "Kubernetes user discussion and Q&A" group. > >>> To unsubscribe from this topic, visit > >>> > https://groups.google.com/d/topic/kubernetes-users/GPaGOGxCDD8/unsubscribe. > > >>> To unsubscribe from this group and all its topics, send an email to > >>> kubernetes-use...@googlegroups.com <javascript:>. > >>> To post to this group, send email to kubernet...@googlegroups.com > <javascript:>. > >>> Visit this group at https://groups.google.com/group/kubernetes-users. > >>> For more options, visit https://groups.google.com/d/optout. > >> > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Kubernetes user discussion and Q&A" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to kubernetes-use...@googlegroups.com <javascript:>. > >> To post to this group, send email to kubernet...@googlegroups.com > <javascript:>. > >> Visit this group at https://groups.google.com/group/kubernetes-users. > >> For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Kubernetes user discussion and Q&A" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to kubernetes-use...@googlegroups.com <javascript:>. > > To post to this group, send email to kubernet...@googlegroups.com > <javascript:>. > > Visit this group at https://groups.google.com/group/kubernetes-users. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
