If I read correctly, they want quota to apply to a subset of pods in a Namespace (by selector) not the whole namespace (so multiple teams can share a namespace), or else they need to pollute names with env-specific decorations.
On Tue, May 9, 2017 at 12:44 AM, 'David Oppenheimer' via Kubernetes user discussion and Q&A <kubernetes-users@googlegroups.com> wrote: >> we'd need to make functionality like access control, quotas and other >> things work based on labels and selectors within the namespace > > So you want access control and quotas to be per service per env, not just > per env? > > > On Mon, May 8, 2017 at 10:20 AM, 'John Huffaker' via Kubernetes user > discussion and Q&A <kubernetes-users@googlegroups.com> wrote: >> >> So that's our current setup (see sam's message): We have namespaces >> following the convention "serviceName-envName". This gives us quotas and >> ABAC, but we lose "semantic naming", as in our conf generation needs to pass >> dev, staging or prod everywhere and services need to hit >> "https://serviceName-envName/";. >> >> In my mind there are two ways to make this clean: >> 1. Hierarchical namespaces like Tim suggested. Where top level namespace >> is env and sub-namespace is service. >> 2. Or the env namespaces like you suggest. Where the namespace is the >> env. Service discovery happens in the clean way you you describe, but we'd >> need to make functionality like access control, quotas and other things work >> based on labels and selectors within the namespace. We'd probably also need >> to be more careful about linking configMaps to their deployments via >> references instead of the "just kill the namespace" approach we take today. >> >> #2 feels both practical and right to me at this point, but it'd obviously >> require some work. >> >> >> >> On Sat, May 6, 2017 at 10:34 PM, 'David Oppenheimer' via Kubernetes user >> discussion and Q&A <kubernetes-users@googlegroups.com> wrote: >>> >>> On Sat, May 6, 2017 at 7:45 PM, 'John Huffaker' via Kubernetes user >>> discussion and Q&A <kubernetes-users@googlegroups.com> wrote: >>>> >>>> So our "dev" env would be composed of N-different services foo, bar and >>>> baz for example. 3 different teams maintain the 3 different services and >>>> their related deployments. We would like to limit operations like apply, >>>> delete, exec and logs to only people on those teams to their respective >>>> services and deployments. the only way we found to get ABAC working in the >>>> way we wanted in 1.2 was to put each service/deployment in their own >>>> namespace (+ "-env"). Additionally for each service's deployment we'd like >>>> to set a quota on how many CPUs/ram they can reserve. As of right now it >>>> looks like that is per-namespace as well. >>> >>> >>> Is there some reason you don't want the three services to be in three >>> different namespaces? If you put them in three different namespaces, you can >>> do everything you described with RBAC and quota. >>> >>>> >>>> >>>> I've been worried about this conflict between service discovery and >>>> abac/quota's interpretation of how namespaces should be used for a while. >>>> >>>> On May 6, 2017 7:31 PM, "'David Oppenheimer' via Kubernetes user >>>> discussion and Q&A" <kubernetes-users@googlegroups.com> wrote: >>>>> >>>>> >>>>> >>>>> On Sat, May 6, 2017 at 7:18 PM, 'John Huffaker' via Kubernetes user >>>>> discussion and Q&A <kubernetes-users@googlegroups.com> wrote: >>>>>> >>>>>> At this point it's the lack of quotas and abac associated with >>>>>> selectors instead of namespaces. >>>>> >>>>> >>>>> Can you say more about what you mean? What are scenarios where you'd >>>>> like to restrict use of selectors? (and on what objects?) >>>>> >>>>>> >>>>>> I haven't looked closely enough at rbac to see if it gives us what >>>>>> we need within a namespace-per-env setup. >>>>>> >>>>>> The other side benefit that we can tool around is that namespace make >>>>>> a good "packaging" mechanism for deployments and their related >>>>>> configMaps/secrets. i.e. Want to delete a deployment just delete it's >>>>>> namespace. >>>>>> >>>>>> On May 6, 2017 6:53 PM, "'David Oppenheimer' via Kubernetes user >>>>>> discussion and Q&A" <kubernetes-users@googlegroups.com> wrote: >>>>>> >>>>>> Re-reading this thread, I'm wondering why the existing service name >>>>>> resolution procedure that Nikhil mentioned doesn't solve Sam's problem >>>>>> (without the need for hierarchical namespaces). Use one namespace per >>>>>> environment, and use the unqualified service name for lookup to find the >>>>>> desired service in the same environment. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Apr 18, 2017 at 4:14 PM, 'Sam Ghods' via Kubernetes user >>>>>> discussion and Q&A <kubernetes-users@googlegroups.com> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> We're struggling internally at Box with a question that we were >>>>>>> hoping the community could help shed some light on. >>>>>>> >>>>>>> At Box we have three main environments: development, staging, and >>>>>>> production. The definition of 'environment' here is primarily a logical >>>>>>> service discovery domain - what instances of other services should >>>>>>> access me >>>>>>> and what instances of other services should I access? (Some >>>>>>> configuration >>>>>>> can vary as well.) The development environment is a collection of >>>>>>> instances >>>>>>> where changes are pushed first and the most frequently changing >>>>>>> environment. >>>>>>> Staging is where changes go right before they're released and where any >>>>>>> manual testing is done. Production is the set of instances serving >>>>>>> customer >>>>>>> traffic. >>>>>>> >>>>>>> Currently, we have four bare-metal datacenters, one is for >>>>>>> non-production workloads (let's call it NP), the three other are for >>>>>>> production workloads (let's call them A, B, C). Each one has a single >>>>>>> large >>>>>>> Kubernetes cluster named after the datacenter it's in. Initially, we >>>>>>> considered equating namespaces to environments, and having the dev and >>>>>>> staging namespaces in the NP cluster and the production namespace in >>>>>>> the A, >>>>>>> B and C clusters. But we could not get good isolation between different >>>>>>> teams and microservices for authentication, quota management, etc. >>>>>>> >>>>>>> So instead, for a service 'foo,' each cluster uses namespaces like >>>>>>> 'foo-dev', 'foo-staging', and 'foo-production', with the first two >>>>>>> namespaces only existing in the NP cluster, but the production namespace >>>>>>> only existing in clusters A, B and C. The foo team only has access to >>>>>>> the >>>>>>> foo namespaces (through ABAC, soon RBAC) and the foo namespaces can have >>>>>>> quotas put on them to ensure they do not overrun a cluster or >>>>>>> environment. >>>>>>> >>>>>>> However, we've started to wonder whether colocating multiple >>>>>>> environments in a single cluster like this is a good idea. The first >>>>>>> thing >>>>>>> that gave us pause was federation and service discovery - as the foo >>>>>>> service, I'd love to be able to deploy to a cluster, then indicate that >>>>>>> I >>>>>>> want to talk to the 'baz' service, and have it automatically find the >>>>>>> baz >>>>>>> service in my cluster, and fall back to a secondary cluster if it's not >>>>>>> there. Having multiple environments in a single cluster means every app >>>>>>> in a >>>>>>> cluster needs to not only know that it reaches a 'baz' service, but it >>>>>>> needs >>>>>>> to know to specifically reach out to 'baz-dev|staging|prod' etc., which >>>>>>> pollutes everyone's configs. This is specifically because there's no >>>>>>> first >>>>>>> class concept for "environment" in Kubernetes at the moment - only what >>>>>>> we've clobbered into namespaces, configs and service names. (Something >>>>>>> like >>>>>>> hierarchical namespaces may be able to help with this.) >>>>>>> >>>>>>> The alternative we're considering is to have each cluster contain >>>>>>> only a single environment. Having one environment per cluster >>>>>>> simplifies a >>>>>>> lot of configuration and object definitions across the board, because >>>>>>> there's only one axis to worry about (cluster) instead of two (cluster + >>>>>>> environment). We can know implicitly that everything in a given cluster >>>>>>> belongs to a specific environment, potentially simplifying configuration >>>>>>> more broadly. It also feels like it might be a lot more natural of a >>>>>>> fit to >>>>>>> Kubernetes' federation plans, but we haven't looked into this in as much >>>>>>> depth. >>>>>>> >>>>>>> But on the flip side, I've always understood Kubernetes' ultimate >>>>>>> goal to be a lot more like Borg or an AWS availability zone or region, >>>>>>> where >>>>>>> the software operates more at an infrastructure layer than the >>>>>>> application >>>>>>> layer, because this dramatically improves hardware utilization and >>>>>>> efficiency and minimizes the number of clusters to operate, scale, etc. >>>>>>> >>>>>>> An extreme alternative we've heard is to actually bootstrap a cluster >>>>>>> per team, but that feels pretty far from the Kubernetes vision, though >>>>>>> we >>>>>>> might be wrong about that as well. >>>>>>> >>>>>>> So, we'd love to hear opinions on not only what's recommended or >>>>>>> possible today with Kubernetes, but what is the vision - should >>>>>>> Kubernetes >>>>>>> clusters exist at an application/environment layer or at the >>>>>>> infrastructure >>>>>>> layer? >>>>>>> >>>>>>> Thank you! >>>>>>> Sam >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Kubernetes user discussion and Q&A" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to kubernetes-users+unsubscr...@googlegroups.com. >>>>>>> >>>>>>> To post to this group, send email to >>>>>>> kubernetes-users@googlegroups.com. >>>>>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in the >>>>>> Google Groups "Kubernetes user discussion and Q&A" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/kubernetes-users/GPaGOGxCDD8/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> kubernetes-users+unsubscr...@googlegroups.com. >>>>>> To post to this group, send email to >>>>>> kubernetes-users@googlegroups.com. >>>>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Kubernetes user discussion and Q&A" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to kubernetes-users+unsubscr...@googlegroups.com. >>>>>> To post to this group, send email to >>>>>> kubernetes-users@googlegroups.com. >>>>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Kubernetes user discussion and Q&A" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/kubernetes-users/GPaGOGxCDD8/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> kubernetes-users+unsubscr...@googlegroups.com. >>>>> To post to this group, send email to kubernetes-users@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Kubernetes user discussion and Q&A" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to kubernetes-users+unsubscr...@googlegroups.com. >>>> To post to this group, send email to kubernetes-users@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Kubernetes user discussion and Q&A" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/kubernetes-users/GPaGOGxCDD8/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> kubernetes-users+unsubscr...@googlegroups.com. >>> To post to this group, send email to kubernetes-users@googlegroups.com. >>> Visit this group at https://groups.google.com/group/kubernetes-users. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Kubernetes user discussion and Q&A" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to kubernetes-users+unsubscr...@googlegroups.com. >> To post to this group, send email to kubernetes-users@googlegroups.com. >> Visit this group at https://groups.google.com/group/kubernetes-users. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
