On Sat, May 6, 2017 at 7:18 PM, 'John Huffaker' via Kubernetes user discussion and Q&A <kubernetes-users@googlegroups.com> wrote:
> At this point it's the lack of quotas and abac associated with selectors > instead of namespaces. > Can you say more about what you mean? What are scenarios where you'd like to restrict use of selectors? (and on what objects?) > I haven't looked closely enough at rbac to see if it gives us what we > need within a namespace-per-env setup. > > The other side benefit that we can tool around is that namespace make a > good "packaging" mechanism for deployments and their related > configMaps/secrets. i.e. Want to delete a deployment just delete it's > namespace. > > On May 6, 2017 6:53 PM, "'David Oppenheimer' via Kubernetes user > discussion and Q&A" <kubernetes-users@googlegroups.com> wrote: > > Re-reading this thread, I'm wondering why the existing service name > resolution procedure > <https://kubernetes.io/docs/concepts/services-networking/service/#dns> > that Nikhil mentioned doesn't solve Sam's problem (without the need for > hierarchical namespaces). Use one namespace per environment, and use the > unqualified service name for lookup to find the desired service in the same > environment. > > > > > On Tue, Apr 18, 2017 at 4:14 PM, 'Sam Ghods' via Kubernetes user > discussion and Q&A <kubernetes-users@googlegroups.com> wrote: > >> Hello, >> >> We're struggling internally at Box with a question that we were hoping >> the community could help shed some light on. >> >> At Box we have three main environments: development, staging, and >> production. The definition of 'environment' here is primarily a logical >> service discovery domain - what instances of other services should access >> me and what instances of other services should I access? (Some >> configuration can vary as well.) The development environment is a >> collection of instances where changes are pushed first and the most >> frequently changing environment. Staging is where changes go right before >> they're released and where any manual testing is done. Production is the >> set of instances serving customer traffic. >> >> Currently, we have four bare-metal datacenters, one is for non-production >> workloads (let's call it NP), the three other are for production workloads >> (let's call them A, B, C). Each one has a single large Kubernetes cluster >> named after the datacenter it's in. Initially, we considered equating >> namespaces to environments, and having the dev and staging namespaces in >> the NP cluster and the production namespace in the A, B and C clusters. But >> we could not get good isolation between different teams and microservices >> for authentication, quota management, etc. >> >> So instead, for a service 'foo,' each cluster uses namespaces like >> 'foo-dev', 'foo-staging', and 'foo-production', with the first two >> namespaces only existing in the NP cluster, but the production namespace >> only existing in clusters A, B and C. The foo team only has access to the >> foo namespaces (through ABAC, soon RBAC) and the foo namespaces can have >> quotas put on them to ensure they do not overrun a cluster or environment. >> >> However, we've started to wonder whether colocating multiple environments >> in a single cluster like this is a good idea. The first thing that gave us >> pause was federation and service discovery - as the foo service, I'd love >> to be able to deploy to a cluster, then indicate that I want to talk to the >> 'baz' service, and have it automatically find the baz service in my >> cluster, and fall back to a secondary cluster if it's not there. Having >> multiple environments in a single cluster means every app in a cluster >> needs to not only know that it reaches a 'baz' service, but it needs to >> know to specifically reach out to 'baz-dev|staging|prod' etc., which >> pollutes everyone's configs. This is specifically because there's no first >> class concept for "environment" in Kubernetes at the moment - only what >> we've clobbered into namespaces, configs and service names. (Something like >> hierarchical namespaces may be able to help with this.) >> >> The alternative we're considering is to have each cluster contain only a >> single environment. Having one environment per cluster simplifies a lot of >> configuration and object definitions across the board, because there's only >> one axis to worry about (cluster) instead of two (cluster + environment). >> We can know implicitly that everything in a given cluster belongs to a >> specific environment, potentially simplifying configuration more broadly. >> It also feels like it might be a lot more natural of a fit to Kubernetes' >> federation plans, but we haven't looked into this in as much depth. >> >> But on the flip side, I've always understood Kubernetes' ultimate goal to >> be a lot more like Borg or an AWS availability zone or region, where the >> software operates more at an infrastructure layer than the application >> layer, because this dramatically improves hardware utilization and >> efficiency and minimizes the number of clusters to operate, scale, etc. >> >> An extreme alternative we've heard is to actually bootstrap a cluster per >> team, but that feels pretty far from the Kubernetes vision, though we might >> be wrong about that as well. >> >> So, we'd love to hear opinions on not only what's recommended or possible >> today with Kubernetes, but what is the vision - should Kubernetes clusters >> exist at an application/environment layer or at the infrastructure layer? >> >> Thank you! >> Sam >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Kubernetes user discussion and Q&A" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to kubernetes-users+unsubscr...@googlegroups.com. >> >> To post to this group, send email to kubernetes-users@googlegroups.com. >> Visit this group at https://groups.google.com/group/kubernetes-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Kubernetes user discussion and Q&A" group. > To unsubscribe from this topic, visit https://groups.google.com/d/to > pic/kubernetes-users/GPaGOGxCDD8/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
