At this point it's the lack of quotas and abac associated with selectors
instead of namespaces.   I haven't looked closely enough at rbac to see if
it gives us what we need within a namespace-per-env setup.

The other side benefit that we can tool around is that namespace make a
good "packaging" mechanism for deployments and their related
configMaps/secrets.  i.e. Want to delete a deployment just delete it's
namespace.

On May 6, 2017 6:53 PM, "'David Oppenheimer' via Kubernetes user discussion
and Q&A" <kubernetes-users@googlegroups.com> wrote:

Re-reading this thread, I'm wondering why the existing service name
resolution procedure
<https://kubernetes.io/docs/concepts/services-networking/service/#dns> that
Nikhil mentioned doesn't solve Sam's problem (without the need for
hierarchical namespaces). Use one namespace per environment, and use the
unqualified service name for lookup to find the desired service in the same
environment.




On Tue, Apr 18, 2017 at 4:14 PM, 'Sam Ghods' via Kubernetes user discussion
and Q&A <kubernetes-users@googlegroups.com> wrote:

> Hello,
>
> We're struggling internally at Box with a question that we were hoping the
> community could help shed some light on.
>
> At Box we have three main environments: development, staging, and
> production. The definition of 'environment' here is primarily a logical
> service discovery domain - what instances of other services should access
> me and what instances of other services should I access? (Some
> configuration can vary as well.) The development environment is a
> collection of instances where changes are pushed first and the most
> frequently changing environment. Staging is where changes go right before
> they're released and where any manual testing is done. Production is the
> set of instances serving customer traffic.
>
> Currently, we have four bare-metal datacenters, one is for non-production
> workloads (let's call it NP), the three other are for production workloads
> (let's call them A, B, C). Each one has a single large Kubernetes cluster
> named after the datacenter it's in. Initially, we considered equating
> namespaces to environments, and having the dev and staging namespaces in
> the NP cluster and the production namespace in the A, B and C clusters. But
> we could not get good isolation between different teams and microservices
> for authentication, quota management, etc.
>
> So instead, for a service 'foo,' each cluster uses namespaces like
> 'foo-dev', 'foo-staging', and 'foo-production', with the first two
> namespaces only existing in the NP cluster, but the production namespace
> only existing in clusters A, B and C. The foo team only has access to the
> foo namespaces (through ABAC, soon RBAC) and the foo namespaces can have
> quotas put on them to ensure they do not overrun a cluster or environment.
>
> However, we've started to wonder whether colocating multiple environments
> in a single cluster like this is a good idea. The first thing that gave us
> pause was federation and service discovery - as the foo service, I'd love
> to be able to deploy to a cluster, then indicate that I want to talk to the
> 'baz' service, and have it automatically find the baz service in my
> cluster, and fall back to a secondary cluster if it's not there. Having
> multiple environments in a single cluster means every app in a cluster
> needs to not only know that it reaches a 'baz' service, but it needs to
> know to specifically reach out to 'baz-dev|staging|prod' etc., which
> pollutes everyone's configs. This is specifically because there's no first
> class concept for "environment" in Kubernetes at the moment - only what
> we've clobbered into namespaces, configs and service names. (Something like
> hierarchical namespaces may be able to help with this.)
>
> The alternative we're considering is to have each cluster contain only a
> single environment. Having one environment per cluster simplifies a lot of
> configuration and object definitions across the board, because there's only
> one axis to worry about (cluster) instead of two (cluster + environment).
> We can know implicitly that everything in a given cluster belongs to a
> specific environment, potentially simplifying configuration more broadly.
> It also feels like it might be a lot more natural of a fit to Kubernetes'
> federation plans, but we haven't looked into this in as much depth.
>
> But on the flip side, I've always understood Kubernetes' ultimate goal to
> be a lot more like Borg or an AWS availability zone or region, where the
> software operates more at an infrastructure layer than the application
> layer, because this dramatically improves hardware utilization and
> efficiency and minimizes the number of clusters to operate, scale, etc.
>
> An extreme alternative we've heard is to actually bootstrap a cluster per
> team, but that feels pretty far from the Kubernetes vision, though we might
> be wrong about that as well.
>
> So, we'd love to hear opinions on not only what's recommended or possible
> today with Kubernetes, but what is the vision - should Kubernetes clusters
> exist at an application/environment layer or at the infrastructure layer?
>
> Thank you!
> Sam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-users+unsubscr...@googlegroups.com.
>
> To post to this group, send email to kubernetes-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to a topic in the
Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/kubernetes-users/GPaGOGxCDD8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
  • [kubernetes... 'Sam Ghods' via Kubernetes user discussion and Q&A
    • Re: [k... Nikhil Jindal
      • Re... 'Tim Hockin' via Kubernetes user discussion and Q&A
        • ... 'EJ Campbell' via Kubernetes user discussion and Q&A
          • ... Matthias Rampke
            • ... Paul Ingles
    • [kuber... 'Brian Grant' via Kubernetes user discussion and Q&A
    • Re: [k... 'David Oppenheimer' via Kubernetes user discussion and Q&A
      • Re... 'John Huffaker' via Kubernetes user discussion and Q&A
        • ... 'David Oppenheimer' via Kubernetes user discussion and Q&A
          • ... 'John Huffaker' via Kubernetes user discussion and Q&A
            • ... 'David Oppenheimer' via Kubernetes user discussion and Q&A
              • ... 'John Huffaker' via Kubernetes user discussion and Q&A
                • ... 'David Oppenheimer' via Kubernetes user discussion and Q&A
                • ... 'Tim Hockin' via Kubernetes user discussion and Q&A
                • ... 'John Huffaker' via Kubernetes user discussion and Q&A
                • ... 'David Oppenheimer' via Kubernetes user discussion and Q&A
                • ... jhuffaker via Kubernetes user discussion and Q&A
                • ... Tim St. Clair

Reply via email to