Great overview Tim! This should be an FAQ item somewhere.

On Fri, Aug 11, 2017 at 4:55 PM 'Tim Hockin' via Kubernetes user discussion
and Q&A <kubernetes-users@googlegroups.com> wrote:

> This is not an easy question to answer without opining.  I'll try.
>
> Kubernetes was designed to model Borg.  This assumes a smaller number
> of larger clusters, shared across applications and users and
> environments.  This design decision is visible in a number of places
> in the system - Namespaces, ip-per-pod, Services, PersistentVolumes.
> We really emphasized the idea that sharing is important, and the
> consumption of global resources (such as ports on a node) is rare and
> carefully guarded.  The benefits of this are myriad: amortization of
> overhead, good HA properties, efficient bin-packing, higher
> utilization, and centralization of cluster administration, just to
> name a few.
>
> What we see most often, though, is people using one cluster for one
> application (typically a number of Deployments, Services, etc. but one
> logical app).  This means that any user of non-trivial size is going
> to have multiple clusters.  This reduces the opportunities for
> efficiency and overhead amortization, and increases the administrative
> burden (and likely decreases the depth of understanding any one admin
> can reach).
>
> So, why?
>
> First, Kubernetes is, frankly, missing a few things that make the Borg
> model truly viable.  Most clouds do not have sub-VM billing abilities.
> Container security is not well trusted yet (though getting better).
> Linux's isolation primitives are not perfect (Google has hundreds or
> thousands of patches) but they are catching up.  The story around
> identity and policy and authorization and security are not where we
> want them to be.
>
> Second, it's still pretty early in the overall life of this system.
> Best practices are still being developed / discovered.  Books are
> still being written.
>
> Third, the system is still evolving rapidly.  People are not sure how
> things like multi-tenancy are going to look as they emerge.  Siloing
> is a hedge against uncertainty.
>
> Fourth, upgrades-in-place are not as easy or robust as we want them to
> be.  It's sometimes easier to just bring up a new cluster and flip the
> workload over.  That is easier when the workload is more contained.
>
> All that said, I still believe the right eventual model is shared.  I
> fully understand why people are not doing that yet, but I think that
> in a couple years time we will look back on this era as Kubernetes'
> awkward adolescence.
>
> Tim
>
> On Fri, Aug 11, 2017 at 4:38 PM,  <terencek...@gmail.com> wrote:
> > First, I'm sorry if this question has already been asked & answered. My
> search-foo may have failed me.
> >
> > We're in the process of moving to k8s and I'm not confident about how
> many clusters I should setup. I know there are many possible options, but
> I'd really appreciate feedback from people running k8s throughout their
> company.
> >
> > Nearly everything we run is containerized, and that includes our
> company-wide internal services like FreeIPA, Gitlab, Jenkins, etc. We also
> have multiple, completely separate, applications with varying
> security/auditing needs.
> >
> > Today, we schedule all of our containers via salt which only allows for
> containers to be mapped to systems in a fixed way (not great). We have a
> group of systems for each application environment and one group for
> internal services. Each group of systems may be subject to different
> network restrictions, depending on what they're running.
> >
> > The seemingly-obvious answer to replace our setup with k8s clusters is
> the following configuration:
> >
> > - Create one cluster for internal services
> > - Create one cluster per application, with environments managed by
> namespaces whenever possible
> >
> > Great, that puts us with several clusters, but a smaller number of
> clusters than our previous "system groups". And, our network rules will
> mostly remain as-is.
> >
> > However, there is another option. It seems that a mix of calico
> ingress/egress rules, namespaces, RBAC, and carefully crafted pod resource
> definitions would allow us to have a single large cluster. Maybe it's just
> my inexperience, but that path seems daunting.
> >
> > So, all that background leads me to the simple question: In general, do
> you create one cluster per application? If not, do you have some other
> general rule that's not just "when latency or redudancy require it, make a
> new cluster"?
> >
> > Thanks in advance!
> > Terence
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Kubernetes user discussion and Q&A" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to kubernetes-users+unsubscr...@googlegroups.com.
> > To post to this group, send email to kubernetes-users@googlegroups.com.
> > Visit this group at https://groups.google.com/group/kubernetes-users.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-users+unsubscr...@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
  • [kubernetes... terencekent
    • Re: [k... 'Tim Hockin' via Kubernetes user discussion and Q&A
      • Re... Brandon Philips
      • Re... 'David Oppenheimer' via Kubernetes user discussion and Q&A
      • Re... terencekent
        • ... 'EJ Campbell' via Kubernetes user discussion and Q&A
          • ... terencekent
            • ... 'EJ Campbell' via Kubernetes user discussion and Q&A
              • ... terencekent
                • ... 'EJ Campbell' via Kubernetes user discussion and Q&A
        • ... 'Timo Reimann' via Kubernetes user discussion and Q&A
          • ... terencekent

Reply via email to