TL:DR - I can’t even summarize this, but it’s worth it to read!

First, I’ll start this off with an apology - I intended to draft an OKD 4
proposal many months ago, but I kept pushing it back to fix “just one more
bug”, and as a result there’s been a real gap in regular summarization
across the project.  While I have talked to many community members
one-on-one, and many of us interact with each other on GitHub and on Slack
and at conferences, I was remiss in highlighting and concentrating the
roadmap, design, and iteration proposals for a large chunk of the last 6
months and I’ll do my best to rectify that starting now.

OKD 3.11 has been out since the fall, and is still getting fixes. It should
be no surprise to folks on this list that the acquisition of CoreOS last
spring triggered a rethink / re-imagining of what OpenShift could / should
be.  There was a broad agreement that we’ve all been doing Kubernetes The
Hard Way™ (even the cloud providers) since the early days of Kube. Some of
these hard things we accepted because Kubernetes was moving so fast.

But Kubernetes is maturing.  The code base is moving from a monorepo to a
much larger set of individual services and extensions.  The ecosystem on
top of Kubernetes is what is now innovating at a rapid pace. Contributors
from both CoreOS and OpenShift asked what a v2 of Tectonic and what a v4 of
OpenShift would look like if:


   we built a platform anchored around Kubernetes

   that allowed us to rapidly include and support the innovation in the
   broader ecosystem

   all the way down to the operating system

   that informed the evolution of operators (the natural way to extend

That took longer than anticipated.  Many of those pieces were big bets that
we weren’t positive could be well integrated, and if you’ve been following
along in the almost a hundred repos that make up OKD you know that some of
those pieces reached maturity only in the last month or so. Some of the
aspects of Tectonic which weren’t open source weren’t immediately replaced,
and, as we evolved the initial CoreOS operating system vision, it wasn’t
clear whether it would be Fedora, or RHEL, or something in between.  Much
of the change happened in the open, but not all of the planning or debate
at the high level.

I’m sorry for that. I will make a concerted effort to summarize what is
going on and what to expect more regularly, and also do more to move those
discussions into the broader forums rather than stay in specific scopes or
specific channels.


So - where are we now (June 2019), and where do we go from here?

The first question is philosophy - what sort of shared goals should we
define for OKD?

I personally feel strongly that the CoreOS mission - secure the internet
with up-to-date open source software - continues to be more relevant with
each passing year. As a contributor to Kubernetes, I know how difficult
keeping an up-to-date version of it can be. As a personal computing user,
I’m horrified at the insecurity of our hardware, our software, and the
services and clouds we use.  And it’s not going to get better unless we
make a concerted effort to fix it.

The OKD mission has always been to create a developer and operations
friendly Kubernetes distribution that isn’t afraid to be opinionated in
order to make running software easier.  That opinionation started with
development tools on top of Kubernetes and security underneath. But I think
we should now take the next step - strengthen our opinions on how we ship
and update (continuously!) and how we run the platform itself.  Not just to
deliver new features, but to deliver fixes and plug security holes.

So I would propose our goal for OKD 4 to be:


The perfect Kubernetes distribution for those who want to continuously be
on the latest Kubernetes and ecosystem components. It should combine an
up-to-date OS, the Kubernetes control plane, and a large number of
ecosystem operators to provide an easy-to-extend distribution of Kubernetes
that is always on the latest released version of ecosystem tools.


Does that resonate with others?  What other goals do people believe in and
are willing to support with their time and effort?


The second step (if we agree with the philosophy) is to articulate the
choices that would describe Kubernetes The Probably Better Than Before Way:

*** First, Kubernetes is the best platform for running distributed apps,
and since we believe that, we should use it to run Kubernetes itself.

You should never have to:


   Restart your control plane by SSHing to a machine

   Remember which components are running as a system service vs as a pod

   Orchestrate pods + node services.

   Take downtime during a control plane reconfiguration

This ensures that the platform benefits from things we add (resiliency,
debuggability, observability), and makes the platform (which HAS to run
successfully 100% of the time) the best possible canary for when people
introduce dumb bugs into Kubernetes that break that promise.  If a CI job
fails because a control plane upgrade doesn’t maintain perfect
availability, the contributor who made the change should be able to see
that right there, before they merge their code.

Running Kube native things natively means operators, and there were a lot
of tricky problems to solve - how do you run an operator that runs the
control plane that happens to run the operator?  How do you recover when a
node crashes? How do you manage etcd as a container? How do you get each
component to have its own operator, and how do those present a coherent
story? This part of the story took a long time to evolve and is still
evolving, as anyone who has followed the operator SDK, operator lifecycle
manager project, controller-runtime project, or the upstream addons project

*** Second, the node and all the software on it exists solely to run
containers for Kubernetes, and nodes should be an innate property of a
cluster, not an external thing to manage.

You should never have to:


   Build a golden VM image

   See the error message “package conflict while updating kubelet”

   Fix a broken VM on the cloud

   Try to perform a manual rollback of packages on a host

To that end:


   The OS should contain all of the binaries necessary to function as a node

   All config on the node should come from the cluster

   The cluster decides when the node upgrades

   Tthe cluster should be able to deliver updates to the nodes just like
   any other component

To accomplish this, we decided to bring many of the key components of both
Container Linux and Atomic into a harmonious whole.  ostree is really good
at treating general purpose OS packages exactly like immutable content
sets. Ignition is the *best* first boot solution that is consistent across
clouds and metal, and has to be part of the OS to do that job well.  The OS
has to be ready to be a node, programmed to join the cluster, securely and
safely, and once it’s joined it needs to be 100% focused on running
containers. That end result is Fedora CoreOS and RHEL CoreOS - but each one
is slightly different, driven by slightly different use cases, and on
different schedules.  A key goal is that the cluster controls the software
on the node, so the expectation should be that OKD4 will control and own
the software on that node from kernel to kubelet.

We also knew that we needed to make node lifecycle trivial.  Not easy,
trivial. We added dynamic compute (just like dynamic storage and dynamic
load balancing) by adopting and stabilizing aspects of the Kubernetes
cluster api (specifically the alpha Machine API) to let you quickly and
easily add new nodes, delete them, update them, and in general treat them
like pods.  Treating nodes like pods is AMAZING (replaceable, fungible,
editable, and no big deal) and is life-changing from an operational
perspective. That said, the upstream project is going through a ton of
change and iteration right now, so for the purposes of having something
stable we chose to expose a limited subset of the machine API as an
OpenShift resource and will continue to help steer the upstream in a
direction that keeps that amazing user experience.  Bringing infrastructure
under the control of the cluster has always been at the heart of
Kubernetes, and extending that to nodes is the biggest operational
improvement next to using operators for managing components.

*** Third, we want to have only the configuration that matters (no
pointless choices), make them trivial to change post install, and make the
config of the cluster Just-Another-API-Object.

You should never have to:


   SSH to the machine to change the config of the cluster

   Have to look in filesystem directories or set filesystem permissions on

   Have to restore anything except your etcd backup

   Ask what fields are supported on a config change

You should be able to declaratively configure a Kubernetes cluster exactly
like you declaratively configure a Kubernetes application.  If you change
this global configuration (the spec) you should get an update when it’s
been applied (status). The hard choices here were how much config to expose
- I’ll be honest, we may have gotten some of it wrong, and some things that
work in 3.11 won’t work in OKD4.  We’ll definitely need to assess where we
went too far and add back some of that config.

But the new pattern is a lot easier to reason about - config changes become:

   oc apply -f my-infrastructure-config-folder

Or the more interactive:

   oc edit authentication

That’s it!  Instead of 2400 ansible parameters, or 600 kubernetes component
flags, we have ~50-60 core parameters on API objects, and a bunch of new
API objects (like machinesets and ingress controllers) that can be adopted
for your own use cases, and ALL of them can be managed with the same tools
you use for your appliactions.

Since we planned to use CRDs for this, there was a LOT of work to make
CRDs… well, frankly, be usable.  That included a ton of performance work at
scale, validation on CRDs, getting server-side ‘get’ working, and
supporting OpenAPI on CRDs:

   oc explain authentication

A huge shout-out to everyone who has been involved in CRDs in Kubernetes -
there was a lot of drudge work to get them up to the standards you’d expect
so that they can be used for config, and everyone will benefit from this.

*** Fourth, you have to believe that updates will Just Work in order to
trust turning on automatic updates

You should never have to:


   Click the upgrade button

   Decide whether it is safe to upgrade

   Forget to upgrade to get the latest security patches

   Take downtime during an upgrade

This is a hard problem, and it involves a lot of practice.  It means better
technology when deploying, but also better testing, better repeatability,
and better standardization of *how* updates are rolled out. It means better
discipline in upstream projects to avoid breaking changes, and the ability
to catch when regressions happen and roll back.  Much of the effort
involved in making updates “just work” comes from a community of users who
are willing to test, offer feedback, and provide fixes. While the three
previous items are a big help, without a community process that encourages
early adoption and incentivizes participation this is the most likely to

I think this is the area we have the most risk in achieving as a community,
so experimentation and steady evolution are important.


There are other parts of this story, but these four form the core -
operators and self management, machines and CoreOS, a simpler and
internally consistent set of on-cluster configuration, and trustworthy
automatic updates.  I believe that we should inherit and do justice to the
CoreOS vision to have a continuously up to date version of Kubernetes that
brings in the wider ecosystem where you are never afraid to upgrade - the
“cloud service” model of Kubernetes anywhere (any platform, cloud or metal)
that hides the boring details so you can Just Run Software.


The work to get from where we are to an OKD4 alpha is probably three major

Because the operating system integration is so critical, we need to make
sure that the major components (ostree, ignition, and the kubelet) are tied
together in a CoreOS distribution that can be quickly refreshed with OKD -
the Fedora CoreOS project is close to being ready for integration in our
CI, so that’s a natural place to start.  That represents the biggest
technical obstacle that I’m aware of to get our first versions of OKD4 out
(the CI systems are currently testing on top of RHEL CoreOS but we have
PoCs of how to take an arbitrary ostree based distro and slot it in).

We would also need to define a release process that benefits and reinforces
the continuous model that operators enable.  There have been a lot of
investments made to CI in OKD and I think that can help us deliver that
fast update premise. I think we’ve also seen that a distribution is more
than just it’s kernel, so concrete versions and version numbers are less
meaningful than in the past.  I suspect a date-based release model where
fixes regularly flow in, things are automatically tested, and you can
easily upgrade might better reflect the reality of the environment we live
in than the hard Kubernetes boundaries of the past, as well as help reduce
the time from problem identification to resolution.

Finally, I’m sensitive to feedback I’ve heard from many people who want the
freedom to mix and match, so I think we can err on the side of “making
forks easy” if you want to experiment with your own distribution.  That
matches a similar desire in the Fedora CoreOS working group to target
different use cases, including those that have nothing to do with
Kubernetes - that freedom to have alternatives is part of what I like the
most about open source, and I do think that everyone should have the
opportunity to make those choices differently via tooling

I will note that until OKD4 alpha is ready, those who are interested can
use to try an OCP4 cluster.  If there are things
that you *don’t* like about OCP4 that OKD4 should learn from, that feedback
is important.


This was a long email - sorry again - but I hope it can help start the
discussion about what comes next.  I suggest we gather feedback and
thoughts for a week or so here, and then try to draft something more
concrete.  In the meantime, getting a read out from where Fedora CoreOS is
currently at, thinking about what pieces of the current integration streams
may not make sense for OKD4, and what additional points of view we should
consider before moving forward would all be very concrete steps we could

I’ll be hosting an OpenShift Commons Briefing on June 26th at 9:00 am
to further explore these topics with the wider community. I hope you’ll
join the conversation and look forward to hearing from the others across
the community.  Meeting details here:

Thank you for your continued support and for being a part of the OKD

Clayton Coleman

Kubernetes and OKD contributor
dev mailing list

Reply via email to