Re: OKD 4 - A Modest Proposal

2019-06-28 Thread Clayton Coleman
On Jun 28, 2019, at 3:38 AM, Daniel Comnea  wrote:



On Fri, Jun 28, 2019 at 4:58 AM Clayton Coleman  wrote:

> > On Jun 26, 2019, at 1:08 PM, Colin Walters  wrote:
> >
> >
> >
> > On Thu, Jun 20, 2019, at 5:20 PM, Clayton Coleman wrote:
> >
> >
> >> Because the operating system integration is so critical, we need to
> >> make sure that the major components (ostree, ignition, and the kubelet)
> >> are tied together in a CoreOS distribution that can be quickly
> >> refreshed with OKD - the Fedora CoreOS project is close to being ready
> >> for integration in our CI, so that’s a natural place to start. That
> >> represents the biggest technical obstacle that I’m aware of to get our
> >> first versions of OKD4 out (the CI systems are currently testing on top
> >> of RHEL CoreOS but we have PoCs of how to take an arbitrary ostree
> >> based distro and slot it in).
> >
> > The tricky thing here is...if we want this to work the same as OpenShift
> 4/OCP
> > with RHEL CoreOS, then what we're really talking about here is a
> *derivative*
> > of FCOS that for example embeds the kubelet from OKD.  And short term
> > it will need to use Ignition spec 2.  There may be other things I'm
> forgetting.
>
> Or we have a branch of mcd that works with ignition 3 before the main
> branch switches.
>

[DC]: wouldn't this be more than just MCD ?e.g - change in installer too
[1] to import the v3 spec and work with it

[1]
https://github.com/openshift/installer/blob/master/pkg/asset/ignition/machine/node.go#L7


Yes, but possibly not a large change or one that is a “use ignition3” flag
or similar.


> I don’t know that it has to work exactly the same, but obviously the
> closer the better.
>
> >
> > Concretely for example, OKDFCOS (to use the obvious if unwieldy acronym)
> > would need to have its own uploaded "bootimages" (i.e. AMIs, PXE media
> etc)
> > that are own its own version number/lifecycle distinct from (but derived
> from)
> > FCOS (and OKD).
>
> Or it just pivots.  Pivots aren’t bad.
>
> >
> > This is completely possible (anything is in software) but the current
> team is
> > working on a lot of things and introducing a 3rd stream for us to
> maintain would
> > be a not at all small cost.  On the other hand, the benefit of doing so
> (e.g.
> > early upstream kernel/selinux-policy/systemd/podman integration testing
> > with kubernetes/OKD) might be worth it alone.
> >
> > ___
> > dev mailing list
> > dev@lists.openshift.redhat.com
> > http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>
> ___
> dev mailing list
> dev@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: OKD 4 - A Modest Proposal

2019-06-28 Thread Daniel Comnea
On Fri, Jun 28, 2019 at 4:58 AM Clayton Coleman  wrote:

> > On Jun 26, 2019, at 1:08 PM, Colin Walters  wrote:
> >
> >
> >
> > On Thu, Jun 20, 2019, at 5:20 PM, Clayton Coleman wrote:
> >
> >
> >> Because the operating system integration is so critical, we need to
> >> make sure that the major components (ostree, ignition, and the kubelet)
> >> are tied together in a CoreOS distribution that can be quickly
> >> refreshed with OKD - the Fedora CoreOS project is close to being ready
> >> for integration in our CI, so that’s a natural place to start. That
> >> represents the biggest technical obstacle that I’m aware of to get our
> >> first versions of OKD4 out (the CI systems are currently testing on top
> >> of RHEL CoreOS but we have PoCs of how to take an arbitrary ostree
> >> based distro and slot it in).
> >
> > The tricky thing here is...if we want this to work the same as OpenShift
> 4/OCP
> > with RHEL CoreOS, then what we're really talking about here is a
> *derivative*
> > of FCOS that for example embeds the kubelet from OKD.  And short term
> > it will need to use Ignition spec 2.  There may be other things I'm
> forgetting.
>
> Or we have a branch of mcd that works with ignition 3 before the main
> branch switches.
>

[DC]: wouldn't this be more than just MCD ?e.g - change in installer too
[1] to import the v3 spec and work with it

[1]
https://github.com/openshift/installer/blob/master/pkg/asset/ignition/machine/node.go#L7

>
> I don’t know that it has to work exactly the same, but obviously the
> closer the better.
>
> >
> > Concretely for example, OKDFCOS (to use the obvious if unwieldy acronym)
> > would need to have its own uploaded "bootimages" (i.e. AMIs, PXE media
> etc)
> > that are own its own version number/lifecycle distinct from (but derived
> from)
> > FCOS (and OKD).
>
> Or it just pivots.  Pivots aren’t bad.
>
> >
> > This is completely possible (anything is in software) but the current
> team is
> > working on a lot of things and introducing a 3rd stream for us to
> maintain would
> > be a not at all small cost.  On the other hand, the benefit of doing so
> (e.g.
> > early upstream kernel/selinux-policy/systemd/podman integration testing
> > with kubernetes/OKD) might be worth it alone.
> >
> > ___
> > dev mailing list
> > dev@lists.openshift.redhat.com
> > http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>
> ___
> dev mailing list
> dev@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: OKD 4 - A Modest Proposal

2019-06-27 Thread Clayton Coleman
> On Jun 26, 2019, at 1:08 PM, Colin Walters  wrote:
>
>
>
> On Thu, Jun 20, 2019, at 5:20 PM, Clayton Coleman wrote:
>
>
>> Because the operating system integration is so critical, we need to
>> make sure that the major components (ostree, ignition, and the kubelet)
>> are tied together in a CoreOS distribution that can be quickly
>> refreshed with OKD - the Fedora CoreOS project is close to being ready
>> for integration in our CI, so that’s a natural place to start. That
>> represents the biggest technical obstacle that I’m aware of to get our
>> first versions of OKD4 out (the CI systems are currently testing on top
>> of RHEL CoreOS but we have PoCs of how to take an arbitrary ostree
>> based distro and slot it in).
>
> The tricky thing here is...if we want this to work the same as OpenShift 4/OCP
> with RHEL CoreOS, then what we're really talking about here is a *derivative*
> of FCOS that for example embeds the kubelet from OKD.  And short term
> it will need to use Ignition spec 2.  There may be other things I'm 
> forgetting.

Or we have a branch of mcd that works with ignition 3 before the main
branch switches.

I don’t know that it has to work exactly the same, but obviously the
closer the better.

>
> Concretely for example, OKDFCOS (to use the obvious if unwieldy acronym)
> would need to have its own uploaded "bootimages" (i.e. AMIs, PXE media etc)
> that are own its own version number/lifecycle distinct from (but derived from)
> FCOS (and OKD).

Or it just pivots.  Pivots aren’t bad.

>
> This is completely possible (anything is in software) but the current team is
> working on a lot of things and introducing a 3rd stream for us to maintain 
> would
> be a not at all small cost.  On the other hand, the benefit of doing so (e.g.
> early upstream kernel/selinux-policy/systemd/podman integration testing
> with kubernetes/OKD) might be worth it alone.
>
> ___
> dev mailing list
> dev@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: OKD 4 - A Modest Proposal

2019-06-26 Thread Daniel Comnea
Sorry for missing out the mailer by mistake, not intentional.
PSB in blue.

On Wed, Jun 26, 2019 at 10:14 PM Daniel Comnea 
wrote:

>
>
> On Wed, Jun 26, 2019 at 6:09 PM Colin Walters  wrote:
>
>>
>>
>> On Thu, Jun 20, 2019, at 5:20 PM, Clayton Coleman wrote:
>>
>>
>> > Because the operating system integration is so critical, we need to
>> > make sure that the major components (ostree, ignition, and the kubelet)
>> > are tied together in a CoreOS distribution that can be quickly
>> > refreshed with OKD - the Fedora CoreOS project is close to being ready
>> > for integration in our CI, so that’s a natural place to start. That
>> > represents the biggest technical obstacle that I’m aware of to get our
>> > first versions of OKD4 out (the CI systems are currently testing on top
>> > of RHEL CoreOS but we have PoCs of how to take an arbitrary ostree
>> > based distro and slot it in).
>>
>> The tricky thing here is...if we want this to work the same as OpenShift
>> 4/OCP
>> with RHEL CoreOS, then what we're really talking about here is a
>> *derivative*
>> of FCOS that for example embeds the kubelet from OKD.  And short term
>> it will need to use Ignition spec 2.  There may be other things I'm
>> forgetting.
>>
> [DC] : in addition to that i think you need changes to installer/ MCO , or
> am i wrong ?
>
>
>> Concretely for example, OKDFCOS (to use the obvious if unwieldy acronym)
>> would need to have its own uploaded "bootimages" (i.e. AMIs, PXE media
>> etc)
>> that are own its own version number/lifecycle distinct from (but derived
>> from)
>> FCOS (and OKD).
>>
> [DC]: curious to understand why it can't be one single FCOS? what other
> avenues FCOS do chase will break by having  OKD baked in components?
> If we are talking about a derivative then i'd go and challenge that maybe
> a CentOS CoreOS based on RHCOS is the best bet and that can deprecate
> Project Atomic. Doing so imo (i could miss some context here) will reduce
> any tension on FCOS charter and will rapidly (hopefully) allow OKD to
> become a thing.
>
>>
>> This is completely possible (anything is in software) but the current
>> team is
>> working on a lot of things and introducing a 3rd stream for us to
>> maintain would
>> be a not at all small cost.  On the other hand, the benefit of doing so
>> (e.g.
>> early upstream kernel/selinux-policy/systemd/podman integration testing
>> with kubernetes/OKD) might be worth it alone.
>>
>> ___
>> dev mailing list
>> dev@lists.openshift.redhat.com
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
>>
>
___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: OKD 4 - A Modest Proposal

2019-06-26 Thread Michael Gugino
In Fedora Atomic, it was trivial to build a custom ostree image.  I
would hope the same is available for the FCOS model as well.

On Wed, Jun 26, 2019 at 1:36 PM Colin Walters  wrote:
>
>
>
> On Wed, Jun 26, 2019, at 1:10 PM, Colin Walters wrote:
>
> > The tricky thing here is...if we want this to work the same as OpenShift 
> > 4/OCP
> > with RHEL CoreOS, then what we're really talking about here is a 
> > *derivative*
> > of FCOS that for example embeds the kubelet from OKD.  And short term
> > it will need to use Ignition spec 2.  There may be other things I'm 
> > forgetting.
>
> It was pointed out to me that there is also some relevant discussion in this 
> issue:
> https://github.com/coreos/fedora-coreos-tracker/issues/93
>
> ___
> dev mailing list
> dev@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



-- 
Michael Gugino
Senior Software Engineer - OpenShift
mgug...@redhat.com
540-846-0304

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: OKD 4 - A Modest Proposal

2019-06-26 Thread Colin Walters



On Wed, Jun 26, 2019, at 1:10 PM, Colin Walters wrote:

> The tricky thing here is...if we want this to work the same as OpenShift 4/OCP
> with RHEL CoreOS, then what we're really talking about here is a *derivative*
> of FCOS that for example embeds the kubelet from OKD.  And short term
> it will need to use Ignition spec 2.  There may be other things I'm 
> forgetting.

It was pointed out to me that there is also some relevant discussion in this 
issue:
https://github.com/coreos/fedora-coreos-tracker/issues/93

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


Re: OKD 4 - A Modest Proposal

2019-06-26 Thread Colin Walters


On Thu, Jun 20, 2019, at 5:20 PM, Clayton Coleman wrote:


> Because the operating system integration is so critical, we need to 
> make sure that the major components (ostree, ignition, and the kubelet) 
> are tied together in a CoreOS distribution that can be quickly 
> refreshed with OKD - the Fedora CoreOS project is close to being ready 
> for integration in our CI, so that’s a natural place to start. That 
> represents the biggest technical obstacle that I’m aware of to get our 
> first versions of OKD4 out (the CI systems are currently testing on top 
> of RHEL CoreOS but we have PoCs of how to take an arbitrary ostree 
> based distro and slot it in).

The tricky thing here is...if we want this to work the same as OpenShift 4/OCP
with RHEL CoreOS, then what we're really talking about here is a *derivative*
of FCOS that for example embeds the kubelet from OKD.  And short term
it will need to use Ignition spec 2.  There may be other things I'm forgetting.

Concretely for example, OKDFCOS (to use the obvious if unwieldy acronym)
would need to have its own uploaded "bootimages" (i.e. AMIs, PXE media etc)
that are own its own version number/lifecycle distinct from (but derived from)
FCOS (and OKD).

This is completely possible (anything is in software) but the current team is
working on a lot of things and introducing a 3rd stream for us to maintain would
be a not at all small cost.  On the other hand, the benefit of doing so (e.g.
early upstream kernel/selinux-policy/systemd/podman integration testing
with kubernetes/OKD) might be worth it alone.

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


OKD 4 - A Modest Proposal

2019-06-20 Thread Clayton Coleman
TL:DR - I can’t even summarize this, but it’s worth it to read!

First, I’ll start this off with an apology - I intended to draft an OKD 4
proposal many months ago, but I kept pushing it back to fix “just one more
bug”, and as a result there’s been a real gap in regular summarization
across the project.  While I have talked to many community members
one-on-one, and many of us interact with each other on GitHub and on Slack
and at conferences, I was remiss in highlighting and concentrating the
roadmap, design, and iteration proposals for a large chunk of the last 6
months and I’ll do my best to rectify that starting now.

OKD 3.11 has been out since the fall, and is still getting fixes. It should
be no surprise to folks on this list that the acquisition of CoreOS last
spring triggered a rethink / re-imagining of what OpenShift could / should
be.  There was a broad agreement that we’ve all been doing Kubernetes The
Hard Way™ (even the cloud providers) since the early days of Kube. Some of
these hard things we accepted because Kubernetes was moving so fast.

But Kubernetes is maturing.  The code base is moving from a monorepo to a
much larger set of individual services and extensions.  The ecosystem on
top of Kubernetes is what is now innovating at a rapid pace. Contributors
from both CoreOS and OpenShift asked what a v2 of Tectonic and what a v4 of
OpenShift would look like if:


   -

   we built a platform anchored around Kubernetes
   -

   that allowed us to rapidly include and support the innovation in the
   broader ecosystem
   -

   all the way down to the operating system
   -

   that informed the evolution of operators (the natural way to extend
   Kubernetes)


That took longer than anticipated.  Many of those pieces were big bets that
we weren’t positive could be well integrated, and if you’ve been following
along in the almost a hundred repos that make up OKD you know that some of
those pieces reached maturity only in the last month or so. Some of the
aspects of Tectonic which weren’t open source weren’t immediately replaced,
and, as we evolved the initial CoreOS operating system vision, it wasn’t
clear whether it would be Fedora, or RHEL, or something in between.  Much
of the change happened in the open, but not all of the planning or debate
at the high level.

I’m sorry for that. I will make a concerted effort to summarize what is
going on and what to expect more regularly, and also do more to move those
discussions into the broader forums rather than stay in specific scopes or
specific channels.

---

So - where are we now (June 2019), and where do we go from here?

The first question is philosophy - what sort of shared goals should we
define for OKD?

I personally feel strongly that the CoreOS mission - secure the internet
with up-to-date open source software - continues to be more relevant with
each passing year. As a contributor to Kubernetes, I know how difficult
keeping an up-to-date version of it can be. As a personal computing user,
I’m horrified at the insecurity of our hardware, our software, and the
services and clouds we use.  And it’s not going to get better unless we
make a concerted effort to fix it.

The OKD mission has always been to create a developer and operations
friendly Kubernetes distribution that isn’t afraid to be opinionated in
order to make running software easier.  That opinionation started with
development tools on top of Kubernetes and security underneath. But I think
we should now take the next step - strengthen our opinions on how we ship
and update (continuously!) and how we run the platform itself.  Not just to
deliver new features, but to deliver fixes and plug security holes.

So I would propose our goal for OKD 4 to be:

***

The perfect Kubernetes distribution for those who want to continuously be
on the latest Kubernetes and ecosystem components. It should combine an
up-to-date OS, the Kubernetes control plane, and a large number of
ecosystem operators to provide an easy-to-extend distribution of Kubernetes
that is always on the latest released version of ecosystem tools.

***

Does that resonate with others?  What other goals do people believe in and
are willing to support with their time and effort?

---

The second step (if we agree with the philosophy) is to articulate the
choices that would describe Kubernetes The Probably Better Than Before Way:


*** First, Kubernetes is the best platform for running distributed apps,
and since we believe that, we should use it to run Kubernetes itself.

You should never have to:


   -

   Restart your control plane by SSHing to a machine
   -

   Remember which components are running as a system service vs as a pod
   -

   Orchestrate pods + node services.
   -

   Take downtime during a control plane reconfiguration


This ensures that the platform benefits from things we add (resiliency,
debuggability, observability), and makes the platform (which HAS to run
successfully 100% of the time) the best possible