[atomic-devel] Getting to full deprecation of the projectatomic/ Github organization

[Colin Walters]

bubblewrap moved: https://github.com/containers/bubblewrap
rpm-ostree moved: https://github.com/coreos/rpm-ostree

Of the things remaining...probably the biggest is our docker branch: 
https://github.com/projectatomic/docker
I feel like it'd be cleanest if we created a new org for this stuff...queue 
naming bikeshed, I know.

There's also: 
https://github.com/projectatomic/ContainerApplicationGenericLabels - did we 
ever standardize that stuff elsewhere?

I think if we got those bits done we could probably mass-archive the remaining 
repos.

Re: [atomic-devel] libvirtd+qemu-kvm on atomic server, make sense?

[Colin Walters]

On Sat, May 4, 2019, at 7:27 PM, Angelos Ching wrote:
> Dear Fellow Dev / Users,
> 
> I'm trying to figure out whether running libvirtd+qemu-kvm on Atomic 
> server make sense.
> 
> One of my CentOS 7 box (my homelab VM host) had been through 
> interrupted OS update twice and it required my intervention to become 
> bootable again in both instances. Atomic Server's ostree way of update 
> seems to be a very good way of avoiding these kind of troubles.

Right.

> Based on my very basic research, it seems that there's an image for 
> libvirtd so I tried `atomic run hrishikesh/libvirtd` but it got error 
> on trying to do something with systemd. Then I noticed there's an 
> official systemd image from CentOS so I tried `atomic run 
> centos/systemd` which failed similarly.

Containerizing libvirt is possible but gets into interesting issues.

>  1. Whether this libvirt+qemu-kvm on atomic use case make sense?

FWIW my home server (and for that matter Silverblue), I do:
`rpm-ostree install libvirt`

Today rpm-ostree is a fully hybrid image/package system, so you're not giving 
up transactional updates.  More at https://github.com/projectatomic/rpm-ostree/

I also do use direct qemu and libvirt qemu:///session from inside containers; 
it's mainly supporting the qemu:///system path that gets interesting when 
containerized as libvirt expects to interact with the host networking, storage 
etc.

Re: [atomic-devel] Rebase Fedora Atomic 27 > 29 fails to boot

[Colin Walters]

On Wed, Mar 20, 2019, at 11:41 AM, Doug Campbell wrote:
> Currently running Fedora Atomic 27.153
> 
> Following instructions at: 
> http://www.projectatomic.io/blog/2018/10/fedora-atomic-28-to-29-upgrade/ to 
> upgrade to version 29.
> 
> Upon reboot everything is fine until I see:
> 
> (1 of 2) A start job is running for Network Manager (3min 6s / 1min 36s)
> [timestmp] systemd-journal[990]: Failed to send WATCHDOG=1 notification 
> message: Connection refused
> [timestmp] systemd-journal[990]: Failed to send WATCHDOG=1 notification 
> message: Transport endpoint is not connected
> 
> The last line continues to repeat periodically for as long as I would wait.

Try sending the systemd journal to the console:
https://freedesktop.org/wiki/Software/systemd/Debugging/
Specifically you want to add to the kernel commandline:

systemd.journald.forward_to_console=1 console=ttyS0,38400 console=tty1

If you're using the default GRUB setup just press `e` at the prompt.

If this is a physical box that's hard to change remotely, one thing I'd 
recommend is to try setting up a similar system in a VM (using the same config 
management etc.) and see if it reproduces there too.  VMs are a lot easier to 
debug.

Re: [atomic-devel] Install CRI-O on Atomic Host

[Colin Walters]

On Wed, Jan 30, 2019, at 9:20 AM, Daniel Walsh wrote:

> Right you can get CRI-O and OKD installed on atomic host as layered
> packages.  But it will not be updated via the atomic upgrades.

Hmm?  rpm-ostree updates layered packages too.  It's a true hybrid 
image/package system.  

Now...there are potential synchronization issues, see e.g. 
https://github.com/projectatomic/rpm-ostree/issues/415

But a lot of those issues can happen in inverse on a "traditional" yum managed 
system; if for example cri-o (or any other separate package) requires an older 
selinux-policy that can end up silently holding back OS updates.

Re: [atomic-devel] Install CRI-O on Atomic Host

[Colin Walters]

On Tue, Jan 29, 2019, at 2:31 PM, mabi wrote:
> Ah ok so standard CentOS would do it... I guess I missed the point, I 
> thought Atomic Host is THE distribution to go with when using such 
> platforms like okd.io/OpenShift...

An interesting topic here is whether saying "Fedora/CentOS/RHEL" means *not* 
{Fedora, CentOS, RHEL} Atomic Host.  I don't think so; or to avoid double 
negatives, I think Atomic/CoreOS style systems are part of the family. 

Now, your question gets to a larger problem we've struggled with around 
OpenShift/Kubernetes and platform choices and how we manage things.  If you've 
noticed for OpenShift 4.0 we're doing something fundamentally different here 
with Red Hat CoreOS by baking in the kubelet/cri-o to the host directly.

Some people often forget that the rpm-ostree technology underlying Atomic Host 
has package layering; I briefly went to the https://cri-o.io/ web page but the 
link it has for CentOS RPMs is wrong as far as I can tell.  But if the RPMs 
exist it almost certainly works to do `rpm-ostree install cri-o`.  But really 
this is more about OKD.

Re: [atomic-devel] "Hello World" - seems to be successful

[Colin Walters]

On Tue, Nov 20, 2018, at 12:02 AM, Bob Gustafson wrote:

> rpm-ostree uninstall  for all of my packages, but there must be a 
> 'blow them all away' command, yes?

rpm-ostree uninstall --all

It's in --help but not the man page because...those get out of sync =/

See also https://github.com/projectatomic/rpm-ostree/pull/1419

Re: [atomic-devel] Bare metal installation without video display??

[Colin Walters]

On Wed, Aug 22, 2018, at 11:12 AM, Bob Gustafson wrote:
> Hi Colin
> 
> Thanks very much for your comments.
> 
> (I have had some success - see my email which was sent 3 mins before 
> yours..)
> 
> I tried accessing your bugzilla.redhat link, but got:
> 
>   You are not authorized to access bug #1057271.

OK, I'll see about getting that bug opened up.  In the meantime,
see the docs in
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/installation_guide/sect-kickstart-howto

The code exists in Fedora too, but per my PR seems broken by a very small bug.

Re: [atomic-devel] Bare metal installation without video display??

[Colin Walters]

On Sun, Aug 19, 2018, at 10:50 AM, Bob Gustafson wrote:
> My particular piece of bare metal seems to have enough memory, disk 
> space, Intel x86_64 architecture w/4 cores, ethernet nics, but it does 
> not have any video support.

See https://bugzilla.redhat.com/show_bug.cgi?id=1057271
and https://github.com/rhinstaller/anaconda/pull/1544
for where I tried to do a fully-automated installation via USB
but hit an Anaconda bug.  Does the OEMDRV feature sound
like what you want?  If so might be worth adding a comment
to the above PR.

Also related is
https://github.com/coreos/fedora-coreos-tracker/issues/18

Re: [atomic-devel] docker optionsin /etc/sysconfig/docker

[Colin Walters]

On Sat, Jun 2, 2018, at 8:30 AM, arnaud gaboury wrote:
> 
>  # systemctl edit docker.service
> [Service]
> Execstart=
> ExecStart=/usr/bin/dockerd --selinux-enabled
> # systemctl restart docker
> # docker run fedora cat /proc/self/attr/current
> system_u:system_r:container_t:s0:c81,c142#

See:
/usr/lib/systemd/system/docker.service
You need all that stuff in the default ExecStart= to have the config files work.

[atomic-devel] AMIs on magnetic storage vs gp2

[Colin Walters]

I'm looking at general things in how CoreOS Container Linux does
things vs Fedora Atomic - specifically I think we should have a file like:
https://alpha.release.core-os.net/amd64-usr/current/coreos_production_ami_all.json

While I was doing this, one thing I noticed is that CL is gp2 only:
https://github.com/coreos/mantle/commit/5f477a671a305a12ae98542f74a8452b7a6d4dc0
and
https://github.com/coreos/mantle/commit/77de264b8cb859f38fa5795668cc33a6494b1006
More info: 
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html

This makes a lot of sense to me - I personally have fallen into the IOPS 
performance cliff for `standard`
magnetic storage.  

Is it worth us continuing to do AMI uploads targeting the legacy `standard` EBS 
storage?
Does anyone use it intentionally?

Re: [atomic-devel] Atomic as Base OS for Standalone Appliance

[Colin Walters]

Hi Shane,

On Mon, May 21, 2018, at 3:09 PM, Shane O'Donnell wrote:
> Hey All –
> 
> We’re building an IoT edge device based on CentOS Atomic Host 

Cool; without knowing more it feels like this falls in between "server"
and "device"?   If it's more "device" like then the newly formed Fedora
IoT group is intended to be the place, although as of yet as far as I
can tell there isn't even a mailing list?

> As an example, we’re trying to follow the “transactional” nature of 
> Atomic, but we’re finding there’s no real “commit”.  When we need to 
> change something in the OS itself, 

A big topic here is whether you derive from the CAH ostree commits (hostimages),
or whether you do a custom compose/build.  The latter is fairly straightforward
and gives you full control.

> we end up stuck with a “hotfix” 
> flagged deployment that we can’t really “commit” to clear the flag, 
> which leaves us unable to make follow-on transactional changes.

It depends on what you want to do - we could probably fairly easily
support a model where there's one or more persistent "layer"s.  I just
filed this: https://github.com/projectatomic/rpm-ostree/issues/1370

But again it depends on what content you want there.

> We are also in the process of migrating our Ansible-based platform 
> customization and software-load (i.e., Docker containers managed via 
> Docker Compose) which seems somewhat unnatural in the Atomic world.

Hmm.  Today we do support Ansible although most of our efforts
as you noticed are oriented towards Kubernetes.  

> The docs seem pretty clearly focused on using Atomic as the OS for a 
> cluster, but our standalone deployments seem to leave us with a lot of 
> questions.  I think the following list is a reasonable summary of the 
> big questions we haven’t really figured out yet:
> 
> 
>   *   Is there a “commit” that would allow us to commit the current 
> “hotfix” to the current deployment?

Not today, though the above issue covers some of it.   On IRC you mentioned
kernel arguments; we could add `--inplace` or something.  That'd likely
be fairly easy to do.  Kernel argument management is a gap for sure.

>   *   All of our appliances start with a common base image.  What’s the 
> recommended approach for changing kernel arguments so they appear in the 
> common base image?

ostree today does not store kernel arguments in the commit.  See also
https://github.com/ostreedev/ostree/issues/479

The high level thing to understand is that the design mostly is that
rpm-ostree can be used in the same places/same way as one might use
"classic" package managers like yum/apt.  Since today one doesn't
store kernel arguments in rpm/deb packages, ostree also didn't do it
by default.  Instead you use kickstart or config management or whatever.
But that doesn't mean we couldn't!   Like I said in the issue, it'd
probably make sense to do, but we'd have to work through some overlap
in terms of how it works with Anaconda, etc.

See also one of our oldest issues:
https://github.com/projectatomic/rpm-ostree/issues/471

> Note that we’re both invested and committed to using Atomic, and we’d 
> like to offer up a resource to help flesh out the “standalone” side of 
> the documentation so we both have some agreed upon contract between 
> users and core devs, as well as to better enable new users with this use 
> case.

That's always nice!  

Re: [atomic-devel] rpm-ostree equivalent of fastest mirror and parallel downloads

[Colin Walters]

On Thu, May 17, 2018, at 8:24 PM, Muayyad AlSadi wrote:
> Hi,
> 
> in /etc/dnf/dnf.conf I have
> 
> fastestmirror=1
> max_parallel_downloads=20
> 
> what is the of that in rpm-ostree?

So...this is an interesting topic.  As the name implies, rpm-ostree
glues together two fundamentally different "engines" - ostree and libdnf.

Theoretically, those config options would apply when downloading
layered packages with libdnf.  Except today, libdnf doesn't parse the
global dnf.conf file, which is (kind of) a bug.

Now if you're talking about what the ostree equivalents of those are:
they don't exist today.  For `fastestmirror` my opinion is
that's a lot of complexity that's better handled by CDNs.  Which
we have configured today for the FAH/Silverblue content, but
it's somewhat neutered by https://github.com/ostreedev/ostree/issues/1541

As far as `max_parallel_downloads` - libostree today already uses
a lot higher number than the default dnf value of `3`.  Big picture
the biggest efficiency win for things like this using HTTP/2 which I hope we 
can re-enable
for Fedora soon.

Now of course with https://github.com/projectatomic/rpm-ostree/issues/1081
things get a lot easier to understand/manage, as content is fetched in a single 
way.
But the libdnf-vs-dnf.conf issue would still apply.  And today librepo doesn't 
enable
HTTP/2 unfortunately, but that shouldn't be too hard to fix.

Re: [atomic-devel] build atomic workstation live usb (iso)

[Colin Walters]

On Wed, May 16, 2018, at 12:38 AM, Damian Ivanov wrote:
> Hello,
> 
> I would like to test Fedora Atomic Workstation and make custom images
> of it (like I used to with livecd-tools). There are guides for
> building VM images, what would be needed to convert them into *iso
> images that can be written to USB?

I think the first thing you want is to use `rpm-ostree compose tree`
to build a custom host image/tree - you can then rebase an existing
install to it.

To make an ISO, it's https://pagure.io/fedora-lorax-templates/tree/master
which are used as inputs to lorax to generate a "traditional" ISO,
the same method is used for FAH and FAW/Silverblue.
Notably currently FAW/Silverblue is not a "live" installer which
is probably more what you're thinking.  (It'd make sense to switch
Silverblue to live, but would require some work in Anaconda)

As for how it's used, the "pungi" tool in Fedora is what knows
how to pass the templates to lorax.  You can get a sense of things work by
looking at logs:

https://kojipkgs.fedoraproject.org/compose//branched/Fedora-28-20180427.n.0/logs/x86_64/AtomicWorkstation/ostree_installer-4/runroot.log

Sometimes to reproduce things locally I've just copy/pasted
a command like that from logs, massaged it to use local data, etc.

Re: [atomic-devel] Failed to start Docker Storage Setup

[Colin Walters]

Hi,

On Tue, May 15, 2018, at 7:44 PM, Rares Vernica wrote:
> Hi,
> 
> I installed the latest CentOS image on bare metal. Docker seems to work
> fine, but I get an error message at start-up about the docker-storage-setup
> service failing to start. I am happy with the current Docker storage setup,
> I just want want this storage service to be happy and not error at start-up.

See
https://github.com/projectatomic/container-storage-setup/issues/267

Re: [atomic-devel] Podman-0.4.4 was just releeased.

[Colin Walters]

On Sat, Apr 28, 2018, at 4:51 AM, Daniel Walsh wrote:
> This version no longer  requires the Buildah package to be installed but 
> can still do podman builds.  It is 36 Mb in size and I believe should 
> now be considered for inclusion into the atomic host by default as a 
> container runtime.

https://pagure.io/atomic-wg/issue/422

Re: [atomic-devel] Fedora Container Release

[Colin Walters]

On Mon, Mar 19, 2018, at 3:38 PM, Clement Verna wrote:

> So if you maintain a container could you please build your container
> against the f28 and master branch.

There's a fairly important deeper question here: Do we really
want to build all of these containers against each fedora release?

Does anyone *care* what underlying base image is used for e.g.
waiverdb?

I understand that we're sort of doing this just because that's
what we do with RPMs...but is that the right thing?

It feels like for most containers we should really only ship
them based on the current "stable" fedora?

I can imagine it being useful to build the containers just
so we can test them for the future when the N+1 becomes the "stable" N,
but I'm not sure if we'd really want to highlight them much?

Also I simply do not understand why we're requiring a human to
initiate builds.

Re: [atomic-devel] Kubernetes manual setup

[Colin Walters]

On Wed, Feb 21, 2018, at 12:34 PM, Chris Negus wrote:

> In my mind, this means that someone trying out vanilla Kubernetes will 
> start with some OS outside of the Fedora/RHEL/CentOS ecosystem. My 
> question is, is it okay to let this content die? Or should we encourage 
> some way to still manually use Kubernetes on Fedora (Atomic or not)?

This is a pretty deep question with a lot of implications and history involved.
Let me start by stating something that's fairly obvious: Kubernetes is pretty
popular!  There are multiple "distros" of Kubernetes now, including now *two*
products owned by Red Hat.

My personal opinion (now) is that we are primarily producing a base operating 
system; we need to
support these different Kubernetes distributions; as well as non-Kubernetes
cases.

Including Kubernetes directly in AH was clearly a historical mistake, one that
we are just finally digging ourselves out from.  Part of the issue here is that
in Atomic we are introducing *two* entirely new ways to deliver software;
via system containers as well as package layering now.  And Kubernetes could
be done via both.

In practice, I think system containers for Kubernetes make a lot of sense 
because
then you get a natural decoupling of the base OS from your container service; 
they
live at different cadences.  But it's been a challenge because it's quite 
different
from all of the other ways to run Kube.

Kubernetes *also* is an excellent use case for the Fedora "Modularity" effort - 
having
just one version of Kubernetes included in the "Everything" repository doesn't 
make
sense when upstream supports multiple.

The way I would describe this then is that we are providing technology that is 
intended
to support this use case, and we also need to be participating in issues that 
cross
the OS/Kubernetes boundary (for example: SELinux policy, host update 
management).
So I think our current efforts at having system containers for upstream 
Kubernetes maintained by
"us" are indeed the right approach - we need to ensure the basics work.  Ideally
of course we have more people gaining traction/buy-in in the upstream 
Kubernetes community.
In practice I think a whole lot of that is mostly "non-Atomic" CentOS/RHEL 
using RPMs
or just plain dropping binaries in /opt - hopefully we can get some of the 
upstream
community using the technologies we've developed here.  But it's tricky because
we have other Kubernetes distributions like OpenShift which are also a primary 
use case.

Anyways another important thing here is: we need to *clearly* distinguish the 
"dev/test"
scenario from the "production" use case.  For the "dev/test" case there's 
minikube/minishift
as well as `oc cluster up` which I use a lot (and a major bonus of using Linux 
as
a desktop including Fedora Atomic Workstation is that you can `oc cluster up` 
directly
on metal and avoid virt overhead).

Re: [atomic-devel] Proposing moving https://github.com/stefwalter/oci-kvm-hook into projectatomic

[Colin Walters]

On Fri, Feb 16, 2018, at 2:30 PM, Josh Berkus wrote:
> On 02/16/2018 11:23 AM, Colin Walters wrote:
> > Hi,
> > 
> > In working on our CI (and just locally in containers in general), I
> > find https://github.com/stefwalter/oci-kvm-hook to be very, very
> > useful.  Yes, there are other ways one can do it; see the thread in
> > https://github.com/stefwalter/oci-kvm-hook/issues/5

> Does Stef *want* it to be moved? It's his repo.

See the issue linked just above: 
https://github.com/stefwalter/oci-kvm-hook/issues/5

Re: [atomic-devel] Proposing moving https://github.com/stefwalter/oci-kvm-hook into projectatomic

[Colin Walters]

On Fri, Feb 16, 2018, at 2:29 PM, Daniel Walsh wrote:

> Does this actually work? 

Yes =)   For example it broke and we fixed it e.g.:
https://github.com/stefwalter/oci-kvm-hook/pull/4

> I would figure the device cgroup would prevent 
> use of the kvm device inside a container unless you also modified the 
> cgroup?
> 
> 
> podman run --device /dev/kvm

I guess the thing is personally, I see it as quite safe to expose
the KVM device nowadays, and having to annotate containers
explicitly for it is annoying, particularly in the Kube/OpenShift
case.  That said the linked thread above contains a proposal
for the Kube equivalent of this.

Re: [atomic-devel] Fedora Atomic Workstation questions

[Colin Walters]

On Wed, Feb 7, 2018, at 1:58 PM, Dusty Mabe wrote:

> > Oh, great! for some reason I assumed rpm-ostree can only download 
> > pre-composed trees from Fedora.

Right, rpm-ostree is a fully hybrid system, it links to libdnf *and* libostree 
and combines
functionality from both.  See:
https://fedorapeople.org/~walters/2018.01-devconf/index.html#/3

> > In the future it might be worth to add some sort of compatibility wrapper 
> > around "dnf install" (and such) like dnf has for yum - to show a message to 
> > let you know that tool is "deprecated" and that rpm-ostree is what they 
> > need to use now. Otherwise people might get confused about "how do I 
> > install anything? non of the tools I'm used to are here".

But `dnf` isn't deprecated!  As a *developer* I spend the vast majority of my 
time
in my dev containers using yum/dnf.  It works well there!  You need to get used 
to thinking in terms of
"host" vs "container/disposable VM/etc".  The rpm-ostree package layering is 
something
you will generally need to understand a lot upfront, but after that it fades 
into the
background, and that's the idea.

Though a cool aspect IMO of all of this is that if you aren't doing development,
the "pure desktop" case of OS + web browser can be pure "ostree" mode, no
client side dependency resolution at all.

> > At the moment, I can have auto-updates for my host system and for my 
> > Flatpaks, but there's no real mechanism to update containers. I assume 
> > people will not be happy if we just automatically run "dnf update" inside 
> > all their containers, but if you have a lot of "contexts" you'll have to 
> > update all of them yourself.

I have definitely spent a lot of time thinking about this too.  Actually,
the rpm-ostree tooling is fairly easily generalizable to generate and maintain
an *arbitrary* set of root filesystems, updated from RPM packages.

But here's the thing - I also need to build/test some software on e.g. Debian.
If we build out a lot of RPM-specific tooling for containers, it doesn't help 
that
case.  Further what people tend to do in containers is things like `sudo pip 
install`
or equivalent.  I actually do that myself sometimes transiently for e.g. `sudo 
pip install python-openstackclient`.
Generalizing "check for updates" to that is extraordinarily hard.

Anyways, you're right this is a big problem space; there's not going
to be one magical solution.  I personally have found the tradeoffs to be
worth it.  And I think we'll iterate towards a more opinionated tool here
that at least handles the default case of containers built from RPMs +
layer of config or so; perhaps maybe even eventually hooking up gnome-software
to understand how to do updates there too.

Personally right now I use a mix of `yum update` periodically and
just rebuild from scratch every few weeks.

[atomic-devel] Atomic/Container talks at devconf.cz 2018

[Colin Walters]

There was a lot of interest in Atomic and Containers at devconf.cz 2018!

This isn't quite the full list of talks; I stalled out a bit searching YouTube. 
 You
can always start from the schedule: https://devconfcz2018.sched.com/grid
However, I've added slide links here for some of them.  If you did give
a talk I recommend you upload your slides and add them to a comment
on the video.

Dusty Mabe, Josh Berkus: Catching Up With Atomic: Retrospective & BOF
Youtube: https://www.youtube.com/watch?v=3ayxEyGMbd4


Containers
---
See full list: 
https://www.youtube.com/channel/UCmYAQDZIQGm_kPvemBc_qwg/search?query=container

Dan Walsh: Latest Container Technologies
Youtube: https://www.youtube.com/watch?v=I0cOn1psf5o

Antonio Murdaca: Secure your Kubernetes cluster with CRI-O
Youtube: https://www.youtube.com/watch?v=xH2I5WcPM4Y


Chris Houseknecht: Ansible Container : Build Better!
Youtube: https://www.youtube.com/watch?v=TMNayhc9A9A


Patrick Uiterwijk, Randy Barlow: How Fedora's solving the container rebuild 
problem
Youtube: https://www.youtube.com/watch?v=Yc5Mq87-DuE


Miroslav Suchý: Catching Bugs In Containers
Youtube: https://www.youtube.com/watch?v=wHB7i4QaxHU


Bill Peck: Lessons Learned Implementing Builder Containers
Youtube: https://www.youtube.com/watch?v=8iRIep6dIEE


Huamin Chen, Dennis Keefe: Container Image Storage Compression and Dedupe
Youtube: https://www.youtube.com/watch?v=sB1QFfaDp0Y



Atomic Host/Workstation
---

Matthew Micene, Giuseppe Scrivano: Moving to Atomic System Containers
Youtube: https://www.youtube.com/watch?v=6ImTdrKv_Q0


Sanja Bonic and Colin Walters: You want a Clean Desktop OS? Containerize it.
Youtube: https://www.youtube.com/watch?v=a4IPWlfkJSo
Slides: 
https://fedorapeople.org/~walters/2018.01-devconf-desktopcontainers/#/


Jan Pazdziora: Minimizing workstation installation
Youtube: https://www.youtube.com/watch?v=eWoFpOoA-tE


Kalev Lember: Atomic Workstation
YouTube: https://www.youtube.com/watch?v=Yc7lvkl5atE
Slides: 
https://kalev.fedorapeople.org/slides/2018-devconf-Atomic-Workstation.pdf


Jonathan Lebon: Fearless Upgrades with Fedora Atomic Workstation
YouTube: https://www.youtube.com/watch?v=7c3GdfhWzcc
Slides: http://jlebon.com/devconf/2018/slides.pdf


Colin Walters: Hybrid image/package OS updates with rpm-ostree:
Youtube: https://www.youtube.com/watch?v=eWoFpOoA-tE
Slides: https://fedorapeople.org/~walters/2018.01-devconf/index.html

Peter Robinson: Using Fedora and OSTree for IoT
Youtube: https://www.youtube.com/watch?v=mRqV38qT-wc

Re: [atomic-devel] [Fedocal] Reminder meeting : General Atomic Community Meeting

[Colin Walters]

On Tue, Jan 30, 2018, at 5:01 AM, Sanja Bonic wrote:
> This meeting will not happen unless someone else wants to hold it - I am in
> flight at that time coming back from FOSDEM.

No worries, I think one of us can do it!  Since there's a lot going on I'd say
let's hold it anyways.

Re: [atomic-devel] Installing packages in Atomic

[Colin Walters]

On Fri, Jan 26, 2018, at 9:53 AM, Chris Negus wrote:
> I'm working on a procedure for installing RHEL Atomic on Azure. As part 
> of that, I want to install the WALinuxAgent RPM package, which is in the 
> rhel-7-server-extras-rpms repository. That repository is not enabled by 
> default. What I did was:
> 
> $ sudo atomic unlock

I don't think this step should be necessary?

> $ sudo vi /etc/yum.repos.d/redhat.repo  (and manually enabled the repo)

Hmm...would need to drill down into this one; I think we should be
enabling the extras repo but it might depend on your subscription state.

> My questions is, is there a tool for enabling repos in Atomic (yum and 
> yum-config-manager aren't there) or is editing .repo files the way to do 
> that?

Just editing .repo files.  I honestly never used yum-config-manager, but
I definitely run into people who have.  It also seems to have been sort
of dropped with the dnf transition too?The thing that *does* trip
everything in the rpm-ostree model up is having RPMs that themselves
contain .repo files; the canonical example is epel-release.  Personally I just
rpm2cpio it or use Ansible to inject the GPG key and repo file, but down
the line I'd like to streamline things for the "RPMs that just have files in 
/etc"
case.

[atomic-devel] In-flight AH size reduction changes

[Colin Walters]

I just want to highlight some "in flight" work on reducing the
on-disk footprint of Atomic Host (and also Fedora userspace
in general to some degree).

https://src.fedoraproject.org/rpms/openssl/pull-request/1
^ in rawhide, may try to nag maintainer to backport to f27
This one isn't *large* per se but IMO it's very symbolic
(hopefully no one is depending on `make` being on the host
 by default)

https://src.fedoraproject.org/rpms/glibc/pull-request/3
^ ditto; and related to that one is:
https://pagure.io/fedora-atomic/pull-request/101

The biggest item here though is:
https://bugzilla.redhat.com/show_bug.cgi?id=1526191
And there you can see links to two pull requests for SELinux policy
that are pending review.

Related to that is:
https://pagure.io/fedora-atomic/pull-request/102

Big picture most of this is sitting in rawhide or in pending
pull requests, but we *are* working on size issues.

Re: [atomic-devel] 2wk atomic release candidate: 20180102

[Colin Walters]

On Tue, Jan 2, 2018, at 3:48 PM, Dusty Mabe wrote:
>
>   ostree 2017.13-3.fc27.x86_64 -> 2017.15-1.fc27.x86_64

Notably this has
https://src.fedoraproject.org/rpms/ostree/c/7198ec08c451ff394024709ad2f7d3df01ae5329?branch=master
related to:
https://pagure.io/atomic-wg/issue/405

Re: [atomic-devel] Proposing a new way to deliver Atomic systems: rpm-ostree jigdo ♲

[Colin Walters]

On Wed, Dec 6, 2017, at 01:11 PM, Colin Walters wrote:

> Basically, we're reviving an old idea for the modern age of images; I'm 
> calling
> it "rpm-ostree jigdo ♲" (emoji are for "recycle package"):
> 
> https://github.com/projectatomic/rpm-ostree/issues/1081

I wrote up a demo for this with a gist and asciinema:

https://gist.github.com/cgwalters/b376302fdb3b661e8a3de3264706b553

[atomic-devel] Proposing a new way to deliver Atomic systems: rpm-ostree jigdo ♲

[Colin Walters]

I've been working on Project Atomic for several years now; first post:

https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2014-April/msg0.html

(And the rpm-ostree/ostree projects predate that; rpm-ostree's "gitbirthday" is
 coming up at Sat Dec 21 19:41:30 2013 -0500)

This entire time we've dealt with 3 binary formats: RPM, Docker (now OCI) and 
OSTree.
The tension is...powerful.   It leaks through into how people manage systems; 
issues
like "how do I mirror this content" are serious blockers for lots of 
organizations
that don't want their systems Intenet connected.  Obviously I've thought about 
this a lot,
and I now have a concrete proposal now to try to get that closer to two formats.

Basically, we're reviving an old idea for the modern age of images; I'm calling
it "rpm-ostree jigdo ♲" (emoji are for "recycle package"):

https://github.com/projectatomic/rpm-ostree/issues/1081

I won't repaste it all here - just the header, which follows.  Feel free though
to follow up here - email works well for this sort of thing.  The high level
status is: things are moving quickly, and the next step is to start fleshing out
the client side.  So it's at the point where I want concrete feedback.

Upstream issue follows below:

Fetching content via both ostree and libdnf ends up mixing the tradeoffs of 
both, requires release engineering to manage both, and makes it harder to 
mirror content. Not to mention the fact that there's the whole OCI/Docker thing 
which also works in a completely different way, and admins need to 
manage/mirror that too.

Now while the "obvious" thing would be to try to align with OCI in some way, 
the complete lack of wire deltas there is very problematic for uses outside of 
server clusters, and given that we already have lots of extensive linkage to 
RPM via libdnf, it makes the most sense to move that direction.

Hence, let's experiment with doing ostree-in-RPM ... [ read more at 
https://github.com/projectatomic/rpm-ostree/issues/1081 ]

Re: [atomic-devel] Atomic Host for ARM

[Colin Walters]

On Fri, Nov 10, 2017, at 03:53 PM, Iván Chavero wrote:
>
> The Cubie is Cortex-A7 (ARMv7-A) 32bits.

We don't currently have any media for this; Fedora Atomic Host
is currently more targeted for non-embedded servers.  That said
I know it's been done in the past - anywhere the general Fedora
packageset works, it should be pretty straightforward to build
something that works like Atomic Host.  Though I suspect a lot of people in 
this area
want to do something custom - perhaps not even use containers.
Like kiosks or UAVs.  There's a lot of interest in ostree upstream
for these types of use cases.  That path is more "custom rpm-ostree compose"
for which we have some docs scattered around; it's not hard, but
does require some investment.  There's been some work we've landed upstream to 
expand
on this, see the latest release:
https://github.com/projectatomic/rpm-ostree/releases/tag/v2017.10

[atomic-devel] Minutes for Fedora Atomic meeting 2017115

[Colin Walters]

Minutes: 
https://meetbot.fedoraproject.org/fedora-meeting-1/2017-11-15/fedora_atomic_wg.2017-11-15-16.30.html
Minutes (text): 
https://meetbot.fedoraproject.org/fedora-meeting-1/2017-11-15/fedora_atomic_wg.2017-11-15-16.30.txt
Log: 
https://meetbot.fedoraproject.org/fedora-meeting-1/2017-11-15/fedora_atomic_wg.2017-11-15-16.30.log.html

Re: [atomic-devel] please help us test atomic workstation/host images for f27

[Colin Walters]

On Sat, Nov 11, 2017, at 07:00 PM, William Brown wrote:
>
> 1) opensc should be part of the base image as it enables freeipa
> smartcard authentication and other related parts to work correctly (I'm
> layering it in for now 

Me too, in my case for yubikey.   That said my opinion here is that we simply 
cannot
meet the conflicting goals of "small base" and "full functionality" without
involving some package layering particularly for individual machines.  I think
the package layering is what makes the system fully practical today.   I also
have `vagrant-libvirt` and `origin-clients` layered, as well as a few random
things like `xsel`, `powerline`, and `git-evtag`.

(On my home Atomic Host server I also have`libvirt` layered, which is
 a big topic in itself)

That said I think it's really interesting to have this debate; the transition
from "no layering" to "some layering" is a large one (suddenly e.g. you
need to download the 40MB repodata just to check if you have an update).
And rpm-ostree is very "loud" in the `status` command about exactly
what you have layered - in large contrast to every other "package 
classification"
in Fedora like comps, which tends to just dissolve into the "bag of packages"
model.

I think we'll need to take some of this to the Workstation list - there's
a lot of bigger picture questions around how much the default set resembles
the default workstation.  For example, IMO we shouldn't include many apps
at all by default, since it's confusing to have both a flatpak version and a 
built-in.
But there are deep issues there - if we go to the point of not having a browser
by default for example, that'll create bootstrapping issues for some people.

Personally though I think the most important thing is to make the "tools/dev"
container flow more friendly.  That's something people can do without
going all the way to Atomic Workstation.

Re: [atomic-devel] Add Nvidia GPU to Centos Atomic Node

[Colin Walters]

On Wed, Nov 8, 2017, at 02:29 PM, Stephen Milner wrote:

> What about literally providing dkms in a container? It wouldn't
> directly take care of the original request but it would give an avenue
> for folks to use the dkms toolchain if they'd like to in AH without
> package layering.

Maybe?  Are we trying to replace dkms for yum-managed systems too?
I'm a bit worried about the sustainability of adding a whole other model
to dkms without trying to delete one at the same time.   That said,
it might not be *too* hard to get a PoC here.

Offhand I feel like focusing on cri-o/kube in-syscontainers is more important?
Just today I saw from IRC chat:
https://github.com/kubernetes/contrib/commit/9bef0bee1c4881469fbd1688b6e6c73a30ebb8d9

Yet there's no
https://github.com/kubernetes-incubator/kubespray
for any *AH =(  (Or for that matter Fedora...I just glanced at it
and it looks like it's python2, which brings us to
https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2017-August/msg1.html
)

Though someone who's doing e.g. OpenShift-on-AH and
wants to do GPGPU/ML using Nvidia GPUs in their cluster
might have a different perspective!

Re: [atomic-devel] Add Nvidia GPU to Centos Atomic Node

[Colin Walters]

On Wed, Nov 8, 2017, at 01:58 PM, Colin Walters wrote:
> oreos-nvidia approach
> is basically "dkms in a container" 

Nope, I looked at this more and I'm wrong - it's building
a container matching each CoreOS release in Travis and
pushing that.  So if that build somehow fails as the host updates, 
the systemd unit will fail on boot.

This approach would be *quite* tricky for Fedora, it
really gets back to 
https://lwn.net/Articles/734789/

Re: [atomic-devel] Add Nvidia GPU to Centos Atomic Node

[Colin Walters]

On Wed, Nov 8, 2017, at 12:44 PM, Stephen Milner wrote:
> On Wed, Nov 8, 2017 at 11:33 AM, Dusty Mabe  wrote:
> 
> > Really you have two options.
> >
> > 1. unlock your ostree and install the rpms
> > 2. build the kernel module for the kernel you are targeting and deliver via 
> > a system
> >container like in https://github.com/giuseppe/hellomod
> 
> I like the idea of using system containers as a way of providing
> kernel modules like these. Since system containers are not meant to be
> scheduled they end up working more like a package which fits the bill.

I think a bifurcation here is probably around CentOS vs Fedora/other -
CentOS gets a lot of the RHEL kABI, so it can work OK to just carry
forward a binary module.  Could that module live in a container?
Probably, though it's not clear to me what that'd gain over
dropping it in /usr/local/lib/modules or whatever?  We do
have container signing at least?

But the reason dkms exists is for the Fedora and other case, to
at least *try* to build from source when the host kernel is upgraded.
That gets more into the "needs to be integrated with host updates"
case.

The https://github.com/src-d/coreos-nvidia approach
is basically "dkms in a container" which seems like a sane short
term, but the problem with it is you have to wait on bootup
for the module to build, and if it fails to build, you only find
out on boot.

(I'll be honest I am mostly just talking about this having
 glanced at README.md files and a little bit of code, I haven't
 actually used a 3rd party kernel module in a long time; I could
 be partially or completely wrong about some of the details)

Re: [atomic-devel] Add Nvidia GPU to Centos Atomic Node

[Colin Walters]

On Wed, Nov 8, 2017, at 12:36 PM, Dusty Mabe wrote:
> 
> 
> On 11/08/2017 12:06 PM, Colin Walters wrote:
> > On Wed, Nov 8, 2017, at 11:33 AM, Dusty Mabe wrote:
> >>
> >> - ostree admin unlock --hotfix
> >> - alias yum='/usr/share/yum-cli/yummain.py'
> > 
> > Let's please not mention `--hotfix`? 
> 
> So the more official answer is: use traditional CentOS

IMO, yep!   This type of thing is a lot of the reason
why we need two models for a while. 

 But just diving into this a bit - while we haven't yet plumbed through
automatic updates into rpm-ostree by default yet,
we finally added `rpm-ostree upgrade --download-only`
which does a *lot* of this.  

And I'd really like to flesh out the background/automatic
updates story more and get to where we can turn it
on by default.

Yet things like dkms have a lot of  hooks into system
updates.  I just looked at the implementation slightly
more closely and it actually forks off a daemon helper
right in the middle of a traditional yum update.

One of the promises we'd make once we enable
background host updates by default is that they *don't touch*
your running OS; we wouldn't try to reload kernel modules
for example if we picked up a newer nvidia driver RPM.
This conflicts directly with the traditional yum/apt
model of trying to apply updates live, and I think things
get really interesting here with dkms as of course the
kernel isn't updated live, but other bits of userspace
like the "kernel-devel" is.

Oh, and another big thing is that our upgrades are
*cancellable*, you can Ctrl-C them at any time.  This
ties in with the "background" story - if we have automatic
updates enabled, you still want `rpm-ostree install`
to preempt that.  If DKMS is going off and starting its
own systemd service making changes that's going to be way harder.

How hard would it be to make dkms play nicely with
this?  I don't honestly know.

Re: [atomic-devel] Add Nvidia GPU to Centos Atomic Node

[Colin Walters]

On Wed, Nov 8, 2017, at 11:33 AM, Dusty Mabe wrote:
> 
> - ostree admin unlock --hotfix
> - alias yum='/usr/share/yum-cli/yummain.py'

Let's please not mention `--hotfix`?  The reason this
breaks is I'm pretty sure DKMS would need a lot of
work to properly interact with the underlying ostree
"snapshots" and in particular if we get to supporting
automatic updates.  Not to mention things like
requiring pulling in kernel headers, gcc etc. to the
host system.  That stuff ideally lives in a container
(which would benefit yum-managed systems as well).

I'm not quite sure when I'll get a chance to look at this,
but we'll get to it eventually.  If anyone else is going to
look at it please ping upstream.

Re: [atomic-devel] PXE Booting Diskless Atomic - Failed to start switchroot

[Colin Walters]

Hi Matt,

On Mon, Nov 6, 2017, at 08:50 PM, Matt Jansen wrote:
> Hi all,
> I'm trying to use iPXE to boot into a live atomic host. 

Basically at the moment we've put work on "live/diskless" AH on hold
because it kind of conflicts with management tools (e.g. Ansible no
longer works persistently), and confuses the release engineering story (because
in case of critical async errata, we'd *always* need to respin
images too instead of just rpm/ostree updates).

But see also 
http://blog-jkonecny.rhcloud.com/2017/06/16/shining-new-anaconda-modularisation/#comment-3366641149
As well as https://github.com/gbraad/minishift-atomic-iso/issues/1

It's not unlikely we'll pick this back up at some point - most of the
pieces are there to build it, but there'd still be open questions
around e.g. ensuring openshift-ansible does something sensible.

Re: [atomic-devel] a better place for system container images?

[Colin Walters]

On Mon, Nov 6, 2017, at 03:57 AM, Giuseppe Scrivano wrote:

> The goal is to build the images automatically on every PR merged.
> Occasional builds (maybe daily?) will prevent to miss changes in the
> base layers or in the installed rpms.

Let's write down these requirements/requests in an issue for Fedora rel-eng to
get fixed?  We could probably help with that too.
I'm sure there are other containers that want something like this.

What we have now for OSBS is pretty slow/cumbersome for sure, but
it can be fixed.

I don't mind doing this as a short term tactical thing, but beyond
what Dusty is saying, other issues are around multi-arch builds, signing,
ensuring the source code is available, etc.

Re: [atomic-devel] please help us test docker images from f27 RC 1.2

[Colin Walters]

On Wed, Nov 1, 2017, at 10:05 PM, Dusty Mabe wrote:

> https://kojipkgs.fedoraproject.org/compose/27/Fedora-27-20171101.0/compose/Docker/aarch64/images/Fedora-Docker-Base-27-1.2.aarch64.tar.xz
> https://kojipkgs.fedoraproject.org/compose/27/Fedora-27-20171101.0/compose/Docker/armhfp/images/Fedora-Docker-Base-27-1.2.armhfp.tar.xz
> https://kojipkgs.fedoraproject.org/compose/27/Fedora-27-20171101.0/compose/Docker/ppc64le/images/Fedora-Docker-Base-27-1.2.ppc64le.tar.xz
> https://kojipkgs.fedoraproject.org/compose/27/Fedora-27-20171101.0/compose/Docker/x86_64/images/Fedora-Docker-Base-27-1.2.x86_64.tar.xz

When do these get pushed to registry.fedoraproject.org/fedora:27 ?

Currently that is:
"Created": "2017-10-17T12:36:31Z",

[atomic-devel] docs hackfest...next steps?

[Colin Walters]

Hey, so we had a docs hackfest at Flock, which was great.
I was very inspired.  Honestly I am terrible about writing
docs, and there's many days I feel like we'd be in a lot
better place if we were better about this.

So the result of the hackfest was:
https://github.com/projectatomic/atomic-host-docs/pulls

Which is stalled out.  One issue is this:
https://github.com/projectatomic/atomic-host-docs/pull/89#discussion_r136595389

I just commented on the linked issue there.

We have since the start had a deep problem around docs for Project Atomic
versus the different distributions and versioning (i.e. what commands/feature
sets are available).  I think my vote there is to just plow ahead and for
things like the AH partition layout change
https://pagure.io/atomic-wg/issue/281
we just describe both?

(Of course this itself overlaps with 
https://docs-old.fedoraproject.org/en-US/Fedora/26/html/Installation_Guide/index.html
 etc)

Re: [atomic-devel] Atomic and NIST-800/STIG compliance

[Colin Walters]

On Thu, Sep 7, 2017, at 02:51 PM, Ryan Barry wrote:
> 
> I'd imagine the same. That said, oVirt Node is also not managed by
> yum, and the specific request there was still to have separate
> filesystems. The theory being that a runaway process or attacker
> which/who fills one of the partitions can't lock out the entire system
> or block logging of activities.> 
> In that sense, I suspect that most of the guidelines still apply.

Right, I'd agree.

> 
> We did something similar in vintage oVirt Node, so I completely
> understand -- we saw a large number of problems with some system
> utilities not playing nice with bind mounts and redirection/symlinks
> on a ro root, but that's a definite tangent.
The only bind mount we have by default is var; the only side
effect I can think of for this is
https://github.com/ostreedev/ostree/issues/960

> >> In my testing, Atomic seems to only take ~3GB of the volume group
> >> when installed,
That's going to change; it already has for FAH (this is why I
tend to useabbreviations like FAH, CAH, and RHELAH since there isn't just one
"Atomic").See https://pagure.io/atomic-wg/issue/281
https://bugzilla.redhat.com/show_bug.cgi?id=1391725

> Updating the document is certainly a solution, though I expect that
> /var/log and /var/log/audit will need to be separate volumes, at
> least. I would also not be surprised to see that /opt and /home needed
> to be also, so docker containers with VOLUME could not maliciously
> fill those and simultaneously fill /var> 
> Is this something we can help with?

I just tested this with 
Fedora-Atomic-ostree-x86_64-26-20170821.0.iso
and it worked fine to make `/var/log` and `/var/log/audit` mount points.
> 
> Outside of that, "full" STIG compliance (minus bootloader locking and
> some other finicky bits -- cloud-init can handle most of the edge
> cases for users who want the whole shebang) is another target. I've
> seen some work on openscap for Atomic, but I haven't tried a run yet
> to see what the status is...
What do you mean by bootloader locking?  Secure boot?  Or
having /boot be read-only?  Something else

Hmm, are you talking about cloud-init on bare metal?  We were
playing withthat but didn't really "productize" it.

Re: [atomic-devel] should we include microdnf in atomic host

[Colin Walters]

On Fri, Aug 25, 2017, at 09:52 AM, Dusty Mabe wrote:
> If there is ever a case where people "need" to `ostree admin unlock --hotfix`
> and install a new patched rpm that hasn't been released yet, should we 
> consider
> including microdnf to make it easier to grab those rpms? 

In general, rpm-ostree already has all of the code from libdnf; microdnf just
wraps libdnf.   So anything we're not supporting is really a bug; right?

Isn't your case already mostly covered by
# for x in 
https://kojipkgs.fedoraproject.org//packages/docker/1.13.1/28.gitb5e3294.fc27/x86_64/docker-{common-,rhel-push-plugin-,}1.13.1-28.gitb5e3294.fc27.x86_64.rpm;
 do curl -L -O $x; done
# rpm-ostree ex override replace ./docker-*
...
# rpm-ostree status
State: idle
Deployments:
  fedora-atomic:fedora/26/x86_64/atomic-host
   Version: 26.110 (2017-08-20 18:10:09)
BaseCommit: 
13ed0f241c9945fd5253689ccd081b5478e5841a71909020e719437bbeb74424
Commit: 
9563ce634fc49bb5efbb6a75d8f6dc4129f154caa9a63a72cca66b4b02a6b5e8
  ReplacedBasePackages: docker-rhel-push-plugin 2:1.13.1-21.git27e468e.fc26 
-> 2:1.13.1-28.gitb5e3294.fc27, docker-common 2:1.13.1-21.git27e468e.fc26 -> 
2:1.13.1-28.gitb5e3294.fc27, docker 2:1.13.1-21.git27e468e.fc26 -> 
2:1.13.1-28.gitb5e3294.fc27

etc?

The `ostree admin unlock` model does allow *arbitrary* changes (not just RPMs)
and `--hotfix` continues that, but it sounds like you were mainly looking at 
RPMs,
right?

Re: [atomic-devel] Atomic and NIST-800/STIG compliance

[Colin Walters]

Hi Ryan, thanks for bringing this up!

On Wed, Aug 23, 2017, at 03:31 PM, Ryan Barry wrote:
> 
> /home
> /opt
> /var
> /var/log
> /var/log/audit

As I understand it, the NIST-800 specification was designed for a "traditional"
system managed via yum.  I imagine they did a default RHEL7 install and
looked at hardening on top of that.

The ostree model is in contrast *very* prescriptive about the system layout.
It's different from how things worked before, but I think still compatible
enough; and worth the change.
There's some more info here:
https://ostree.readthedocs.io/en/latest/manual/adapting-existing/
But this boils down to: every mount point you've listed above actually lives
underneath /var.  Stated even more strongly, *all local state* lives under
/var or /etc, and everything else appears immutable (though rpm-ostree can
mutate it).

> 
> (ideally with any 'persistent' data like the rpmdb relocated off of /var,

Indeed:

```
$ rpm-ostree status
State: idle
Deployments:
  atomicws:fedora/x86_64/workstation
   Version: 26.64 (2017-08-23 07:44:49)
    BaseCommit: 
73f5c6bfa0365f4170921b95ae420439f2f904564c7bdb12680dc1c8ddd47986
$ ls -al /var/lib/rpm
lrwxrwxrwx. 1 root root 19 Apr 18 15:29 /var/lib/rpm -> ../../usr/share/rpm
```

>  with the contents of /var[/*] being the same across all ostree instances, so 
> logs are not lost if users need to roll back).

Yep, that's also enforced.  ostree (and rpm-ostree) will never touch /var, 
whether
upgrading or rolling back.  The only twist here is that we generate systemd 
tmpfiles.d
entries - so for example `rpm-ostree install postgresql` will cause some 
directories
to be generated on the *next* boot.

Related to all of this, rpm-ostree for example will cleanly reject
installing RPMs that do things like have content in /home:
https://github.com/projectatomic/rpm-ostree/issues/233

> In my testing, Atomic seems to only take ~3GB of the volume group when 
> installed, though I understand that the remainder of the volume group is 
> often used for Docker image storage. We performed a conversion to a NIST-800 
> layout as part of an update on oVirt Node, but we were fortunate enough to be 
> using lvmthin, so we didn't need to worry too much about it, but I'm not sure 
> how this would be done on Atomic. I know that /var was added recently, so 
> some shuffling must be possible, but I haven't looked into the details of how 
> that was performed.

Yep, it's ironic that it took so long for 
https://github.com/ostreedev/ostree/pull/859
to land - it really helps complete the picture of having basically all
local system state easily put onto separate storage, consistently backed up, 
etc.

> Additionally, getting as close as possible to full STIG compliance would be 
> ideal. I see that atomic supports scanning containers running on Atomic 
> hosts, but I'm not sure what the actual status of the host itself is.

I think we can do that more easily now that the above PR landed; however,
it seems likely to me that the guidelines need to be updated to account
for things like /home → /var/home.

Also unfortunately for RHEL7, systemd refusing to mount on symlinks
is fairly problematic.  See: https://github.com/systemd/systemd/pull/6293
But administrators can work around that by using /var/home as the mount point.

Anyways, so I think the current F26AH's support for /var as a mount
point covers most of this; the tricky part is probably updating the document
to account for the ostree/AH differences?

Re: [atomic-devel] RC hardware test -- Fail

[Colin Walters]

On Mon, Aug 21, 2017, at 10:23 PM, Josh Berkus wrote:
> On 08/21/2017 04:17 PM, Josh Berkus wrote:
> > Dusty,
> > 
> > https://kojipkgs.fedoraproject.org/compose/twoweek/Fedora-Atomic-26-20170821.0/compose/Atomic/x86_64/iso/Fedora-Atomic-ostree-x86_64-26-20170821.0.iso
> > 
> > Passed the minnowboard install test.\
> > 
> > Some new warnings, about null bytes in the image shell script.  Do we care?
> > 
> 
> Reported too soon.  The image installs fine, but won't boot once installed.

Is the symptom similar to https://bugzilla.redhat.com/show_bug.cgi?id=1482922 ?

[atomic-devel] Added microcode_ctl to FAH, tuned discussion

[Colin Walters]

Hey, so previously we tended to discuss Atomic Host content
changes on this list, but since then the
https://pagure.io/atomic-wg/
tracker was created and has been used more.
See https://pagure.io/atomic-wg/issue/315
for tuned.

See this PR for why the PR to add microcode_ctl was pushed:
https://pagure.io/fedora-atomic/pull-request/77

[atomic-devel] Why Atomic Host should be built using Modularity

[Colin Walters]

There was a discussion today in the Atomic WG about using Modules.
Meeting log: 
https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2017-August/msg4.html
Agenda discussion: 
https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2017-August/msg2.html
(Side note; this doc was originally stored at 
)

This post is the "why" that I'd written earlier.  Some of the rationale
here crosscuts with other active discussions; for example, keeping
Python 2 in Atomic Host, as well as Bodhi and update cadences.

There’s an effort in Fedora called
[“Modularity”](https://docs.pagure.org/modularity/) that effectively gives
multiple lifecycles for different components.

Today (and since the start), Fedora Atomic Host *is* Fedora - we have just added
another build process at the end of the "Everything" build process (and the same
is true of the `registry.fedoraproject.org/fedora` container). There's a lot of
benefits to that; any security fixes to e.g. `glibc` or `openssl` automatically
propagate.

Content streams
---

But on the other hand, Fedora Atomic Host is locked into the current Fedora
cycles; the freezes, then the "open the floodgates" model of Bodhi.

The "at most once a day updates" that Bodhi implements is quite suboptimal
for the server case.  We go to quite a bit of effort to implement a "two
week" cadence for the ostree updates, distinct from whatever happens in Bodhi
day to day.

Further, we really have no idea what might land in Bodhi day to day.  There
are a lot of updates that are basically non-critical, and should ideally
be batched together.  Things like `rsync`.

Another issue is that we currently don't have any ability to do *overrides* 
when we need
to. If a package gets karma or the maintainer decides to ship it anyways, that's
basically it. Yes, ideally we don't have overrides, but the reality is we need
the ability, we just want to avoid using it in any long term capacity.


Moving towards a single stream
--

One of the major benefits of the rpm-ostree technology is that there's
no special distinction between minor and major upgrades, as we've ended
up with between `yum upgrade` and `dnf system-upgrade`.  The underlying
ostree side is "image like" - we could easily do changes on the scale of
[UsrMove](https://fedoraproject.org/wiki/Features/UsrMove) without requiring
any special work or downtime, in the same way we can do small incremental 
updates
atomically.  For example, imagine that we decided to switch to
the [Debian Multiarch](https://wiki.debian.org/Multiarch) layout.  That'd
just be another atomic upgrade (with rollback, etc.)

See [this issue](https://pagure.io/atomic-wg/issue/228) for more background
discussion on this topic.

We'd like "Fedora Atomic Host" to simply be a single stream for most users.
This implies more than just having a unified update mechanism; we also
need to ensure that any Changes that land in "Fedora" are minimally
disruptive for the server case.

Good examples here are:

 - 
https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/ZNQW72UP36UAFMX53HPFFQTWTQDZVJ3M/
   We'd probably take this change, but delay it quite a while.  It makes
   sense for desktops, and as an opt-in for server admins who want it.
 - Dropping /usr/bin/python: 
https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/JO4WGEQVGRJMLNDLMZURHIQEGPXUZCOH/
   This badly affects Ansible (which is quite important for Atomic Host)

Now, a lot of how this works comes down to how quickly Modularity
takes over Fedora and the changes in general.  We're not going to get
out of being able to make changes like the above; but I think
we want more control over when they land - for example, allow
Workstation to actually do something earlier.

Basically, we want to create a minimal server OS focused
on container hosting which you can install to e.g. a
bare metal machine, and upgrade seamlessly in place
without any special work for years.

How many years?  This proposal doesn't address that,
but it certainly seems to me like we could easily handle
2-3 years without major breaking changes.

Obviously again, a lot of this relies on containerization to be
truly successful.  (And containerization in turn really requires
Modularity too!)

One important aspect of this the "system containers" effort to
move even things like Kubernetes into containers that are decoupled from
the host lifecycle.  We'd likely have Kubernetes containers as modules,
with likely at least 2 major versions available.

Development benefits for AH
---

We want to minimize the number of versions of our critical path
components that we maintain; think `docker`, `ostree` etc.  Now,
perhaps in a Modularity world we would end up offering multiple
`docker` RPMs (and containers), but I don't see a reason to maintain
multiple 

[atomic-devel] Python 2 and Atomic Host

[Colin Walters]

Hey, just a quick note here that I've been waging a fight to keep
/usr/bin/python as Python 2 for Fedora Atomic Host, and in
general to support Ansible.

This was covered on LWN:
https://lwn.net/Articles/729366/

For openshift-ansible I think we're generally OK with supporting
Python 3 and working through that.  But if you're a user of
Atomic Host + Ansible with Python 2, it'd be useful to respond
here to let me know this fight is worth continuing =)

There's a bigger picture question around how long we keep
Python 2.  I'll post something related to that soon.

Re: [atomic-devel] 2wk atomic release candidate: 20170723

[Colin Walters]

On Mon, Jul 24, 2017, at 10:50 AM, Dusty Mabe wrote:
>
>   ostree 2017.7-2.fc26.x86_64 -> 2017.8-3.fc26.x86_64
>   ostree-grub2 2017.7-2.fc26.x86_64 -> 2017.8-3.fc26.x86_64
>   ostree-libs 2017.7-2.fc26.x86_64 -> 2017.8-3.fc26.x86_64

This also switches ostree to libcurl (which is why libsoup dropped out).
Among other things we now speak HTTP/2.  But it's a pretty different
HTTP stack, so if anyone sees anything unusual with `atomic host upgrade` and
HTTP, this could be the cause.

Re: [atomic-devel] Fedora Atomic Host Two Week Release Announcement

[Colin Walters]

On Tue, Jul 11, 2017, at 12:28 PM, Dusty Mabe wrote:
>
> We will try to release blogs and documentation over the next few weeks
> to show you new features and guide how to upgrade your existing hosts.

We have this wiki page for the latter, I've updated it now:
https://fedoraproject.org/wiki/Atomic_Host_upgrade

I also sent a mail to get this linked from 
https://fedoramagazine.org/upgrading-fedora-25-fedora-26/

Re: [atomic-devel] Moving https://github.com/cgwalters/projectatomic-ci-infra into the org

[Colin Walters]

On Thu, Jun 22, 2017, at 12:20 PM, Colin Walters wrote:
> Hey,
> 
> I propose moving:
> https://github.com/cgwalters/projectatomic-ci-infra
> into our org, and renaming it to paci.  This follows on from the
> move of PAPR (formerly redhat-ci) into the org.
> 
> See: https://github.com/projectatomic/papr/issues/29#issuecomment-304020880
> 
> The main content of this repository is currently *just* our instance of
> Homu, but it's likely that we'll add other OpenShift/Kube-ized services.

Since there were no objections, I've done the transfer.

Re: [atomic-devel] gpg2 warning with gpg verification enabled on atomic host

[Colin Walters]

On Sun, Jun 25, 2017, at 07:35 AM, Dusty Mabe wrote:
> Saw this today. I was on the serial console so kernel messages were coming to 
> the console.
> Not sure if it is an issue or can safely be ignored.
> 
> [root@localhost ~]# rpm-ostree status 
> [   23.266519] gpg2 (1095) used greatest stack depth: 10704 bytes lef

That's normal, it comes from CONFIG_DEBUG_STACK_USAGE.
See https://lwn.net/Articles/600644/
and https://www.google.com/search?q="used+greatest+stack+depth;

Re: [atomic-devel] Changing partitioning defaults discussion

[Colin Walters]

On Mon, Jun 5, 2017, at 03:12 PM, Dusty Mabe wrote:
> 
> - Why not stick with the distro default? 

Well, what is "distro default"?  This is a messy topic; some things
*are* defaults in blivet, and apparently the default filesystem is ext4,
but for Fedora Server it's overridden.  I'm honestly not sure a lot
of serious thought is put into those blivet defaults, it seems to me
to be a lot more like inertia.

I've attached an updated patch which links to the Server thread.
For us, the argument that it's the RHEL default also carries weight,
but for containers, it's what we test with the most downstream.  

A good specific technical advantage XFS has over ext4 for this
use case (with overlayfs) is that inodes are dynamic, and when
you're doing containers you simply tend to have more filesystem
trees.

> - If we have a good reason can we do this in a different commit with a 
>   commit message that is relevant?

I updated the existing commit message, I think the "match Fedora Server"
rationale should cover it, right?
From 430a8722acd55c93d8794e93c54c992e2c8a7ee7 Mon Sep 17 00:00:00 2001
From: Colin Walters <walt...@verbum.org>
Date: Mon, 5 Jun 2017 12:46:40 -0400
Subject: [PATCH] Match Fedora Server's partitioning

The max size of 3GB is pretty restrictive for most bare metal installs where we
expect peopl to have at least one large physical drive.  This gives
a lot more headroom for things that may be stored in `/` like e.g. the systemd
journal.

The max size of `15GB` ensures that if the user wants to make other LVs, there's
enough space in the VG to do so.

Note this deletes the `/boot` override; while I don't like the new Blivet
default of `1G`, it's a messy topic, and I don't think it's worth carrying
an override in the baremetal path just for that.

Note we also explicitly start matching the Fedora Server default of XFS,
with the same rationale; there's a [long thread](https://lists.fedoraproject.org/archives/list/ser...@lists.fedoraproject.org/thread/W4KWEEKADZDCAFK4XJYEJG34DZLTC3BB/)

For AH, we've been using XFS by default, this just ensures it's used
across all filesystems by default.

One specific reason why XFS is good over ext4 is support for dynamic inodes is
really quite nice when using `overlayfs` and containers.
---
 installclass_atomic.py | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/installclass_atomic.py b/installclass_atomic.py
index cce1470..c3677ca 100644
--- a/installclass_atomic.py
+++ b/installclass_atomic.py
@@ -36,11 +36,13 @@ class AtomicInstallClass(FedoraBaseInstallClass):
 name = "Atomic Host"
 sortPriority = 11000
 hidden = False
+defaultFS = "xfs"
 
+# This is intended right now to match Fedora Server; if changing this,
+# please discuss on https://lists.projectatomic.io/projectatomic-archives/atomic-devel/
 def setDefaultPartitioning(self, storage):
-# 3GB is obviously arbitrary, but we have to pick some default.
 autorequests = [PartSpec(mountpoint="/", fstype=storage.default_fstype,
-size=Size("1GiB"), max_size=Size("3GiB"),
+size=Size("3GiB"), max_size=Size("15GiB"),
 grow=True, lv=True)]
 
 bootreqs = platform.set_default_partitioning()
@@ -56,7 +58,6 @@ class AtomicInstallClass(FedoraBaseInstallClass):
 if autoreq.fstype is None:
 if autoreq.mountpoint == "/boot":
 autoreq.fstype = storage.default_boot_fstype
-autoreq.size = Size("300MiB")
 else:
 autoreq.fstype = storage.default_fstype
 
-- 
2.9.4

[atomic-devel] Changing partitioning defaults discussion

[Colin Walters]

As a followup to the previous thread on storage, also
to try to fix this BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=1391725

I'd like to revisit partitioning again.  There are a number of
intersecting factors that have changed since things were introduced;
the biggest is by far the introduction of overlay2.  The second is
that system containers may use the ostree repo, but that's stored
in /.   (The advantage of that is the container content can be deduped
with the host, the downside is their storage is entangled and needs to
be budgeted together)

Partitioning I think is going to bifurcate a bit around physical versus
virtual/cloud. 

I'd like to propose the attached patch for immediate commit to rawhide
which causes us to match Fedora Server for physical installs.  See the
commit message for rationale.

For people doing physical installs, they have a choice and
it seems very likely to me that they want to customize things anyways.
(For example, some people will want dm-crypt, and the fact that
 ostree supports this is a definite feature)

Where things get more interesting is the virtual/cloud area.  Here,
we choose/hardcode partitioning.  At a technical level, most of it lives here:
https://pagure.io/fedora-kickstarts/blob/master/f/fedora-atomic.ks#_25

Except that one aspect that I'd forgotten about that's really
critical when thinking about the kickstart numbers, which is
the *total size* of the disk.  It turns out this is defined in the Fedora Pungi
configs.   And what tricked me even more is that we have different
defaults for the qcow2 vs vagrant:

https://pagure.io/pungi-fedora/blob/6d898b136b6f612aa197a223155e79a667432ef8/f/fedora-atomic.conf#_227
https://pagure.io/pungi-fedora/blob/6d898b136b6f612aa197a223155e79a667432ef8/f/fedora-atomic.conf#_247

The qcow2 version is also what ends up in the AMI.  Now, the rationale
for this separation I think is that (AIUI) very common for operations
people to use separately allocated disks (EBS volumes etc) for storage
of things they care about.  Concretely, it's likely that Kube/OpenShift
operators are going to stick Docker storage (both LVM/overlay) in
a separate disk.

Of course, we use GROWPART=true with container-storage-setup, which
will expand the root volume group if one allocates a larger disk - this
 is also quite common to do in IaaS environments (AWS,GCE etc.) in particular.

So let me propose this:

QCOW2/AMI ("Production cloud image")
- Use overlay2 by default (Already done)
- Expand the / fully, don't reserve any space in the VG, i.e.:
```
  -logvol / --size=3000 --fstype="xfs" --name=root --vgname=atomicos
  +logvol / --size=3000 --grow --fstype="xfs" --name=root --vgname=atomicos
```

We retain the 6GB disk size default, but in practice, I expect operators to 
either allocate
larger root storage disks, or choose separate volumes.  Note that operators
who want to use devicemapper instead need separate volumes; this should
be well supported with container-storage-setup.

Vagrant box:
I suspect the rationale behind the 40GB number is that Vagrant
implementations are usually thinly provisioned, and that adding
separate disks is basically not in the default Vagrant workflow.
(Although it's certainly possible)

So let's do the same thing here as for the QCOW2/AMI, except keep
the 40GB number? 
From ccbe400f0340eda93b7d8f3c052c355dea17aebc Mon Sep 17 00:00:00 2001
From: Colin Walters <walt...@verbum.org>
Date: Mon, 5 Jun 2017 12:46:40 -0400
Subject: [PATCH] Match Fedora Server's partitioning

The max size of 3GB is pretty restrictive for most bare metal installs where we
expect peopl to have at least one large physical drive.  This gives
a lot more headroom for things that may be stored in `/` like e.g. the systemd
journal.

The max size of `15GB` ensures that if the user wants to make other LVs, there's
enough space in the VG to do so.
---
 installclass_atomic.py | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/installclass_atomic.py b/installclass_atomic.py
index cce1470..c3677ca 100644
--- a/installclass_atomic.py
+++ b/installclass_atomic.py
@@ -36,11 +36,13 @@ class AtomicInstallClass(FedoraBaseInstallClass):
 name = "Atomic Host"
 sortPriority = 11000
 hidden = False
+defaultFS = "xfs"
 
+# This is intended right now to match Fedora Server; if changing this,
+# please discuss on https://lists.projectatomic.io/projectatomic-archives/atomic-devel/
 def setDefaultPartitioning(self, storage):
-# 3GB is obviously arbitrary, but we have to pick some default.
 autorequests = [PartSpec(mountpoint="/", fstype=storage.default_fstype,
-size=Size("1GiB"), max_size=Size("3GiB"),
+size=Size("3GiB"), max_size=Size("15GiB"),
 grow=True, lv=True)]
 
 bootre

Re: [atomic-devel] Thinking about CRI-O and Docker on Atomic Host

[Colin Walters]

On Thu, Jun 1, 2017, at 12:00 PM, Clayton Coleman wrote:
> oc cluster up will likely continue to use docker for the near future.

That makes sense to me - and for that matter from my perspective
AH is going to include docker for the near future.   But my original goal
was to talk about the not-near future =)

That said, I think it's clear the primary thing we need to support
is co-existence, even in the not-near future.  While I honestly haven't
actually even tried running a CRI-O-ized
Kube locally on AH, to repeat what I said before I suspect the biggest
pain point is going to be storage.  But if we go pure overlay2 for CRI-O,
and one is using overlay2 for Docker as well, things are a lot simpler since
one could shrink the Docker LV, or even delete it and just have one big /.

So given the switch to overlay2 for Docker in AH, perhaps we simply
don't need to do anything aside from writing some demo commands
to shrink the Docker LV and give more space to CRI-O.

[atomic-devel] Thinking about CRI-O and Docker on Atomic Host

[Colin Walters]

I've seen some interesting work on CRI-O for Kube/OpenShift.  But
I'm wondering about what people are thinking the future of 
docker.service and /usr/bin/docker is (particularly for Atomic Host).

The particular intersection with AH is handling container storage;
AIUI right now you can't have CRI-O and Docker share the same
storage pool.

So...should a CRI-O installer `systemctl stop docker.service; systemctl mask 
docker.service`
and reuse the partition we've set up for /var/lib/docker?  Or do we
expect people to set up a separate partition?
(It feels like the answer is probably "both")

Perhaps in the big picture for the single-node case we point people
at `oc cluster up` etc?  (Which itself should be a system container probably
to avoid the obvious bootstrapping problem?)

Re: [atomic-devel] Openshift Origin + CRI-O

[Colin Walters]

On Wed, May 10, 2017, at 10:08 AM, Antonio Murdaca wrote:

> I've started working on the integration between Openshift Origin and
> CRI-O some time ago with nice initial results.

Nice work!

> For anyone who wants to step in and try out Openshift Origin and CRI-O, I've
> created some scripts to setup a Fedora 25 VM to be provisioned for
> Openshift Origin and CRI-O (works with 26 probably, but totally
> untested on fedora atomic hosts). 

Let me break this out since I think it's an interesting topic!  It's not a 
criticism,
but more of an architectural point.
The Ansible `dnf` module doesn't (yet) work on AH.  But even if it did,
I think installing development tools directly on a host should be considered an 
anti-pattern:

- hosts: all
  remote_user: root
  tasks:
- name: install stuff
  dnf: name={{ item }} state=latest
...
- golang
- btrfs-progs-devel

Basically, I think all builds should be done in a container.  If you then want 
to install
the *result* (e.g. an RPM) on the host, that makes sense.  On the AH side, we
now have support for local RPM install (but not yet *live* installs).  
Personally
for development/hacking I tend to use `ostree admin unlock` still with a flow
like this:

container$ sudo make install DESTDIR=/host

Although lately I've switched to only sharing /srv, so it's two steps:
container$ sudo make install DESTDIR=/srv/install
host$ rsync -rlv /srv/install/usr /usr/

The "no devel tools on the host" is also a goal of 
https://fedoraproject.org/wiki/Workstation/Atomic

Re: [atomic-devel] Many new dependencies in f25 updates-testing

[Colin Walters]

On Wed, Apr 26, 2017, at 12:57 PM, Jonathan Lebon wrote:
> I traced it down to:
> 
> http://pkgs.fedoraproject.org/cgit/rpms/atomic.git/commit/?h=f25=7d15e4a0be2db29deda4b92a039a041d81bbe205
> http://pkgs.fedoraproject.org/cgit/rpms/atomic.git/commit/?h=f25=cb845639e7388bb9aa3b5aef7dabffa3434b1ae1
> 
> I.e. atomic needs to build RPMs for the system container
> work.

I propose we revert that dependency and revisit the rpm construction
upstream.  Any objections?

Re: [atomic-devel] Many new dependencies in f25 updates-testing

[Colin Walters]

On Wed, Apr 26, 2017, at 12:04 PM, Dusty Mabe wrote:
> 
> Difference between 25.113 from fedora-atomic/25/x86_64/updates/docker-host
> and 25.119 from fedora-atomic/25/x86_64/testing/docker-host introduces
> quite a few new deps:
> 
> Upgraded:

I suspect one of these...

>   binutils-2.26.1-1.fc25.x86_64
>   deltarpm-3.6-17.fc25.x86_64
>   dnf-conf-1.1.10-6.fc25.noarch
>   dnf-plugins-core-0.1.21-5.fc25.noarch
>   dwz-0.12-2.fc24.x86_64
>   elfutils-0.168-1.fc25.x86_64
>   fpc-srpm-macros-1.0-1.fc25.noarch
>   gdb-headless-7.12.1-48.fc25.x86_64

Pulled in this.   See also
https://github.com/projectatomic/rpm-ostree/issues/718

I was debugging something similar a while ago with just raw RPM
and it took me a while to figure out the gdb → dnf-conf dependendy
was a Recommends, not a Requires.

Re: [atomic-devel] Httpd vs. Containers

[Colin Walters]

On Mon, Apr 24, 2017, at 02:18 PM, Josh Berkus wrote:
> 
> 1. Is there a *reason* we didn't relocate the HTTPD logs to Journald
> when Fedora went Systemd?

It impacts performance:
https://bugzilla.redhat.com/show_bug.cgi?id=963620

Re: [atomic-devel] firewalld in atomic host

[Colin Walters]

On Fri, Apr 21, 2017, at 10:16 AM, Dusty Mabe wrote:
> NOTE: if you respond to this message please 'reply-all'.
> 
> I'd like to discuss firewalld on atomic host. 

I think there here are two cases:

AH-as-Kube/OpenShift host:  In this I'd turn the conversation around - do
 Kube/OpenShift want to depend on firewalld?   This has come up before,
 see https://bugzilla.redhat.com/show_bug.cgi?id=1403331

Standalone/"pet" AH: I think that package layering solves
this today (and other "pet" cases), and ideally we would also provide a 
container.

Basically I don't have a definitive answer myself, but hopefully at
least the above bz link is useful.

Re: [atomic-devel] Running VMs in Openshift

[Colin Walters]

On Thu, Mar 30, 2017, at 03:34 PM, Colin Walters wrote:
> On Thu, Mar 30, 2017, at 03:04 PM, Stef Walter wrote:
> > After starting a VM in kubevirt, can access the qemu monitor or have
> > libvirt access to that VM ... from a container in a kubernetes pod?
> 
> To rephrase what Stef is saying:
> 
> First, this is mostly about using VMs for *testing*.

Actually, there is a generalization of this - situations where
running a VM is just an implementation detail of a container,
and hence they should be "lifecycle bound" together in
the same physical location.

For example in
https://cloud.google.com/security/security-design/

>  These techniques include normal Linux user separation,
> language and kernel-based sandboxes, and hardware virtualization.
> In general, we use more layers of isolation for riskier workloads;
> for example, when running complex file format converters on
> user-supplied data or when running user supplied code for products
> like Google App Engine or Google Compute Engine.

So for example if you have a video transcoding service, you might
use a VM as a processing pipeline *locally*, without having to
pay the cost of sending the data to/from a separate service.

Re: [atomic-devel] using rpm-ostree compose tree no subject

[Colin Walters]

On Thu, Mar 23, 2017, at 07:07 AM, Chen Fan wrote:

> ...



> the logging showing no subject,  is it a bug? how can I add any
> statement to this commit?


See:

https://github.com/projectatomic/rpm-ostree/issues/162



TL;DR: we actually haven't really plumbed through a story on
making commit
messages useful in general, but if you plan to use it, it's a pretty
easy patch,
I can take a look at doing it.

[atomic-devel] Status of containerizing docker and https://github.com/projectatomic/atomic-system-containers

[Colin Walters]

Hey,

Does anyone know what the status of
https://github.com/projectatomic/atomic-system-containers
is in general, and in particular I'm interested in the
"containerized docker" approach.

Can someone who knows a bit more about this add
e.g. a `README.md` with getting started instructions?

I did find https://hub.docker.com/r/gscrivano/docker-fedora/
which has some info, but it appears generic and not specific
to this container.  The tradeoffs/implementation details
of containerizing Docker in particular seem worth having
a specific doc.

(Also, that image is auto-built from github:giuseppe/atomic-oci-containers
 which is different from the projectatomic one?)

I just tried this:

```
# atomic host status
State: idle
Deployments:
● fedora-atomic:fedora-atomic/25/x86_64/docker-host
 Version: 25.80 (2017-03-13 23:35:50)
  Commit: 
24d4499420ffb2cc49681020bbe5aa6780d780d2b811eab1f5ffea6446b5a4c5
  OSName: fedora-atomic
# atomic install --system gscrivano/docker-fedora
# systemctl start docker-fedora
...
Mar 15 16:02:02 localhost.localdomain runc[4867]: container_linux.go:247: 
starting container process caused "process_linux.go:359: container init caused 
\"rootfs_linux.go:89: jailing process inside rootfs caused \\\"pivot_root 
invalid argument\\\"\""
```

Re: [atomic-devel] Buildah in projectatomic/

[Colin Walters]

On Tue, Mar 14, 2017, at 06:56 PM, Josh Berkus wrote:
> Folks,
> 
> The Buildah project (https://github.com/nalind/buildah) would like to
> move under projectatomic.

Now that we have some CI services maintained by people in this org,
part of this template should be:

 - Do you want to use https://github.com/jlebon/redhat-ci ?  (Which
   we should also consider moving under the org, though blocks on 
https://github.com/jlebon/redhat-ci/issues/29)
 - Do you want to use our Homu instance 
(https://homu-projectatomic-ci.svc.ci.openshift.org/queue/all)
which is configured to do fast-forwards so that you get both tests-on-merge
*and* your "git log" isn't the (IMO) barely legible garbage that results 
from the default
Github PR merge button when you're merging just one (or two) commits.

Re: [atomic-devel] 2wk atomic release candidate: 20170228

[Colin Walters]

On Tue, Feb 28, 2017, at 01:37 AM, Dusty Mabe wrote:

> The atomic images are here:
^ Host =)

> https://kojipkgs.fedoraproject.org/compose/twoweek/Fedora-Atomic-25-20170228.0/compose/CloudImages/x86_64/images/

Autocloud:
https://apps.fedoraproject.org/autocloud/jobs/857
Looks like it flaked.

> https://kojipkgs.fedoraproject.org/compose/twoweek/Fedora-Atomic-25-20170228.0/compose/Atomic/x86_64/iso/Fedora-Atomic-ostree-x86_64-25-20170228.0.iso

Right now for the ISO, openQA isn't really integrated into our processes, but 
one can find results here:
https://openqa.fedoraproject.org/group_overview/1
Which has:
https://openqa.fedoraproject.org/tests/overview?distri=fedora=25=Fedora-Atomic-25-20170228.0=1
So that seems OK.

Re: [atomic-devel] Fedora Atomic 2wk: Release Candidate

[Colin Walters]

On Tue, Jan 24, 2017, at 05:32 AM, Dusty Mabe wrote:

> The vm images are here:
> 
> https://kojipkgs.fedoraproject.org/compose/twoweek/Fedora-Atomic-25-20170124.1/compose/CloudImages/x86_64/images/

As a reminder, Atomic Host uses rpm-ostree (which uses ostree) for updates,
and in the future[1] hopefully the releases will be based on that, and disk 
images like
this will be a secondary artifact.

Hence, we should include the OSTree commit hash in these messages.
Here's a quick way to extract it from the disk image without booting:

$ (echo run; echo mount /dev/mapper/atomicos-root /; echo ls 
/ostree/deploy/fedora-atomic/deploy) | env LIBGUESTFS_BACKEND=direct guestfish 
--ro -a /srv/libvirt/gold/Fedora-Atomic-25-20170124.1.x86_64.qcow2 
27b1ae24686697235c35b793b5c8ab0822b8427e892d3659f0cb300c400979fa.0
27b1ae24686697235c35b793b5c8ab0822b8427e892d3659f0cb300c400979fa.0.origin
$

So anyone who wants to upgrade in place to this hash can do so via e.g.:

$ atomic host deploy 
27b1ae24686697235c35b793b5c8ab0822b8427e892d3659f0cb300c400979fa

This is also: Version: 25.46 (2017-01-24 02:48:29)

[1] https://pagure.io/releng/issue/6545

Re: [atomic-devel] Fedora 26 change: using overlayfs as default

[Colin Walters]

On Tue, Dec 13, 2016, at 12:45 PM, Clayton Coleman wrote:

> Are the POSIX issues in applications running on overlay mostly
> resolved now?  I.e. if we flipped the default would be reasonably able
> to support a diverse range of Linux workloads without the risk that
> previously existed?


overlayfs will never be fully POSIX compatible, but I think that's OK,

because remember - you shouldn't use overlayfs for persistent data,

or really anything that's not code/config files (and we want to get

to where that's overlayfs-type semantics for builds, and read-only

for deployment).  Data should be in Kube persistent volumes etc.



I think the thing to focus on is tools that are run during builds - the
yum-in-overlayfs bug is a good example, because the RPM database

*is* a database which is the type of workload that's going to

be sensitive to the overlayfs semantics.  How many of those

are there?  Probably not many, I suspect most of the compat

issues with userspace have been shaken out by now.



(But long term we may end up in a situation where people

 who want to run e.g. rhel5's yum in a container need to 

 somehow fall back to devmapper)

Re: [atomic-devel] UEFI Fedora atomic partition layout and bootloader install error

[Colin Walters]

On Thu, Dec 8, 2016, at 07:25 PM, Vasiliy Tolstov wrote:
> 2016-12-09 0:18 GMT+03:00 Josh Berkus :
> > Vasiliy,
> >
> > You can follow the UEFI bug here:
> >
> > https://pagure.io/atomic-wg/issue/185

There's a patch available for this now.

> Thanks, also i have problem on uefi system with my layout:
> /dev/mmcblk0p1 /boot/efi EFI System Partition 500Mb (vfat)
> /dev/mmcblk0p2 pv with one lv root / 10Gb (ext4)

This one however may be running into the no-/boot
partition case:

https://github.com/ostreedev/ostree/pull/215
https://github.com/ostreedev/ostree/pull/268

I may look at rebasing that patch set this weekend, it just hasn't been
a major priority for Atomic Host, but it is definitely
nicer for the embedded cases, which I want to do as well.

Re: [atomic-devel] systemd broken in fedora 25 atomic

[Colin Walters]

On Mon, Nov 28, 2016, at 10:00 AM, Vasiliy Tolstov wrote:

> I found hack in pagure.io, and in post process script remove protected
> tmp and protected home. After that all works fine.

Yeah, we'll carry the hack until systemd upgrades.  For
reference, this is:
https://github.com/systemd/systemd/issues/4082
https://pagure.io/fedora-atomic/pull-request/13

Re: [atomic-devel] recheck local tree against remote

[Colin Walters]

On Tue, Nov 22, 2016, at 02:56 AM, Vasiliy Tolstov wrote:
> Does it possible to recheck tree against remote, for example i have
> local storage that can lost data (ssd/flash/hdd) and want to compare
> local objects with remotes (via  fingerprint, hash or something like
> this).
> Does it possible now via ostree or rpm-ostree ?

https://github.com/ostreedev/ostree/pull/345

Re: [atomic-devel] Atomic Host Vagrant box crashes when inserting kernel module

[Colin Walters]

On Sat, Nov 19, 2016, at 10:26 PM, Yun-Chih Chen wrote:

> I use the Vagrantfile[1] in rpm-ostree to boot up a CentOS 7
> Atomic Host.
> The kernel panics when running "rpm-ostree install fuse-sshfs", in the
> stage of "Copying /etc changes: xx modified, xx removed, xx added".
> It turns out that a kernel module called "raid6" get inserted and its
> dependent module "raid6_pq" crashes during initialization.  The panic
> can be reproduced by simply "modprobe raid6".  The crashed vmcore can
> be downloaded here: https://goo.gl/r0XgxN and some relevant system
> info are included as attachments.
> 

> I am not sure if this report belongs here or centos-devel; just
> curious about why such kernel module get inserted when running
> rpm-ostree.


This is a generic issue with grub2's "os-prober", which currently will
be rerun any time a kernel is changed.  It scans all block devices for
operating system installs.   At some point we'll likely change ostree to
suppress this but it requires some rework in grub2.


That said, `modprobe raid6` (and hence `rpm-ostree install` etc.) work
for me (vagrant-libvirt on F25 workstation).  I thinking there must be
some sort of local environmental issue here, but I don't have a lot of
great ideas as to what that could be offhand.

Re: [atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Colin Walters]

On Fri, Oct 14, 2016, at 10:22 AM, Jason DeTiberus wrote:
> The other issue is that we don't require users to manage their
> environments with Ansible, so our temporary modifications would
> also need to be documented and implemented separately for non-
> Ansible users.

I see the point, but the tuned switches have to be a very small part of
what one would need to re-implement if a site decided to use Puppet or
whatever instead.  Right?

Particularly since they're just tuning, and not strictly required for
baseline operation (right?), I'd say we have an argument that anyone
implementing alternative Origin/OSE installers has to reference openshift-
ansible as a baseline, and that would include the tuning settings.

So basically I'm arguing for the tuned settings being owned by openshift-
ansible, and possibly those in turn deriving from some upstream
kubernetes/ansible roles or so.  That said, if you guys feel strongly
about shipping via RPM we can certainly look at that more.

Re: [atomic-devel] We are looking at using OSTree as a backend for sharing file systems into an OCID Container runtime

[Colin Walters]

On Fri, Oct 14, 2016, at 02:37 PM, Daniel J Walsh wrote:

> If we block the creation of the devices when exploding a OCI Image
> Bundle, we end up with something that is different then what is
> downloaded and this could potentially cause problems with mtree checking
> of the image on disk versus the image as shipped.

https://github.com/ostreedev/ostree/pull/443
would be a good place to discuss this.

> Is there a reason OSTree does not work with Device Inodes? 

Yes, I think there's no good reason to include devices in OS or container
images.  For OS images (i.e. ostree-as-base), all modern systems
use devtmpfs. 

(Also for that matter, a bit more strictly, FIFOs either.
 OSTree very intentionally only supports regular files and
 symlinks)

For the Ubuntu Docker image, the user can't even see them
because Docker (correctly) mounts over /dev/ with
the "API devices" as systemd calls them.

So basically...my proposal would be that we
ignore them, and that's indeed what the proposed OSTree
API above does.

If we implement any other verification layer like mtree, it
should then just skip devices and FIFOs (or more generally
anything except regular files and symlinks).

Re: [atomic-devel] bubblewrap 0.1.3 (fixes CVE-2016-8659)

[Colin Walters]

On Fri, Oct 14, 2016, at 12:53 PM, Colin Walters wrote:
> A new release of bubblewrap is available:
> 
> https://github.com/projectatomic/bubblewrap/releases/tag/v0.1.3
...

> So, expect updates to land in:
>  
>  - EPEL7

https://bodhi.fedoraproject.org/updates/bubblewrap-0.1.3-2.el7

>  - CentOS AH Alpha

Done, see:
https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel

Note this pulled in a number of other updates
as we do binary promotion of continuous -> alpha.

The atomic/skopeo train continues to rev, and
there are some smaller fixes to the ostree stack.

Changed:
  atomic 
1.13.0.4-adc8956f50a38d15b65ebf34dacee6a418524e20.c6c664b91e099b3f2078e562af19220c2ff7562d.el7.centos
 -> 
1.13.1.21-aed68da24c0db3ea7e2e3bd4de0904c6b67115e4.dd4522961a9990fd47096ad652d3b6d9059051d7.el7.centos
  bubblewrap 0.1.2.1-169db041313862c3ef03235dde30e6d3c96e2a6a.el7.centos -> 
0.1.2.5-7d035f1bbcce30a80a40fc564427ddbf7589e5f1.el7.centos
  ostree 
2016.11.1-0446cdb0d38e1e05fc51202b580e28c44993b76c.ee3685405394f0193b77e9a9db9256cd1750e69d.el7.centos
 -> 
2016.11.5-a660e5650d0f460810ea75c44d044b6924659f59.ee3685405394f0193b77e9a9db9256cd1750e69d.el7.centos
  ostree-grub2 
2016.11.1-0446cdb0d38e1e05fc51202b580e28c44993b76c.ee3685405394f0193b77e9a9db9256cd1750e69d.el7.centos
 -> 
2016.11.5-a660e5650d0f460810ea75c44d044b6924659f59.ee3685405394f0193b77e9a9db9256cd1750e69d.el7.centos
  rpm-ostree 
2016.10.0-af23b948f12b01db99143bfab82ff60a3e4a4aee.7081d79d1a21dc196c1286ea012b740d47d68e1e.el7.centos
 -> 
2016.10.3-a15364c185df0a72d99440b2dcd210432b9da1d0.7081d79d1a21dc196c1286ea012b740d47d68e1e.el7.centos
  skopeo 
1.14-7d786d11d668f2932e524da53afb658438c8ebfc.0aa48eb05ef9b646ac444ddc878eb93dbd3d6d98.el7.centos
 -> 
1.14-66160a083bed5ec3b551eab62729c6ad5b0889aa.ec8b80abfd4d904e882bcdf340ee21a852e2f979.el7.centos
Added:
  
skopeo-containers-1.14-66160a083bed5ec3b551eab62729c6ad5b0889aa.ec8b80abfd4d904e882bcdf340ee21a852e2f979.el7.centos.x86_64

[atomic-devel] bubblewrap 0.1.3 (fixes CVE-2016-8659)

[Colin Walters]

A new release of bubblewrap is available:

https://github.com/projectatomic/bubblewrap/releases/tag/v0.1.3

Which fixes a local privilege escalation.  Specifically relevant to Project 
Atomic,
this applies only to CentOS7/RHEL7 systems which have
bubblewrap installed as privileged code.

Notably, we *do* install it by default as /usr/bin/bwrap in
CentOS Atomic Host Alpha, but not in the primary CentOS Atomic Host
release, where it exists solely as /usr/libexec/rpm-ostree/bwrap for
use by rpm-ostree's package layering, but is not installed as
privileged and hence is not a vulnerability vector.

Fedora, because it unconditionally enables `CLONE_NEWUSER`
access, is not vulnerable to this.

So, expect updates to land in:
 
 - EPEL7
 - CentOS AH Alpha

soon.

[atomic-devel] Atomic Workstation development work

[Colin Walters]

Hey, so we've talked about this a lot, and there
are now two change pages:

https://fedoraproject.org/wiki/Changes/WorkstationOstree
  This is in Fedora release engineering, and the scope is basically
  rpm-ostree + flatpak

https://fedoraproject.org/wiki/Workstation/AtomicWorkstation
  But I'd like to revisit this, since I want to argue strongly for adding
  Docker to the mix.  I now use "pet" Docker containers for most of my
  random software building/hacking, and I think this use case is
  not really covered by flatpak, and installing build tools on the host
  system I believe should be an explicit anti-pattern.

I've set up https://pagure.io/atomic-ws
which is running in CentOS CI.  This specifically configures Docker
in a useful way (overlayfs), and starts by default.

Basically, by adding Docker, I think this feels closer to Project Atomic,
hence the rebranding.

Besides the link on the pagure page, I copied a snapshot of the ISO
here: https://fedorapeople.org/~walters/atomicws-installer-25.2016.61.iso

There are hence a few things to do to merge this into Fedora:

 - Decide on tooling emphasis (and management; does PackageKit
   need to learn anything about docker?)
 - Decide on the branding
 - Merge in the partitioning changes I made in 
https://pagure.io/atomic-ws/blob/master/f/overlay.yml#_46
   (switch to something similar to Server - xfs by default, no split /home, but 
overlayfs for docker)
 - Fix the ostree management in Fedora (right now at-most-once-a-day
   is far too slow for development and far too fast for most users, I think it
   could make sense to do a two-week cadence (+async security) as
  we're planning for Atomic Host).  This will also enable static deltas
  and make the experience more pleasant; see 
https://fedorahosted.org/rel-eng/ticket/6313
  In CentOS for Atomic Host we use a "promotion" model. 

Re: [atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Colin Walters]

On Tue, Oct 11, 2016, at 02:45 PM, Jeremy Eder wrote:
> Because layered products (not just OpenShift) do not want to be
> coupled to the RHEL release schedule to update their profiles.  They
> want to own their profiles and rely on the tuned daemon to be there.

I see two aspects to this discussion:

1) Generic tradeoffs with host configuration
2) The specific discussion about tuned profiles

Following 2) if I run:

$ cd ~/src/github/openshift/origin
$ git describe --tags --always
v1.3.-rc1-14-ge9081ae
$ git log --follow contrib/tuned/origin-node-host/tuned.conf

There are a grand total of *two* commits that aren't mere
code reorganization:

commit d959d25a405bb28568a17f8bf1b79e7d427ae0dc
Author: Jeremy Eder 
AuthorDate: Tue Mar 29 10:40:03 2016 -0400
Commit: Jeremy Eder 
CommitDate: Tue Mar 29 10:40:03 2016 -0400

bump inotify watches

commit c11cb47c07e24bfeec22a7cf94b0d6d693a00883
Author: Scott Dodson 
AuthorDate: Thu Feb 12 13:06:57 2015 -0500
Commit: Scott Dodson 
CommitDate: Wed Mar 11 16:41:08 2015 -0400

Provide both a host and guest profile

That level of change seems quite sufficient for the slower
RHEL cadence, no?

Particularly when one considers that something like the
inotify watch bump could easily be part of a "tuned updates"
in the installer that would live there until the base tuned
profile updates.

Right?

> Before we go the layered RPM route I just want to make sure you're
> onboard with it, as I was not aware of any existing in-product users
> of that feature.  Are there any? If we're the first that's not an
> issue, just want to make sure we get it right.

In this particular case of tuned, I'd argue that Atomic Host should come
out of the box with these profiles,
and that any async updates could be done via the openshift-ansible
installer.

Re: [atomic-devel] rpm-ostree error: Bus owner changed, aborting.

[Colin Walters]

On Tue, Oct 11, 2016, at 10:10 AM, Colin Walters wrote:

> https://bugzilla.redhat.com/show_bug.cgi?id=1383708

https://bodhi.fedoraproject.org/updates/gnutls-3.5.5-2.fc25

Re: [atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Colin Walters]

On Tue, Oct 11, 2016, at 01:36 PM, Jeremy Eder wrote:

> Going fwd, I think we would rather not maintain two locations (atomic-
> * and atomic-openshift-* tuned profiles with identical content.

Yes, agreed.

>
> So, trying to reason a way to get those profiles onto an AH since we
> can't install the tuned-atomic-openshift RPM

That's not true.  We've been shipping package layering for quite a
while.

> ...We could copy them to /etc/tuned and enable them manually...but I'm
> not sure that jives with how we're supposed to use AH and it seems
> kind of hacky since there would be "orphan files" in /etc.  Thoughts?

I wouldn't say they're orphaned if something "owns" it.  Ownership
doesn't have to just be RPM, it can also be Ansible.

Although a common trap with management systems like Ansible and Puppet
is (by default) they're subject
to https://en.wikipedia.org/wiki/Hysteresis - if one version of the
installer creates a tuned snippet, then
we later don't want it to apply, the Ansible rules have to carry code to
explicitly ensure it's deleted.  Whereas
with RPM (and ostree) the system does synchronize to the new state,
automatically deleting files
no longer shipped.

Anyways, I'm a bit confused here - why isn't the fix to:

1) Put the profile in the tuned RPM
2) Atomic Host installs it by default
3) Installers like openshift-ansible ensure it's installed (noop on AH)

Re: [atomic-devel] rpm-ostree error: Bus owner changed, aborting.

[Colin Walters]

On Mon, Oct 10, 2016, at 04:36 PM, Dusty Mabe wrote:

> -bash-4.3# rpm-ostree upgrade
> Updating from: fedora-atomic:fedora-atomic/25/x86_64/docker-host
> error: Bus owner changed, aborting.

https://bugzilla.redhat.com/show_bug.cgi?id=1383708

Re: [atomic-devel] creating an ostree repo from a running atomic host?

[Colin Walters]

On Sun, Sep 18, 2016, at 02:27 PM, Tom McKay wrote:
> I'm trying to wrap my head around ostree/rpm-ostree/atomic to think
> about how Satellite-6 (and upstream katello) can contribute to the
> flow. A couple questions...
> First, I see rpm-ostree-toolkit referenced in a lot of places but am
> struggling to find it as an rpm. Is it upstream somewhere?

Yes, it's in both Fedora
http://pkgs.fedoraproject.org/cgit/rpms/rpm-ostree-toolbox.git/
and CentOS:
http://cbs.centos.org/repos/atomic7-testing/x86_64/os/Packages/

> Next, is there a way to create an ostree repo from a running atomic
> host? I've seen the various articles and docs describing a compose
> server w/ modified json, etc. As a very new user, though, the simplest
> way to experiment with making new repos is directly on the host
> (ostree pkg-add, etc.). How can I take that host and provide it as a
> template to other hosts?

Basically, "where is the docker push equivalent"?  So...we don't really
have that, and it's somewhat intentional, for the same reason the Docker
documentation discourages `docker commit` - the philosophy here is
you're generating something without recording *how* it was built and
hence it's not reproducible.  (Well, at least not recording

As far as a "template" here though, it's basically just adding the
package names to the "treefile" json and doing a full compose, then
pointing those client systems at it via rebase.  (Or doing an initial
kickstart install)

[atomic-devel] [PATCH] post: Neuter all systemd PrivateTmp= and Protect(Home|System)

[Colin Walters]

https://pagure.io/fedora-atomic/pull-request/13

[atomic-devel] [PATCH] Remove dracut-live (master, f25)

[Colin Walters]

https://pagure.io/fedora-atomic/pull-request/12

Re: [atomic-devel] package layering question

[Colin Walters]

On Thu, Aug 25, 2016, at 08:13 AM, Vasiliy Tolstov wrote:
> I'm build latest ostree for fedora 23 and 24 and want to try
> rpm-ostree pkg-add, how this command determine from which repo
> download package? Does i need dnf in system and configured repos in
> /etc/yum/repos.d ?

rpm-ostree uses libdnf (also known as libhif), which reads from 
/etc/yum.repos.d,
and generally has similar semantics. You don't need the `dnf` program installed
(on the host) though it can be convenient to use for read-only operations like
`search` until we implement it in rpm-ostree. I tend to use yum/dnf inside pet
Docker containers for things like that now.

Re: [atomic-devel] Moving towards containerized Kube/layered packages

[Colin Walters]

So far, if there were a Metacritic for email posts, this one would
be about 100/100 "Universal Acclaim" (based on 4 critics)!  So, let's move 
forwards:

https://github.com/CentOS/sig-atomic-buildscripts/pull/144#issuecomment-242189717
https://pagure.io/fedora-atomic/c/7d50906d91097f6eef4af91a6f693b98c57a9b91?branch=master
(As mentioned for Fedora I just changed 25 and rawhide, not 24)

I'll cut a new CAHC Alpha soon, and keep looking at Ansible and fixes
for both syscontainers and rpm-ostree layering.

[atomic-devel] Moving towards containerized Kube/layered packages

[Colin Walters]

Hi, I'd like to propose a fairly fundamental rework of Atomic Host.  TL;DR:

- Move towards "system containers" (or layered packages) for flannel/etcd
- Move towards containers (system, or Docker) for kubernetes-master
- Move towards layered packages for kubernetes-node and storage (ceph/gluster)

In progress PR:

https://github.com/CentOS/sig-atomic-buildscripts/pull/144

There are advantages to this and disadvantages; I think we'll have some
short term transition pain, but past that short term the advantages
outweigh the disadvantages a lot.

== Advantage: Version flexibility ==

etcd for should really have its own identity in a clustered environment, and
not necessarily roll fowards/backwards with the underlying host version.  I've
had users report things like hitting an e.g. Kubernetes or Docker issue and 
rolling back their
host, which rolled back etcd as well, but the store isn't forwards-compatible,
which then breaks.  There's also a transition to etcd2 coming, which again
one should really manage distinct from host upgrades.

Another major example is that while we chose to include Kubernetes in
the host, it's a fast moving project, and many people want to use a newer
version, or a different distribution like OpenShift.  The version flexibility
also applies to other components like Ceph/Gluster and flannel.

== Advantage: Size and fit to purpose ==

We included things like the Ceph and GlusterFS drivers in the base
host, but that was before we had layered packages, and there's
also continuing progress on containerized drivers.  If one is using
an existing IaaS environment like OpenStack or AWS, many users
want to reuse Cinder/AWS, rather than maintaining their own storage.

Similarly, while flannel is a good general purpose tool, there are
lots of alternatives, and some users already have existing SDN solutions.

== Disadvantage: More assembly required ==

This is a superficial disadvantage I think - in practice, since we didn't
pick a single official installation/upgrade system (like OpenShift has
openshift-ansible), if you want to run a Kubernetes cluster, you need
to do a lot of assembly anyways.  Adding a bit more to that I suspect
isn't going to be too bad for most users.

Down the line I'd like to revisit the installation/upgrade story - there's
work happening upstream in
https://github.com/kubernetes/contrib/tree/master/ansible
and I think there's also interest and some work in
having parts of openshift-ansible be available for baseline Kubernetes
and accessible on Galaxy etc.

== Disadvantage: Dependency on new tooling ==

Both `rpm-ostree pkg-add` and `atomic install --system` are pretty new.
They both could use better documentation, more real world testing, and
in particular haven't gained management tool awareness yet (for example, they 
need
better Ansible support). 

== Summary ==

If people agree, I'd like to merge the PR pretty soon and do a new CentOS AH 
Alpha,
and we can collaborate on updating docs/tools around this.  For Fedora...it's
probably simplest to leave 24 alone and just do 25 for now.

What I'd like to focus on is having AH be more of a good "building block"
rather than positioning it as a complete solution.  We ensure that the base
Docker/kernel/SELinux/systemd block works together, system management tools
work, and look at working more in the upstream Kubernetes (and OpenShift)
communities, particularly around Ansible.

Re: [atomic-devel] bwrap-oci: convert from OCI to bubblewrap command line

[Colin Walters]

On Tue, Aug 9, 2016, at 12:55 PM, Giuseppe Scrivano wrote:
> Hi,
> 
> I would like to introduce bwrap-oci, a tool to convert from an OCI
> configuration file to a command line for bubblewrap:
> 
> https://github.com/giuseppe/bwrap-oci
> 
> While system containers are executed through runc which uses directly
> the OCI configuration file, containers that run as non root will use
> bubblewrap[1] through bwrap-oci.  With bwrap-oci the same image format
> can be used for both variants, as the OCI configuration file will be
> converted to the bubblewrap command line to run the container.

And this is likely to become a dependency of the atomic install --user
work right?  In that case, makes sense to me.  If no one has any further
comment let's do the move tomorrow.

Re: [atomic-devel] right way to upgrade fedora atomic host

[Colin Walters]

On Sat, Aug 6, 2016, at 12:02 PM, Vasiliy Tolstov wrote:
 
> so does it possible to use on compose server rpm-ostree command, and
> on host system ostree without atomic command, that need to pull some
> dependencies...

Yes - we're keeping the core ostree level to be low-dependency C code
to better accommodate classical embedded system use cases (OS + app merged).

[atomic-devel] Preparing for overlayfs as equal option

[Colin Walters]

Upstream docker has a decent page:
https://docs.docker.com/engine/userguide/storagedriver/selectadriver/

(One thing they don't mention explicitly though is the page cache
sharing that overlayfs has over devicemapper or btfs, which can be substantial)

Though the patches to use overlayfs with SELinux are still experimental, and
not yet in our CentOS stream, I'd like to lay the groundwork for it.

In particular, overlayfs has a significant reduction in "administrative
cognitive overhead", since we can default to one LV+filesystem for
the OS that encompasses both the OS (yum/ostree) data and container
images, and hence not have to juggle LV sizes.

Another way to look at this is it makes "yum install docker" on CentOS 7
work with a single disk default.

This is all related to single node - and there are a lot of potentially
better way to manage images in a cluster, but the single node experience
is important too.  It's relevant both for desktop systems that use Docker and
Vagrant boxes, etc.

To that end:
https://github.com/CentOS/sig-atomic-buildscripts/pull/134
landed, which ensures that newly formatted xfs filesystems are compatible.

Our CI then updated the installer and cloud images, so I verified that
the vagrant-libvirt box here:
https://ci.centos.org/artifacts/sig-atomic/centos-continuous/images/cloud/latest/images/
has:

# xfs_info /|grep ftype
naming   =version 2  bsize=4096   ascii-ci=0 ftype=1

There's a number of TODOs here like making it easier to default to overlayfs 
from Anaconda/kickstart.

But I think the important thing for us to flesh out better is transition
paths.  Obviously, one can just reinstall a node.  Many environments will
be set up to do that, but we should also support transitioning a dm to overlay
when one doesn't want to reinstall.

 I've verified that from a current CentOS AH Alpha, I can:

systemctl stop docker
rm /var/lib/docker/* -rf
# (configure docker to use overlay)
lvm lvremove atomicos/docker-pool
lvm lvcreate -n docker-images -L 10G atomicos # (TODO: also tweak this to 
auto-grow?)
mkfs.xfs -n ftype=1 /dev/mapper/atomicos-docker--images
mount /dev/mapper/atomicos-docker--images /var/lib/docker
# (add that to /etc/fstab too)
systemctl start docker

This keeps the atomicos/root LV with an old-format XFS filesystem and
won't give you a unified storage pool, but does give you the runtime
advantages of overlay.

Re: [atomic-devel] update without full reboot

[Colin Walters]

On Fri, Aug 5, 2016, at 09:11 AM, Vasiliy Tolstov wrote:
> Hi! Does it possible to update atomic host and do kexec skipping bios time?

We haven't investigated that, but it shouldn't be too hard to wire together.
In the case where the kernel hasn't changed we could probably teach
systemd to do a switchroot without kexec at all, which would be a lot faster 
itself.

Re: [atomic-devel] A new policy rpm for Atomic?

[Colin Walters]

On Tue, Jul 5, 2016, at 03:18 PM, Colin Walters wrote:
> On Mon, Jun 27, 2016, at 05:04 AM, Miroslav Grepl wrote:
> 
> > So we could start to discuss how it is possible to ship new policy on
> > Atomic to solve these urgent issues.

Wait, I may have been confused here.  Is this actually just solving:
https://bugzilla.redhat.com/show_bug.cgi?id=1290659

In that case I'd still lean a bit more towards:
https://bugzilla.redhat.com/show_bug.cgi?id=1290659#c24

It shouldn't be too hard for us to carry a delta in rpm-ostree to
undo the move to /var - then you don't have to have a separate
policy build.  As long as the selinux userspace isn't confused by
this and knows how to look in both /etc/selinux and /var.

(It would be nice to consider moving the RPM back even on yum-managed
 systems)

[atomic-devel] rpm-ostree v2016.4

[Colin Walters]

I sent a mail to fedora-devel about this:

https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/2LQ7ELNPQP6EDWFDW4RWOZJFJFWDEUF6/

Re: [atomic-devel] Smaller fedora & centos images

[Colin Walters]

On Mon, Jun 20, 2016, at 01:57 PM, Micah Abbott wrote:
> On 06/20/2016 09:38 AM, Joe Brockmeier wrote:
> > Have we published any comparisons of an Alpine image "fully loaded"
> > (e.g., with the actual tools) vs. Fedora, etc.? AIUI, when you actually
> > install things like Apache httpd, or whatnot the comparison looks much
> > closer.
> 
> I hacked up some quick Dockerfiles for this particular example (httpd) 
> and the end result is that alpine was still smaller - 8.652 MB vs. 232.8 MB

The largest chunk of this is that the httpd RPM pulls in systemd, and that's
going to be true presently for a lot of our RPMs.  So that's a whole subthread 
to this.

It'd be possible to teach micro-yuminst to just ignore requirements
on specific packages like systemd, while still allowing anyone who explicitly
wants them to pull them in.

Re: [atomic-devel] Smaller fedora & centos images

[Colin Walters]

On Wed, Jul 13, 2016, at 04:40 PM, Muayyad AlSadi wrote:
> What about my question about the equivalent of "--setopt
> tsflags=nodocs"
>
> @walters does micro-yuminst assume this option
 
https://github.com/cgwalters/micro-yuminst/issues/1
 
I'll rework the command line parsing soon to better support things like
this.
 

Re: [atomic-devel] Smaller fedora & centos images

[Colin Walters]

On Wed, Jul 13, 2016, at 09:40 AM, Tim St. Clair wrote:
> Awesome!
>
> Do we have a formal position, or is this still POC?
 
Still a PoC, but I believe it'd be relatively easy for downstreams to
productize.  For
example, we're using librepo[1] which is the same library used by dnf
(and rpm-ostree)
that knows how to speak TLS client certificates[2] that Red Hat
Enterprise Linux uses
to gate access to content.  And for that matter implements RPM GPG
verification the same
way.
 
In general the micro-yuminst layer will require some maintenance but not
that much;
"yum -y install" is 95% of anyone wants to do in Docker images.
 
Probably the next interesting question is whether the current base
images should derive from this.
 
Anyways, I encourage feedback as issues in
https://github.com/cgwalters/centos-dockerbase-minimal
 
If there's enough interest, we'll see about importing it into the
projectatomic/ github org and
taking next steps like integrating it into the CentOS build processes.
 
BTW, I set up a Jenkins job for this:
https://ci.centos.org/job/atomic-dockerimage-centosmin/
[1] https://github.com/rpm-software-management/librepo
[2] https://github.com/rpm-software-management/libhif/pull/144
 

Re: [atomic-devel] Fedora 24 Atomic images

[Colin Walters]

On Wed, Jul 13, 2016, at 08:30 AM, Jan Pazdziora wrote:

>   There are no Atomic images available for Fedora 24. Please
>   stand by and checkout this page in a few days, we are planning
>   to build them within the first week of July.

See
https://lists.fedoraproject.org/archives/list/cl...@lists.fedoraproject.org/thread/2XEQTIXVJ4AU7JQM7BQIJCKOESVNTNAS/

Re: [atomic-devel] Smaller fedora & centos images

[Colin Walters]

...3 weeks later:
 
On Tue, Jun 21, 2016, at 04:59 PM, Colin Walters wrote:
>
> It does seem viable to create a `centosmin` image that in some cases
> uses different package builds (e.g. ensuring rpm doesn't
> pullrelatively close in being min-coreutils + bash + yum.  Some
> postprocessing on the depchain such as deleting `.py{,c}` files etc.
> would help.
 
I realized recently that the work we'd been doing in libhif[1] which is
a C library for package management
would allow us to have a minimal "yum -y install" reimplementation[2]
using libhif, which would be
good for such a minimal image as then we could drop Python for example.
 
I spent some of last Friday's plane flight back from the Summit
working on it:
https://github.com/cgwalters/centos-dockerbase-minimal
 
You can try it with:
 
docker pull docker.io/cgwalters/centosmin
 
I've only done some basic smoketesting on it.  Compressed it's 28MB,
uncompressed 77MB right now.
I estimate it wouldn't be too hard to get within 60MB, but past that
things get a bit trickier as we
need to investigate single-binary coreutils, a minimal libcurl build,
and in general trimming
out a lot of the duplication in our C libraries like only having openssl
and not nss, trimming down
glib2 etc.
 
I'm curious what people think.  The tradeoff is we now have two base
images (per distribution).
 
[1] https://github.com/rpm-software-management/libhif
[2] https://gitlab.com/cgwalters/micro-yuminst

Re: [atomic-devel] ostree and ssd/emmc

[Colin Walters]

On Fri, Jul 8, 2016, at 07:08 AM, Vasiliy Tolstov wrote:
> I'm already build own image using ostree, now i want to experiment and
> build image for my notebook.
> But i have emmc and want to know - how fast it dead if i use ostree
> image to update my system? (i'm plan do update image each weekend)
> How ostree compared to git in case of storing files ? Does it
> expensive to usage on ssd drives?

I haven't done any measurements but it's likely fairly close
to the same number of disk writes as yum.  I'd guess that the
biggest cost for yum in particular is the RPM metadata - for
Fedora it's huge.  (rpm-ostree with package layering pays this
same cost, but you can avoid it with custom tree composes of course).  

I'd be more concerned about sources of constant writes like having the
systemd journal be persistent.

Re: [atomic-devel] Looking for stability when composing trees

[Colin Walters]

On Fri, Jul 1, 2016, at 05:07 PM, Mark Dyer wrote:
> We considered making mirrors of all the repos with just the versions
> of packages that we want, but there were concerns about being able to
> go back to older versions and /or the complexity of maintaining yum
> mirrors for every version we release.
 
We could enhance rpm-ostree to support this at some point, it
likely woudn't
be too hard.  It just hasn't been a priority because there are so many
tools for managing promotion of RPM repositories (e.g.
http://www.pulpproject.org/ )
 
 

Re: [atomic-devel] CPU and Memory accounting for containerized kubernetes

[Colin Walters]

On Mon, Jul 11, 2016, at 10:34 AM, Jan Chaloupka wrote:
> Hi,
> 
> upcoming build of kubernetes in Fedora (and RHEL) is about to have CPU 
> and Memory systemd accounting enabled by default (upstream issue [1]).
> So far, Openshift took steps to update its spec file [1]. Ansible 
> installing kubernetes does the very same [2]. The works for 
> non-containerized kubernetes. Question is how to deal with containerized 
> kubernetes cause installing kubernetes-node rpm inside will not be enough.

https://github.com/kubernetes/contrib/blob/master/ansible/roles/common/tasks/main.yml#L52

It seems to me that having Ansible write those same config files for
the containerized install is sufficient.  Right?

We could investigate doing this in `atomic install` which allows containers
to drop such files on the host, but it's currently mutually exclusive with
`atomic install --system` which I think is better for things like this.

Broadly speaking, I think we should consider RPM (and containers) as just a 
binary delivery
format, and use Ansible for anything that requires a modicum of intelligence.
At least sufficient to bootstrap Kubernetes.

Re: [atomic-devel] Refspec used by "atomic host upgrade"

[Colin Walters]

On Thu, Jul 7, 2016, at 07:21 AM, Marius Vollmer wrote:
> Hi,
> 
> when creating our Atomic test images, we want to first run "atomic host
> upgrade" and then "ostree checkout $ref /var/local-tree".  We do this so
> that we can modify /var/local-tree via rpm and boot into it to run the
> tests.
>
> 
> What should $ref be so that /var/local-tree will be the same as what
> "atomic host upgrade" has just downloaded and deployed for the next
> boot?

One intent of ostree is to clearly and concisely define the base system.
If I say I'm running "fedora/24/x86_64/docker-host", that should be
unique - i.e. no one else should choose that name.  Now unlike
Docker there's no defacto centralized naming registry (in fact
ostree intentionally has a more distributed "remote" model), but anyways
the idea is the branches are unique and descriptive.

This design does make it more awkward to manipulate
the system locally, because conceptually then, the system *isn't*
"fedora/24/x86_64/docker-host".  Unlike docker, OSTree doesn't
natively have a model for derivation.  For rpm-ostree though,
the layered packages are clearly distinguished.

All of that philosophy aside, there are many ways to do this:

#1) Create a local webserver and archive repo with new updates
  and change the url= line in the /etc/ostree/remotes.d file

This one involves a bit more data movement because you'd have
to pull the data from the system into an "archive-z2" repository.

This is how the unit tests for rpm-ostree work:
https://github.com/projectatomic/rpm-ostree/blob/master/tests/common/libtest.sh#L303

#2)  Generate a local commit and *rebase* to it

The current rpm-ostree vagrant-based tests do something like this:

https://github.com/projectatomic/rpm-ostree/blob/master/tests/vmcheck/test.sh#L98

#3) Generate a local commit that overrides the origin remote

This is what you're trying to do now...but it's really battling
the system.  Here's how to generate a quick local commit that
appears to be like the remote:

ref=centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
cd /sysroot/ostree/repo/tmp
ostree checkout --fsync=false ${ref} work
mkdir -p mnt
rofiles-fuse work mnt
echo somenewbinary > mnt/usr/bin/anewbinary
fusermount -u mnt
ostree commit --link-checkout-speedup -b ${ref} -s '' --tree=dir=work

But the problem here is that `atomic host upgrade` is going to talk
to the remote webserver.

Also, this will fail if the remote is configured for GPG verification.

> We used dig it out of /usr/share/rpm-ostree/treefile.json, giving

There is a DBus API for this, and we recently added `status --json`:
https://github.com/projectatomic/rpm-ostree/pull/315

> Now we need to prefix the remote as well, like this:
> 
> ostree checkout \
>   rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard \
>   /var/local-tree
> 
> "atomic host upgrade" has a way to compute this refspec.  How can we
> reliably get the same value?

It's in the .origin file.

I think in the big picture, what you're trying to do here should be part of the
core model - with package layering we now cleanly support layering,
this is *overriding* - but in the end "it's just files", the challenge is mostly
around presentation in the command line etc.

For now though I'd look at #1 or #2.