Re: [atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Colin Walters]

On Fri, Oct 14, 2016, at 10:22 AM, Jason DeTiberus wrote:
> The other issue is that we don't require users to manage their
> environments with Ansible, so our temporary modifications would
> also need to be documented and implemented separately for non-
> Ansible users.

I see the point, but the tuned switches have to be a very small part of
what one would need to re-implement if a site decided to use Puppet or
whatever instead.  Right?

Particularly since they're just tuning, and not strictly required for
baseline operation (right?), I'd say we have an argument that anyone
implementing alternative Origin/OSE installers has to reference openshift-
ansible as a baseline, and that would include the tuning settings.

So basically I'm arguing for the tuned settings being owned by openshift-
ansible, and possibly those in turn deriving from some upstream
kubernetes/ansible roles or so.  That said, if you guys feel strongly
about shipping via RPM we can certainly look at that more.

Re: [atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Jason DeTiberus]

On Fri, Oct 14, 2016 at 7:40 AM, Jeremy Eder  wrote:

> On Wed, Oct 12, 2016 at 10:29 AM, Colin Walters 
> wrote:
>
>>
>> On Tue, Oct 11, 2016, at 02:45 PM, Jeremy Eder wrote:
>>
>> Because layered products (not just OpenShift) do not want to be coupled
>> to the RHEL release schedule to update their profiles.  They want to own
>> their profiles and rely on the tuned daemon to be there.
>>
>>
>> I see two aspects to this discussion:
>>
>> 1) Generic tradeoffs with host configuration
>> 2) The specific discussion about tuned profiles
>>
>> Following 2) if I run:
>>
>> $ cd ~/src/github/openshift/origin
>> $ git describe --tags --always
>> v1.3.0-rc1-14-ge9081ae
>> $ git log --follow contrib/tuned/origin-node-host/tuned.conf
>>
>> There are a grand total of *two* commits that aren't mere
>> code reorganization:
>>
>> commit d959d25a405bb28568a17f8bf1b79e7d427ae0dc
>> Author: Jeremy Eder 
>> AuthorDate: Tue Mar 29 10:40:03 2016 -0400
>> Commit: Jeremy Eder 
>> CommitDate: Tue Mar 29 10:40:03 2016 -0400
>>
>> bump inotify watches
>>
>> commit c11cb47c07e24bfeec22a7cf94b0d6d693a00883
>> Author: Scott Dodson 
>> AuthorDate: Thu Feb 12 13:06:57 2015 -0500
>> Commit: Scott Dodson 
>> CommitDate: Wed Mar 11 16:41:08 2015 -0400
>>
>> Provide both a host and guest profile
>>
>> That level of change seems quite sufficient for the slower
>> RHEL cadence, no?
>>
>
> Decoupling profiles from RHEL has already been negotiated with many
> different engineering teams.  As you can imagine, it has ties into our
> channels and distribution mechanics.  Making an exception here doesn't make
> sense to me when it's working fine everywhere else.
>

Given the reboot issue gets addressed, I think I would prefer this approach
as well. We are working as best as we can to decouple the underlying host
management from the cluster management especially around upgrades. Being
able to update and ship the tuned profiles as needed would allow us to
manage it as part of the cluster management without having to query
underlying host state to determine if we need to do temporary
modifications. The other issue is that we don't require users to manage
their environments with Ansible, so our temporary modifications would also
need to be documented and implemented separately for non-Ansible users.


Particularly when one considers that something like the
>> inotify watch bump could easily be part of a "tuned updates"
>> in the installer that would live there until the base tuned
>> profile updates.
>>
>> Right?
>>
>
> ​Personally I would prefer to keep tuning centralized into tuned and not
> have 5 difference places where it's being done...but to your point around
> having two commits ... I'm losing that consolidation battle because
> Kubernetes has hardcoded certain sysctl adjustments that ideally we really
> should have carried in tuned :-/  But if we can at least avoid doing things
> in openshift-ansible at least that's one less place to track.​
>

I can understand why Kubernetes wouldn't want to require tuned, but maybe
we can drive changes upstream to make sysctl management optional. Then we
would be able to add the tuned requirement in our packaging and handle it
there without forcing tuned upstream.



>
>
>
>> Before we go the layered RPM route I just want to make sure you're
>> onboard with it, as I was not aware of any existing in-product users of
>> that feature.  Are there any? If we're the first that's not an issue, just
>> want to make sure we get it right.
>>
>>
>> In this particular case of tuned, I'd argue that Atomic Host should come
>> out of the box with these profiles,
>> and that any async updates could be done via the openshift-ansible
>> installer.
>>
>
> Realistically speaking -- we may want to use AH with another
> product...we've developed
> ​realtime and ​
> NFV profiles which again exist in another
> ​
> channel and there is no such thing as openshift-ansible there.
> ​
> What would be your approach if the openshift-ansible option did not exist?
> (back to scattered tuning)​
> ​​
>
>


-- 
Jason DeTiberus

Re: [atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Jeremy Eder]

On Wed, Oct 12, 2016 at 10:29 AM, Colin Walters  wrote:

>
> On Tue, Oct 11, 2016, at 02:45 PM, Jeremy Eder wrote:
>
> Because layered products (not just OpenShift) do not want to be coupled to
> the RHEL release schedule to update their profiles.  They want to own their
> profiles and rely on the tuned daemon to be there.
>
>
> I see two aspects to this discussion:
>
> 1) Generic tradeoffs with host configuration
> 2) The specific discussion about tuned profiles
>
> Following 2) if I run:
>
> $ cd ~/src/github/openshift/origin
> $ git describe --tags --always
> v1.3.0-rc1-14-ge9081ae
> $ git log --follow contrib/tuned/origin-node-host/tuned.conf
>
> There are a grand total of *two* commits that aren't mere
> code reorganization:
>
> commit d959d25a405bb28568a17f8bf1b79e7d427ae0dc
> Author: Jeremy Eder 
> AuthorDate: Tue Mar 29 10:40:03 2016 -0400
> Commit: Jeremy Eder 
> CommitDate: Tue Mar 29 10:40:03 2016 -0400
>
> bump inotify watches
>
> commit c11cb47c07e24bfeec22a7cf94b0d6d693a00883
> Author: Scott Dodson 
> AuthorDate: Thu Feb 12 13:06:57 2015 -0500
> Commit: Scott Dodson 
> CommitDate: Wed Mar 11 16:41:08 2015 -0400
>
> Provide both a host and guest profile
>
> That level of change seems quite sufficient for the slower
> RHEL cadence, no?
>

Decoupling profiles from RHEL has already been negotiated with many
different engineering teams.  As you can imagine, it has ties into our
channels and distribution mechanics.  Making an exception here doesn't make
sense to me when it's working fine everywhere else.

Particularly when one considers that something like the
> inotify watch bump could easily be part of a "tuned updates"
> in the installer that would live there until the base tuned
> profile updates.
>
> Right?
>

​Personally I would prefer to keep tuning centralized into tuned and not
have 5 difference places where it's being done...but to your point around
having two commits ... I'm losing that consolidation battle because
Kubernetes has hardcoded certain sysctl adjustments that ideally we really
should have carried in tuned :-/  But if we can at least avoid doing things
in openshift-ansible at least that's one less place to track.​



> Before we go the layered RPM route I just want to make sure you're onboard
> with it, as I was not aware of any existing in-product users of that
> feature.  Are there any? If we're the first that's not an issue, just want
> to make sure we get it right.
>
>
> In this particular case of tuned, I'd argue that Atomic Host should come
> out of the box with these profiles,
> and that any async updates could be done via the openshift-ansible
> installer.
>

Realistically speaking -- we may want to use AH with another
product...we've developed
​realtime and ​
NFV profiles which again exist in another
​
channel and there is no such thing as openshift-ansible there.
​
What would be your approach if the openshift-ansible option did not exist?
(back to scattered tuning)​
​​

Re: [atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Colin Walters]

On Tue, Oct 11, 2016, at 02:45 PM, Jeremy Eder wrote:
> Because layered products (not just OpenShift) do not want to be
> coupled to the RHEL release schedule to update their profiles.  They
> want to own their profiles and rely on the tuned daemon to be there.

I see two aspects to this discussion:

1) Generic tradeoffs with host configuration
2) The specific discussion about tuned profiles

Following 2) if I run:

$ cd ~/src/github/openshift/origin
$ git describe --tags --always
v1.3.-rc1-14-ge9081ae
$ git log --follow contrib/tuned/origin-node-host/tuned.conf

There are a grand total of *two* commits that aren't mere
code reorganization:

commit d959d25a405bb28568a17f8bf1b79e7d427ae0dc
Author: Jeremy Eder 
AuthorDate: Tue Mar 29 10:40:03 2016 -0400
Commit: Jeremy Eder 
CommitDate: Tue Mar 29 10:40:03 2016 -0400

bump inotify watches

commit c11cb47c07e24bfeec22a7cf94b0d6d693a00883
Author: Scott Dodson 
AuthorDate: Thu Feb 12 13:06:57 2015 -0500
Commit: Scott Dodson 
CommitDate: Wed Mar 11 16:41:08 2015 -0400

Provide both a host and guest profile

That level of change seems quite sufficient for the slower
RHEL cadence, no?

Particularly when one considers that something like the
inotify watch bump could easily be part of a "tuned updates"
in the installer that would live there until the base tuned
profile updates.

Right?

> Before we go the layered RPM route I just want to make sure you're
> onboard with it, as I was not aware of any existing in-product users
> of that feature.  Are there any? If we're the first that's not an
> issue, just want to make sure we get it right.

In this particular case of tuned, I'd argue that Atomic Host should come
out of the box with these profiles,
and that any async updates could be done via the openshift-ansible
installer.

Re: [atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Jeremy Eder]

On Tue, Oct 11, 2016 at 2:14 PM, Colin Walters  wrote:

> On Tue, Oct 11, 2016, at 01:36 PM, Jeremy Eder wrote:
>
> Going fwd, I think we would rather not maintain two locations (atomic-*
> and atomic-openshift-* tuned profiles with identical content.
>
>
> Yes, agreed.
>
>
> So, trying to reason a way to get those profiles onto an AH since we can't
> install the tuned-atomic-openshift RPM
>
>
> That's not true.  We've been shipping package layering for quite a while.
>

​OK, I hadn't seen it.​  Just read the blog that Jason sent.  Looks good.

> ...We could copy them to /etc/tuned and enable them manually...but I'm not
> sure that jives with how we're supposed to use AH and it seems kind of
> hacky since there would be "orphan files" in /etc.  Thoughts?
>
>
> I wouldn't say they're orphaned if something "owns" it.  Ownership doesn't
> have to just be RPM, it can also be Ansible.
>
> Although a common trap with management systems like Ansible and Puppet is
> (by default) they're subject
> to https://en.wikipedia.org/wiki/Hysteresis - if one version of the
> installer creates a tuned snippet, then
> we later don't want it to apply, the Ansible rules have to carry code to
> explicitly ensure it's deleted.  Whereas
> with RPM (and ostree) the system does synchronize to the new state,
> automatically deleting files
> no longer shipped.
>
> Anyways, I'm a bit confused here - why isn't the fix to:
>
> 1) Put the profile in the tuned RPM
> 2) Atomic Host installs it by default
> 3) Installers like openshift-ansible ensure it's installed (noop on AH)
>

​Because layered products (not just OpenShift) do not want to be coupled to
the RHEL release schedule to update their profiles.  They want to own their
profiles and rely on the tuned daemon to be there.
​

Before we go the layered RPM route I just want to make sure you're onboard
with it, as I was not aware of any existing in-product users of that
feature.  Are there any? If we're the first that's not an issue, just want
to make sure we get it right.

Now, what would the implementation look like ... basically
openshift-ansible would do what the blog does?
http://www.projectatomic.io/blog/2016/08/new-centos-atomic-host-with-package-layering-support/

Also using the layered RPMs seems to currently have a reboot requirement.
Is that correct?  At least until we have
https://bugzilla.gnome.org/show_bug.cgi?id=767977
​ ?​

Re: [atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Colin Walters]

On Tue, Oct 11, 2016, at 01:36 PM, Jeremy Eder wrote:

> Going fwd, I think we would rather not maintain two locations (atomic-
> * and atomic-openshift-* tuned profiles with identical content.

Yes, agreed.

>
> So, trying to reason a way to get those profiles onto an AH since we
> can't install the tuned-atomic-openshift RPM

That's not true.  We've been shipping package layering for quite a
while.

> ...We could copy them to /etc/tuned and enable them manually...but I'm
> not sure that jives with how we're supposed to use AH and it seems
> kind of hacky since there would be "orphan files" in /etc.  Thoughts?

I wouldn't say they're orphaned if something "owns" it.  Ownership
doesn't have to just be RPM, it can also be Ansible.

Although a common trap with management systems like Ansible and Puppet
is (by default) they're subject
to https://en.wikipedia.org/wiki/Hysteresis - if one version of the
installer creates a tuned snippet, then
we later don't want it to apply, the Ansible rules have to carry code to
explicitly ensure it's deleted.  Whereas
with RPM (and ostree) the system does synchronize to the new state,
automatically deleting files
no longer shipped.

Anyways, I'm a bit confused here - why isn't the fix to:

1) Put the profile in the tuned RPM
2) Atomic Host installs it by default
3) Installers like openshift-ansible ensure it's installed (noop on AH)

Re: [atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Jason DeTiberus]

On Tue, Oct 11, 2016 at 1:53 PM, Scott Dodson  wrote:

> On Tue, Oct 11, 2016 at 1:44 PM, Jason DeTiberus 
> wrote:
> >
> >
> > On Tue, Oct 11, 2016 at 1:36 PM, Jeremy Eder  wrote:
> >>
> >> Hi,
> >>
> >> Right now we've got the tuned package in the base atomic content.  There
> >> are atomic-host and atomic-guest tuned profiles which are currently
> >> identical to the atomic-openshift ones.  We'd like to make a change to
> the
> >> atomic-openshift-node/master profiles (which are distributed with the
> >> openshift product).
> >>
> >> Going fwd, I think we would rather not maintain two locations (atomic-*
> >> and atomic-openshift-* tuned profiles with identical content.
> >>
> >> So, trying to reason a way to get those profiles onto an AH since we
> can't
> >> install the tuned-atomic-openshift RPM...We could copy them to
> /etc/tuned
> >> and enable them manually...but I'm not sure that jives with how we're
> >> supposed to use AH and it seems kind of hacky since there would be
> "orphan
> >> files" in /etc.  Thoughts?
> >
> >
> > With a sufficiently recent version of atomic host, you could use layered
> > packages:
> > http://www.projectatomic.io/blog/2016/08/new-centos-
> atomic-host-with-package-layering-support/
>
> Is that a solution we want to support OCP customers on?
>

I don't see why not. As we go down the path of reducing the size of atomic
host, we'll need to install dependencies that may not make sense to deploy
containerized (storage dependencies for example), so package layering seems
like the proper place to manage those dependencies. I would view the tuned
profiles in the same way.


-- 
Jason DeTiberus

Re: [atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Jason DeTiberus]

On Tue, Oct 11, 2016 at 1:36 PM, Jeremy Eder  wrote:

> Hi,
>
> Right now we've got the tuned package in the base atomic content.  There
> are atomic-host and atomic-guest tuned profiles which are currently
> identical to the atomic-openshift ones.  We'd like to make a change to the
> atomic-openshift-node/master profiles (which are distributed with the
> openshift product).
>
> Going fwd, I think we would rather not maintain two locations (atomic-*
> and atomic-openshift-* tuned profiles with identical content.
>
> So, trying to reason a way to get those profiles onto an AH since we can't
> install the tuned-atomic-openshift RPM...We could copy them to /etc/tuned
> and enable them manually...but I'm not sure that jives with how we're
> supposed to use AH and it seems kind of hacky since there would be "orphan
> files" in /etc.  Thoughts?
>

With a sufficiently recent version of atomic host, you could use layered
packages:
http://www.projectatomic.io/blog/2016/08/new-centos-atomic-host-with-package-layering-support/


-- 
Jason DeTiberus

[atomic-devel] How to apply non-atomic tuned profiles to atomic host

[Jeremy Eder]

Hi,

Right now we've got the tuned package in the base atomic content.  There
are atomic-host and atomic-guest tuned profiles which are currently
identical to the atomic-openshift ones.  We'd like to make a change to the
atomic-openshift-node/master profiles (which are distributed with the
openshift product).

Going fwd, I think we would rather not maintain two locations (atomic-* and
atomic-openshift-* tuned profiles with identical content.

So, trying to reason a way to get those profiles onto an AH since we can't
install the tuned-atomic-openshift RPM...We could copy them to /etc/tuned
and enable them manually...but I'm not sure that jives with how we're
supposed to use AH and it seems kind of hacky since there would be "orphan
files" in /etc.  Thoughts?