Re: [openstack-dev] [TripleO] Spec Minimum Review Proposal

[Macdonald-Wallace, Matthew]

So given the increased complexity of a spec, why not make it 2 specs per week?

Matt

> -Original Message-
> From: Ben Nemec [mailto:openst...@nemebean.com]
> Sent: 23 July 2014 14:21
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [TripleO] Spec Minimum Review Proposal
> 
> For everyone's reference, the tripleo-specs stats can be found here:
> http://www.nemebean.com/reviewstats/tripleo-specs-30.txt
> 
> Note that looking at the stats, over 30 days 1 review per week is only 4, 
> which
> most of our cores are already doing anyway.  I'm not sure codifying a
> requirement to do at least that is going to help much.  To move the needle I'm
> thinking we would need at least 3 - most of our cores aren't meeting that 
> today
> so it would actually require everyone to do more reviews.  Spec reviews are
> difficult and tend to take a significant amount of time, so that would be a
> considerable increase in time commitments for cores.  I'm not sure how I feel
> about that, although I'm probably biased because I'm not at 3 per week right
> now.
> :-)
> 
> -Ben
> 
> On 2014-07-22 15:18, Jay Dobies wrote:
> > At the meetup today, the topic of our spec process came up. The
> > general sentiment is that the process is still young and the hiccups
> > are expected, but we do need to get better about making sure we're
> > staying on top of them.
> >
> > As a first step, it was proposed to add 1 spec review a week to the
> > existing 3 reviews per day requirement for cores.
> >
> > Additionally, we're going to start to capture and review the metrics
> > on spec patches specifically during the weekly meeting. That should
> > help bring to light how long reviews are sitting in the queue without
> > being touched.
> >
> > What are everyone's feelings on adding a 1 spec review per week
> > requirement for cores?
> >
> > Not surprisingly, I'm +1 for it  :)
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] os-refresh-config run frequency

[Macdonald-Wallace, Matthew]

Any chance for getting it streamed or at least IRC'd for those of us who have 
an interest in this but can't attend?

From: Robert Collins [robe...@robertcollins.net]
Sent: 20 July 2014 20:30
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency


Sure. Put it in the agenda perhaps Tuesday morning

On 20 Jul 2014 12:11, "Chris Jones" mailto:c...@tenshu.net>> 
wrote:
Hi

I also have some strong concerns about this. Can we get round a table this week 
and hash it out?

Cheers,
Chris


>> On 20 Jul 2014, at 14:51, Dan Prince 
>> mailto:dpri...@redhat.com>> wrote:
>>
>>> On Thu, 2014-07-17 at 15:54 +0100, Michael Kerrin wrote:
>>> On Thursday 26 June 2014 12:20:30 Clint Byrum wrote:
>>>
>>> Excerpts from Macdonald-Wallace, Matthew's message of 2014-06-26
>> 04:13:31 -0700:
>>
 Hi all,
>>
>>
 I've been working more and more with TripleO recently and whilst
>> it does
>>
 seem to solve a number of problems well, I have found a couple of
>>
 idiosyncrasies that I feel would be easy to address.
>>
>>
 My primary concern lies in the fact that os-refresh-config does
>> not run on
>>
 every boot/reboot of a system. Surely a reboot *is* a
>> configuration
>>
 change and therefore we should ensure that the box has come up in
>> the
>>
 expected state with the correct config?
>>
>>
 This is easily fixed through the addition of an "@reboot" entry in
>>
 /etc/crontab to run o-r-c or (less easily) by re-designing o-r-c
>> to run
>>
 as a service.
>>
>>
 My secondary concern is that through not running os-refresh-config
>> on a
>>
 regular basis by default (i.e. every 15 minutes or something in
>> the same
>>
 style as chef/cfengine/puppet), we leave ourselves exposed to
>> someone
>>
 trying to make a "quick fix" to a production node and taking that
>> node
>>
 offline the next time it reboots because the config was still left
>> as
>>
 broken owing to a lack of updates to HEAT (I'm thinking a "quick
>> change"
>>
 to allow root access via SSH during a major incident that is then
>> left
>>
 unchanged for months because no-one updated HEAT).
>>
>>
 There are a number of options to fix this including Modifying
>>
 os-collect-config to auto-run os-refresh-config on a regular basis
>> or
>>
 setting os-refresh-config to be its own service running via
>> upstart or
>>
 similar that triggers every 15 minutes
>>
>>
 I'm sure there are other solutions to these problems, however I
>> know from
>>
 experience that claiming this is solved through "education of
>> users" or
>>
 (more severely!) via HR is not a sensible approach to take as by
>> the time
>>
 you realise that your configuration has been changed for the last
>> 24
>>
 hours it's often too late!
>>
>>> So I see two problems highlighted above.
>>
>>
>>> 1) We don't re-assert ephemeral state set by o-r-c scripts. You're
>> right,
>>
>>> and we've been talking about it for a while. The right thing to do
>> is
>>
>>> have os-collect-config re-run its command on boot. I don't think a
>> cron
>>
>>> job is the right way to go, we should just have a file in /var/run
>> that
>>
>>> is placed there only on a successful run of the command. If that
>> file
>>
>>> does not exist, then we run the command.
>>
>>
>>> I've just opened this bug in response:
>>
>>
>>> https://bugs.launchpad.net/os-collect-config/+bug/1334804
>>
>>
>>
>>
>> I have been looking into bug #1334804 and I have a review up to
>> resolve it. I want to highlight something.
>>
>>
>>
>> Currently on a reboot we start all services via upstart (on debian
>> anyways) and there have been quite a lot of issues around this -
>> missing upstart scripts and timing issues. I don't know the issues on
>> fedora.
>>
>>
>>
>> So with a fix to #1334804, on a reboot upstart will start all the
>> services first (with potentially out-of-date configuration), then
>> o-c-c will start o-r-c and will now configure all services and restart
>> them or start them if upstart isn't configured properly.
>>
>>
>>
>> I would like to turn off all boot scripts for services we configure
>> and leave all this to o-r-c. I think this will simplify things and put
>> us in control of starting services. I believe that it will also narrow
>> the gap between fedora and debian or debian and debian so what works
>> on one should work on the other and make it easier for developers.
>
> I'm not sold on this approach. At the very least I think we want to make
> this optional because not all deployments may want to have o-r-c be the
> central service starting agent. So I'm opposed to this being our (only!)
> default...
>
> The job of o-r-c in this regard is to assert state... which to me means
> making sure that a service is configured correctly (config files, set to
> start on boot, and initially started). Requiring o-r-c to be the service
> starting agent (always

Re: [openstack-dev] [TripleO] Proposal to add Jon Paul Sullivan and Alexis Lee to core review team

[Macdonald-Wallace, Matthew]

+1 from me.

Matt

> -Original Message-
> From: Clint Byrum [mailto:cl...@fewbar.com]
> Sent: 09 July 2014 16:52
> To: openstack-dev@lists.openstack.org
> Subject: [openstack-dev] [TripleO] Proposal to add Jon Paul Sullivan and 
> Alexis
> Lee to core review team
> 
> Hello!
> 
> I've been looking at the statistics, and doing a bit of review of the 
> reviewers, and
> I think we have an opportunity to expand the core reviewer team in TripleO. We
> absolutely need the help, and I think these two individuals are well 
> positioned to
> do that.
> 
> I would like to draw your attention to this page:
> 
> http://russellbryant.net/openstack-stats/tripleo-reviewers-90.txt
> 
> Specifically these two lines:
> 
> +---+---++
> |  Reviewer | Reviews   -2  -1  +1  +2  +A+/- % | Disagreements* |
> +---+---++
> |  jonpaul-sullivan | 1880  43 145   0   077.1% |   28 ( 14.9%)  |
> |   lxsli   | 1860  23 163   0   087.6% |   27 ( 14.5%)  |
> 
> Note that they are right at the level we expect, 3 per work day. And I've 
> looked
> through their reviews and code contributions: it is clear that they understand
> what we're trying to do in TripleO, and how it all works. I am a little 
> dismayed at
> the slightly high disagreement rate, but looking through the disagreements,
> most of them were jp and lxsli being more demanding of submitters, so I am 
> less
> dismayed.
> 
> So, I propose that we add jonpaul-sullivan and lxsli to the TripleO core 
> reviewer
> team.
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] os-refresh-config run frequency

[Macdonald-Wallace, Matthew]

And the spec is now up at https://review.openstack.org/104524 for everyone to 
pull apart... ;)

Matt

> -Original Message-
> From: Macdonald-Wallace, Matthew
> Sent: 03 July 2014 11:18
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency
> 
> FWIW, I've just registered https://blueprints.launchpad.net/tripleo/+spec/re-
> assert-system-state and I'm about to start work on the spec.
> 
> Matt
> 
> > -Original Message-
> > From: Clint Byrum [mailto:cl...@fewbar.com]
> > Sent: 27 June 2014 17:01
> > To: openstack-dev
> > Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency
> >
> > Excerpts from Macdonald-Wallace, Matthew's message of 2014-06-27
> > 00:14:49
> > -0700:
> > > Hi Clint,
> > >
> > > > -Original Message-
> > > > From: Clint Byrum [mailto:cl...@fewbar.com]
> > > > Sent: 26 June 2014 20:21
> > > > To: openstack-dev
> > > > Subject: Re: [openstack-dev] [TripleO] os-refresh-config run
> > > > frequency
> > > >
> > >
> > > > So I see two problems highlighted above.
> > > >
> > > > 1) We don't re-assert ephemeral state set by o-r-c scripts. You're
> > > > right, and we've been talking about it for a while. The right
> > > > thing to do is have os-collect- config re-run its command on boot.
> > > > I don't think a cron job is the right way to go, we should just
> > > > have a file in /var/run that is placed there only on a successful
> > > > run of the command. If
> > that file does not exist, then we run the command.
> > > >
> > > > I've just opened this bug in response:
> > > >
> > > > https://bugs.launchpad.net/os-collect-config/+bug/1334804
> > >
> > >
> > > Cool, I'm more than happy for this to be done elsewhere, I'm glad
> > > that people
> > are in agreement with me on the concept and that work has already
> > started on this.
> > >
> > > I'll add some notes to the bug if needed later on today.
> > >
> > > > 2) We don't re-assert any state on a regular basis.
> > > >
> > > > So one reason we haven't focused on this, is that we have a
> > > > stretch goal of running with a readonly root partition. It's
> > > > gotten lost in a lot of the craziness of "just get it working",
> > > > but with rebuilds blowing away root now, leading to anything not
> > > > on the state drive (/mnt currently), there's a good chance that this 
> > > > will
> work relatively well.
> > > >
> > > > Now, since people get root, they can always override the readonly
> > > > root and make changes. we hates thiss!.
> > > >
> > > > I'm open to ideas, however, os-refresh-config is definitely not
> > > > the place to solve this. It is intended as a non-resident command
> > > > to be called when it is time to assert state. os-collect-config is
> > > > intended to gather configurations, and expose them to a command
> > > > that it runs, and thus should be the mechanism by which os- 
> > > > refresh-config
> is run.
> > > >
> > > > I'd like to keep this conversation separate from one in which we
> > > > discuss more mechanisms to make os-refresh-config robust. There
> > > > are a bunch of things we can do, but I think we should focus just
> > > > on "how do we
> > re-assert state?".
> > >
> > > OK, that's fair enough.
> > >
> > > > Because we're able to say right now that it is only for running
> > > > when config changes, we can wave our hands and say it's ok that we
> > > > restart everything on every run. As Jan alluded to, that won't
> > > > work so well if we run it every 20 minutes.
> > >
> > > Agreed, and chatting with Jan and a couple of others yesterday we
> > > came to
> > the conclusion that whatever we do here it will require "tweaking" of
> > a number of elements to safely restart services.
> > >
> > > > So, I wonder if we can introduce a config version into 
> > > > os-collect-config.
> > > >
> > > > Basically os-collect-config would keep a version along with its cache.
> > > > Wh

Re: [openstack-dev] [TripleO] os-refresh-config run frequency

[Macdonald-Wallace, Matthew]

FWIW, I've just registered 
https://blueprints.launchpad.net/tripleo/+spec/re-assert-system-state and I'm 
about to start work on the spec.

Matt

> -Original Message-
> From: Clint Byrum [mailto:cl...@fewbar.com]
> Sent: 27 June 2014 17:01
> To: openstack-dev
> Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency
> 
> Excerpts from Macdonald-Wallace, Matthew's message of 2014-06-27 00:14:49
> -0700:
> > Hi Clint,
> >
> > > -Original Message-
> > > From: Clint Byrum [mailto:cl...@fewbar.com]
> > > Sent: 26 June 2014 20:21
> > > To: openstack-dev
> > > Subject: Re: [openstack-dev] [TripleO] os-refresh-config run
> > > frequency
> > >
> >
> > > So I see two problems highlighted above.
> > >
> > > 1) We don't re-assert ephemeral state set by o-r-c scripts. You're
> > > right, and we've been talking about it for a while. The right thing
> > > to do is have os-collect- config re-run its command on boot. I don't
> > > think a cron job is the right way to go, we should just have a file
> > > in /var/run that is placed there only on a successful run of the command. 
> > > If
> that file does not exist, then we run the command.
> > >
> > > I've just opened this bug in response:
> > >
> > > https://bugs.launchpad.net/os-collect-config/+bug/1334804
> >
> >
> > Cool, I'm more than happy for this to be done elsewhere, I'm glad that 
> > people
> are in agreement with me on the concept and that work has already started on
> this.
> >
> > I'll add some notes to the bug if needed later on today.
> >
> > > 2) We don't re-assert any state on a regular basis.
> > >
> > > So one reason we haven't focused on this, is that we have a stretch
> > > goal of running with a readonly root partition. It's gotten lost in
> > > a lot of the craziness of "just get it working", but with rebuilds
> > > blowing away root now, leading to anything not on the state drive
> > > (/mnt currently), there's a good chance that this will work relatively 
> > > well.
> > >
> > > Now, since people get root, they can always override the readonly
> > > root and make changes. we hates thiss!.
> > >
> > > I'm open to ideas, however, os-refresh-config is definitely not the
> > > place to solve this. It is intended as a non-resident command to be
> > > called when it is time to assert state. os-collect-config is
> > > intended to gather configurations, and expose them to a command that
> > > it runs, and thus should be the mechanism by which os- refresh-config is 
> > > run.
> > >
> > > I'd like to keep this conversation separate from one in which we
> > > discuss more mechanisms to make os-refresh-config robust. There are
> > > a bunch of things we can do, but I think we should focus just on "how do 
> > > we
> re-assert state?".
> >
> > OK, that's fair enough.
> >
> > > Because we're able to say right now that it is only for running when
> > > config changes, we can wave our hands and say it's ok that we
> > > restart everything on every run. As Jan alluded to, that won't work
> > > so well if we run it every 20 minutes.
> >
> > Agreed, and chatting with Jan and a couple of others yesterday we came to
> the conclusion that whatever we do here it will require "tweaking" of a number
> of elements to safely restart services.
> >
> > > So, I wonder if we can introduce a config version into os-collect-config.
> > >
> > > Basically os-collect-config would keep a version along with its cache.
> > > Whenever a new version is detected, os-collect-config would set a
> > > value in the environment that informs the command "this is a new
> > > version of config". From that, scripts can do things like this:
> > >
> > > if [ -n "$OS_CONFIG_NEW_VERSION" ] ; then
> > >   service X restart
> > > else
> > >   if !service X status ; then service X start fi
> > >
> > > This would lay the groundwork for future abilities to compare
> > > old/new so we can take shortcuts by diffing the two config versions.
> > > For instance if we look at old vs. new and we don't see any of the
> > > keys we care about changed, we can skip restarting.
> >
> > I like this approach - does this require a new spec? If so, I'll start an 
> > etherpad
> to collect thoughts on it before writing it up for approval.
> 
> I think this should be a tripleo spec. If you're volunteering write it, 
> hooray \o/. It
> will require several work items. Off the top of my head:
> 
> - Add version awareness to os-collect-config
> - Add version awareness to all os-refresh-config scripts that do
>   disruptive things.
> - Add periodic command run to os-collect-config
> 
> Let's call it 're-assert-system-state'. Sound good?
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] os-refresh-config run frequency

[Macdonald-Wallace, Matthew]

> -Original Message-
> From: Chris Jones [mailto:c...@tenshu.net]
> Sent: 26 June 2014 20:58
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency
> 
> Hi
> 
> On 26 Jun 2014, at 20:55, Clint Byrum  wrote:
> 
> > Let's not focus on the else, but the primary condition which is "If we
> > changed, restart". The point is, if we didn't change, try not to be
> > disruptive.
> 
> Agreed :)

+1

Matt

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] os-refresh-config run frequency

[Macdonald-Wallace, Matthew]

Hi Chris,

> -Original Message-
> From: Chris Jones [mailto:c...@tenshu.net]
> Sent: 26 June 2014 20:38
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency
> 
> Hi
> 
> Given...
> 
> On 26 Jun 2014, at 20:20, Clint Byrum  wrote:
> > we should just have a file in /var/run that
> 
> and...
> 
> > I think we should focus just on "how do we re-assert state?".
> 
> ... for the reboot case, we could have os-collect-config check for the 
> presence
> of the /var/run file when it starts, if it doesn't find it, unconditionally 
> call o-r-c
> and then write out the file. Given that we're starting o-c-c on boot, this 
> seems
> like a fairly simple way to get o-r-c to run on boot (and one that could be 
> trivially
> disabled by configuration or just dumbly pre-creating the /var/run file).

Sounds good.

> > Whenever a new version is detected, os-collect-config would set a
> > value in the environment that informs the command "this is a new
> > version of
> 
> I like the idea of exposing the fact that a new config version has arrived, 
> to o-r-c
> scripts, but...
> 
> >  if !service X status ; then service X start
> 
> ... I always worry when I see suggestions to have periodic state-assertion 
> tasks
> take care of starting services that are not running, but in this case I will 
> try to
> calm my nerves with the knowledge that service(1) is almost certainly talking 
> to
> a modern init which is perfectly capable of supervising daemons :D

Heh, we could always insist on supervisord as a dependency for all elements 
that run a service... ;) 

Matt


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] os-refresh-config run frequency

[Macdonald-Wallace, Matthew]

Hi Clint,

> -Original Message-
> From: Clint Byrum [mailto:cl...@fewbar.com]
> Sent: 26 June 2014 20:21
> To: openstack-dev
> Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency
>

> So I see two problems highlighted above.
> 
> 1) We don't re-assert ephemeral state set by o-r-c scripts. You're right, and
> we've been talking about it for a while. The right thing to do is have 
> os-collect-
> config re-run its command on boot. I don't think a cron job is the right way 
> to
> go, we should just have a file in /var/run that is placed there only on a 
> successful
> run of the command. If that file does not exist, then we run the command.
> 
> I've just opened this bug in response:
> 
> https://bugs.launchpad.net/os-collect-config/+bug/1334804


Cool, I'm more than happy for this to be done elsewhere, I'm glad that people 
are in agreement with me on the concept and that work has already started on 
this.

I'll add some notes to the bug if needed later on today.

> 2) We don't re-assert any state on a regular basis.
> 
> So one reason we haven't focused on this, is that we have a stretch goal of
> running with a readonly root partition. It's gotten lost in a lot of the 
> craziness of
> "just get it working", but with rebuilds blowing away root now, leading to
> anything not on the state drive (/mnt currently), there's a good chance that 
> this
> will work relatively well.
> 
> Now, since people get root, they can always override the readonly root and
> make changes. we hates thiss!.
> 
> I'm open to ideas, however, os-refresh-config is definitely not the place to 
> solve
> this. It is intended as a non-resident command to be called when it is time to
> assert state. os-collect-config is intended to gather configurations, and 
> expose
> them to a command that it runs, and thus should be the mechanism by which os-
> refresh-config is run.
> 
> I'd like to keep this conversation separate from one in which we discuss more
> mechanisms to make os-refresh-config robust. There are a bunch of things we
> can do, but I think we should focus just on "how do we re-assert state?".

OK, that's fair enough.

> Because we're able to say right now that it is only for running when config
> changes, we can wave our hands and say it's ok that we restart everything on
> every run. As Jan alluded to, that won't work so well if we run it every 20
> minutes.

Agreed, and chatting with Jan and a couple of others yesterday we came to the 
conclusion that whatever we do here it will require "tweaking" of a number of 
elements to safely restart services.

> So, I wonder if we can introduce a config version into os-collect-config.
> 
> Basically os-collect-config would keep a version along with its cache.
> Whenever a new version is detected, os-collect-config would set a value in the
> environment that informs the command "this is a new version of config". From
> that, scripts can do things like this:
> 
> if [ -n "$OS_CONFIG_NEW_VERSION" ] ; then
>   service X restart
> else
>   if !service X status ; then service X start fi
> 
> This would lay the groundwork for future abilities to compare old/new so we
> can take shortcuts by diffing the two config versions. For instance if we 
> look at
> old vs. new and we don't see any of the keys we care about changed, we can
> skip restarting.

I like this approach - does this require a new spec? If so, I'll start an 
etherpad to collect thoughts on it before writing it up for approval.

Matt

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] os-refresh-config run frequency

[Macdonald-Wallace, Matthew]

Hi all,

I've been working more and more with TripleO recently and whilst it does seem 
to solve a number of problems well, I have found a couple of idiosyncrasies 
that I feel would be easy to address.

My primary concern lies in the fact that os-refresh-config does not run on 
every boot/reboot of a system.  Surely a reboot *is* a configuration change and 
therefore we should ensure that the box has come up in the expected state with 
the correct config?

This is easily fixed through the addition of an "@reboot" entry in /etc/crontab 
to run o-r-c or (less easily) by re-designing o-r-c to run as a service.

My secondary concern is that through not running os-refresh-config on a regular 
basis by default (i.e. every 15 minutes or something in the same style as 
chef/cfengine/puppet), we leave ourselves exposed to someone trying to make a 
"quick fix" to a production node and taking that node offline the next time it 
reboots because the config was still left as broken owing to a lack of updates 
to HEAT (I'm thinking a "quick change" to allow root access via SSH during a 
major incident that is then left unchanged for months because no-one updated 
HEAT).

There are a number of options to fix this including Modifying os-collect-config 
to auto-run os-refresh-config on a regular basis or setting os-refresh-config 
to be its own service running via upstart or similar that triggers every 15 
minutes

I'm sure there are other solutions to these problems, however I know from 
experience that claiming this is solved through "education of users" or (more 
severely!) via HR is not a sensible approach to take as by the time you realise 
that your configuration has been changed for the last 24 hours it's often too 
late!

I'd welcome thoughts on the above,

Kind regards,

Matt

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Reviews - we need your help!

[Macdonald-Wallace, Matthew]

Thanks Tomas,

http://bit.ly/1lsg3SH now contains the missing projects and has been re-ordered 
slightly so that you see outdated reviews first then the Jenkins/-1 stuff.

Cheers,

Matt

> -Original Message-
> From: Tomas Sedovic [mailto:tsedo...@redhat.com]
> Sent: 12 June 2014 19:03
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [TripleO] Reviews - we need your help!
> 
> On 12/06/14 16:02, Macdonald-Wallace, Matthew wrote:
> > FWIW, I've tried to make a useful dashboard for this using Sean
> > Dague's gerrit-dash-creator [0].
> >
> >
> >
> > Short URL is http://bit.ly/1l4DLFS long url is:
> >
> >
> >
> >
> https://review.openstack.org/#/dashboard/?foreach=%28project%3Aopenstack
> %2Ftripleo-incubator+OR+project%3Aopenstack%2Ftripleo-image-
> elements+OR+project%3Aopenstack%2Ftripleo-heat-
> templates+OR+project%3Aopenstack%2Ftripleo-
> specs+OR+project%3Aopenstack%2Fos-apply-
> config+OR+project%3Aopenstack%2Fos-collect-
> config+OR+project%3Aopenstack%2Fos-refresh-
> config+OR+project%3Aopenstack%2Fdiskimage-
> builder%29+NOT+owner%3Aself+NOT+label%3AWorkflow%3C%3D-
> 1+label%3AVerified%3E%3D1%252cjenkins+NOT+label%3ACode-
> Review%3E%3D-
> 2%252cself+branch%3Amaster+status%3Aopen&title=TripleO+Reviews&Your+a
> re+a+reviewer%2C+but+haven%27t+voted+in+the+current+revision=reviewer%
> 3Aself&Passed+Jenkins%2C+No+Negative+Feedback=NOT+label%3ACode-
> Review%3C%3D-
> 1+limit%3A100&Changes+with+no+code+review+in+the+last+48hrs=NOT+label
> %3ACode-
> Review%3C%3D2+age%3A48h&Changes+with+no+code+review+in+the+last+5+
> days=NOT+label%3ACode-
> Review%3C%3D2+age%3A5d&Changes+with+no+code+review+in+the+last+7+d
> ays=NOT+label%3ACode-Review%!
>  3C%3D2+age
> %3A7d&Some+adjustment+required+%28-1+only%29=label%3ACode-
> Review%3D-1+NOT+label%3ACode-Review%3D-
> 2+limit%3A100&Dead+Specs+%28-2%29=label%3ACode-Review%3C%3D-2
> >
> >
> >
> > I'll add it to my fork and submit a PR if people think it useful.
> 
> I was about to mention this, too. The gerrit-dash-creator is fantastic.
> 
> This one is missing the Tuskar-related projects (openstack/tuskar,
> openstack/tuskar-ui and openstack/python-tuskarclient) and also openstack/os-
> cloud-config, though.
> 
> 
> >
> >
> >
> > Matt
> >
> >
> >
> > [0] https://github.com/sdague/gerrit-dash-creator
> >
> >
> >
> > *From:*James Polley [mailto:j...@jamezpolley.com]
> > *Sent:* 12 June 2014 06:08
> > *To:* OpenStack Development Mailing List (not for usage questions)
> > *Subject:* [openstack-dev] [TripleO] Reviews - we need your help!
> >
> >
> >
> > During yesterday's IRC meeting, we realized that our review stats are
> > starting to slip again.
> >
> > Just after summit, our stats were starting to improve. In the
> > 2014-05-20 meeting, the TripleO  "Stats since the last revision
> > without -1 or -2"[1] looked like this:
> >
> > 1rd quartile wait time: 1 days, 1 hours, 11 minutes
> >
> > Median wait time: 6 days, 9 hours, 49 minutes
> >
> > 3rd quartile wait time: 13 days, 5 hours, 46 minutes
> >
> >
> >
> > As of yesterdays meeting, we have:
> >
> > 1rd quartile wait time: 4 days, 23 hours, 19 minutes
> >
> > Median wait time: 7 days, 22 hours, 8 minutes
> >
> > 3rd quartile wait time: 13 days, 19 hours, 17 minutes
> >
> >
> >
> > This really hurts our velocity, and is especially hard on people
> > making their first commit, as it can take them almost a full work week
> > before they even get their first feedback.
> >
> > To get things moving, we need everyone to make a special effort to do
> > a few reviews every day. It would be most helpful if you can look for
> > older reviews without a -1 or -2 and help those reviews get over the line.
> >
> > If you find reviews that are just waiting for a simple fix - typo or
> > syntax fixes, simple code fixes, or a simple rebase - it would be even
> > more helpful if you could take a few minutes to make those patches,
> > rather than just leaving the review waiting for the attention of the
> > original submitter.
> >
> > Please keep in mind that these stats are based on all of our projects,
> > not just tripleo-incubator. To save you heading to the wiki, here's a
> > handy link that shows you all open code reviews in all our projects:
> >
> > bit.ly/1hQco1N <http://bit.ly/1hQco1N>
> >
> > If you'd prefer the long version:
> > ht

Re: [openstack-dev] [TripleO] Reviews - we need your help!

[Macdonald-Wallace, Matthew]

FWIW, I’ve tried to make a useful dashboard for this using Sean Dague’s 
gerrit-dash-creator [0].

Short URL is http://bit.ly/1l4DLFS long url is:

https://review.openstack.org/#/dashboard/?foreach=%28project%3Aopenstack%2Ftripleo-incubator+OR+project%3Aopenstack%2Ftripleo-image-elements+OR+project%3Aopenstack%2Ftripleo-heat-templates+OR+project%3Aopenstack%2Ftripleo-specs+OR+project%3Aopenstack%2Fos-apply-config+OR+project%3Aopenstack%2Fos-collect-config+OR+project%3Aopenstack%2Fos-refresh-config+OR+project%3Aopenstack%2Fdiskimage-builder%29+NOT+owner%3Aself+NOT+label%3AWorkflow%3C%3D-1+label%3AVerified%3E%3D1%252cjenkins+NOT+label%3ACode-Review%3E%3D-2%252cself+branch%3Amaster+status%3Aopen&title=TripleO+Reviews&Your+are+a+reviewer%2C+but+haven%27t+voted+in+the+current+revision=reviewer%3Aself&Passed+Jenkins%2C+No+Negative+Feedback=NOT+label%3ACode-Review%3C%3D-1+limit%3A100&Changes+with+no+code+review+in+the+last+48hrs=NOT+label%3ACode-Review%3C%3D2+age%3A48h&Changes+with+no+code+review+in+the+last+5+days=NOT+label%3ACode-Review%3C%3D2+age%3A5d&Changes+with+no+code+review+in+the+last+7+days=NOT+label%3ACode-Review%3C%3D2+age%3A7d&Some+adjustment+required+%28-1+only%29=label%3ACode-Review%3D-1+NOT+label%3ACode-Review%3D-2+limit%3A100&Dead+Specs+%28-2%29=label%3ACode-Review%3C%3D-2

I’ll add it to my fork and submit a PR if people think it useful.

Matt

[0] https://github.com/sdague/gerrit-dash-creator

From: James Polley [mailto:j...@jamezpolley.com]
Sent: 12 June 2014 06:08
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [TripleO] Reviews - we need your help!

During yesterday's IRC meeting, we realized that our review stats are starting 
to slip again.
Just after summit, our stats were starting to improve. In the 2014-05-20 
meeting, the TripleO  "Stats since the last revision without -1 or -2"[1] 
looked like this:

1rd quartile wait time: 1 days, 1 hours, 11 minutes

Median wait time: 6 days, 9 hours, 49 minutes

3rd quartile wait time: 13 days, 5 hours, 46 minutes

As of yesterdays meeting, we have:

1rd quartile wait time: 4 days, 23 hours, 19 minutes

Median wait time: 7 days, 22 hours, 8 minutes

3rd quartile wait time: 13 days, 19 hours, 17 minutes

This really hurts our velocity, and is especially hard on people making their 
first commit, as it can take them almost a full work week before they even get 
their first feedback.
To get things moving, we need everyone to make a special effort to do a few 
reviews every day. It would be most helpful if you can look for older reviews 
without a -1 or -2 and help those reviews get over the line.
If you find reviews that are just waiting for a simple fix - typo or syntax 
fixes, simple code fixes, or a simple rebase - it would be even more helpful if 
you could take a few minutes to make those patches, rather than just leaving 
the review waiting for the attention of the original submitter.
Please keep in mind that these stats are based on all of our projects, not just 
tripleo-incubator. To save you heading to the wiki, here's a handy link that 
shows you all open code reviews in all our projects:

bit.ly/1hQco1N

If you'd prefer the long version:
https://review.openstack.org/#/q/status:open+%28project:openstack/tripleo-incubator+OR+project:openstack/tuskar+OR+project:openstack/tuskar-ui+OR+project:openstack-infra/tripleo-ci+OR+project:openstack/os-apply-config+OR+project:openstack/os-collect-config+OR+project:openstack/os-refresh-config+OR+project:openstack/os-cloud-config+OR+project:openstack/tripleo-image-elements+OR+project:openstack/tripleo-heat-templates+OR+project:openstack/diskimage-builder+OR+project:openstack/python-tuskarclient+OR+project:openstack/tripleo-specs%29,n,z

https://wiki.openstack.org/wiki/TripleO#Bug_Triage has links to reviews for 
individual projects as well as more information about how we triage bugs.

[1] http://www.nemebean.com/reviewstats/tripleo-open.html
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Location of Monitoring Checks

[Macdonald-Wallace, Matthew]

Thanks all for the comments and I’m glad you agree with one of the proposed 
approaches! ☺

I’ll take a look at using an env var first and then look at a check_mk.d style 
directory if I get a chance later on.

Thanks again,

Matt

From: Robert Collins [mailto:robe...@robertcollins.net]
Sent: 29 April 2014 23:04
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [tripleo] Location of Monitoring Checks


Or have a check-mk.d directory and pull stuff on from there automatically
On 30 Apr 2014 04:03, "Gregory Haynes" 
mailto:g...@greghaynes.net>> wrote:
> > Hi all,
> >
> > I have a patch in flight at the moment [0] to install check_mk server and 
> > compliment the already merged installation of check_mk agent [1] so my 
> > thoughts are now turning to how we would recommend adding new service 
> > checks.
> >
> > The concept behind check_mk makes this really simple to do.  You just place 
> > a script that outputs "   
> > " into the agent's "local" directory 
> > (/usr/lib/check_mk_agent/local on Ubuntu for example) and it will be picked 
> > up the next time an inventory of the system is run.
> >
> > There are two ways that we can recommend doing this:
> >
> > 1) We ask users to update the check_mk_agent T-I-E every time they wish to 
> > add a new check
> > 2) We ask users to distribute checks from their own T-I-E into the correct 
> > location
> >
> > In my opinion, requiring an update to check_mk_agent for every new check is 
> > the wrong way of doing this as it means that all systems get all checks 
> > regardless of function.  Far more preferable would be option 2, however I'm 
> > open to other ideas, especially if they mean that organisations using this 
> > don't have to go through the review process if they have checks they wish 
> > to keep "behind the firewall" for IP/Licensing reasons.
>
> Agreed.  Option 2 sounds like the only way to go here.  Adding
> instructions on how and where to add a check to the README file and
> maybe having a sample check element that users can look at for reference
> should be sufficient, I would think.
>

++ This is a common pattern in many of the elements. Additionally, It
might be worth exporting a variable in check_mk's environment.d for
other elements to use as the agent local directory if this path isnt
entirely consistent across distros.

> >
> > Thanks in advance for any help people can give on this,
> >
> > Matt
> >
> > [0] https://review.openstack.org/#/c/87226/
> > [1] https://review.openstack.org/#/c/81485/
> >

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] Location of Monitoring Checks

[Macdonald-Wallace, Matthew]

Hi all,

I have a patch in flight at the moment [0] to install check_mk server and 
compliment the already merged installation of check_mk agent [1] so my thoughts 
are now turning to how we would recommend adding new service checks.

The concept behind check_mk makes this really simple to do.  You just place a 
script that outputs "   " 
into the agent's "local" directory (/usr/lib/check_mk_agent/local on Ubuntu for 
example) and it will be picked up the next time an inventory of the system is 
run.

There are two ways that we can recommend doing this:

1) We ask users to update the check_mk_agent T-I-E every time they wish to add 
a new check
2) We ask users to distribute checks from their own T-I-E into the correct 
location

In my opinion, requiring an update to check_mk_agent for every new check is the 
wrong way of doing this as it means that all systems get all checks regardless 
of function.  Far more preferable would be option 2, however I'm open to other 
ideas, especially if they mean that organisations using this don't have to go 
through the review process if they have checks they wish to keep "behind the 
firewall" for IP/Licensing reasons.

Thanks in advance for any help people can give on this,

Matt

[0] https://review.openstack.org/#/c/87226/
[1] https://review.openstack.org/#/c/81485/


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Tripleo] Reviews wanted for new TripleO elements

[Macdonald-Wallace, Matthew]

Apologies to both.

I have been asking in IRC but with little luck and had missed that original 
email.

I'll stick to IRC... :)

Matt

> -Original Message-
> From: Ben Nemec [mailto:openst...@nemebean.com]
> Sent: 21 April 2014 16:59
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [Tripleo] Reviews wanted for new TripleO elements
> 
> Please don't make review requests on the list.  Details here:
> http://lists.openstack.org/pipermail/openstack-dev/2013-
> September/015264.html
> 
> Thanks.
> 
> -Ben
> 
> On 04/20/2014 02:44 PM, Macdonald-Wallace, Matthew wrote:
> > Hi all,
> >
> > Can I please ask for some reviews on the following:
> >
> > https://review.openstack.org/#/c/87226/ - Install checkmk_agent
> > https://review.openstack.org/#/c/87223/ - Install icinga cgi interface
> >
> > I already have a souple of +1s and jenkins is happy, all I need is +2
> > and +A! :)
> >
> > Thanks,
> >
> > Matt
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Tripleo] Reviews wanted for new TripleO elements

[Macdonald-Wallace, Matthew]

Hi all,

Can I please ask for some reviews on the following:

https://review.openstack.org/#/c/87226/ - Install checkmk_agent
https://review.openstack.org/#/c/87223/ - Install icinga cgi interface

I already have a souple of +1s and jenkins is happy, all I need is +2 and +A! :)

Thanks,

Matt

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposed Logging Standards

[Macdonald-Wallace, Matthew]

> -Original Message-
> From: Ben Nemec [mailto:openst...@nemebean.com]
> Sent: 31 January 2014 16:01
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] Proposed Logging Standards
> > I'd just love to see the ability in the python logger to include the
> > application name, not just the class/module that created the log
> > message (it's in 3. but I don't think we can justify a
> > switch to Python 3 just based on logging!):
> >
> > 
> >   
> >
> > At the moment, all of the above is possible except for the
> >  part.  Is there anything we can do to add this to the
> > context or similar?
> >
> > Matt
> 
> Wish granted. :-)

YAY! \o/
 
> https://github.com/openstack/oslo-
> incubator/commit/8c3046b78dca8eae1d911e3421b5938c19f20c37
> 
> The plan is to turn that on by default as soon as we can get through the
> deprecation process for the existing format.

Awesome, good to hear.

Thanks,

Matt

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposed Logging Standards

[Macdonald-Wallace, Matthew]

> -Original Message-
> From: Sean Dague [mailto:s...@dague.net]
> Sent: 31 January 2014 12:29
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] Proposed Logging Standards
> 
> On 01/30/2014 07:15 PM, John Dickinson wrote:
> > 1) Every log message is one line (ends with \n) and the log fields are
> > space-delineated. eg (`log_line = ' '.join(urllib.quote(x) for x in
> > log_fields_list)`)

+1 for this - multiple lines (even in in DEBUG mode!) are a PITA to handle with 
most log analyser software.

>
> > 2) The only definition of a log format is the prefix and the message is a 
> > set of
> fields defined by the service actually doing the logging.
> 
> So, actually, most of my concern at this point wasn't the line format.
> It was the concern about when projects were calling the loggers, and what kind
> of information should be logged at each level.
> 
> Given that most projects are using the oslo defaults today, much of the line
> format is handled. I think that if you have concerns on that front, it's 
> probably a
> different conversation with the oslo team.
> 
> I do agree we should standards a little more on project (i.e. logger "name"),
> because in most projects this is just defaulting to module.
> Which is fine for debug level, but not very user friendly at ops levels.

I'd just love to see the ability in the python logger to include the 
application name, not just the class/module that created the log message (it's 
in 3. but I don't think we can justify a switch to Python 3 just 
based on logging!):

  
 

At the moment, all of the above is possible except for the  part. 
 Is there anything we can do to add this to the context or similar?

Matt

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposed Logging Standards

[Macdonald-Wallace, Matthew]

No idea, I only really work on Nova, but as this is in Oslo I expect so!

Matt

> -Original Message-
> From: Sanchez, Cristian A [mailto:cristian.a.sanc...@intel.com]
> Sent: 30 January 2014 13:44
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] Proposed Logging Standards
> 
> Hi Matt,
> What about the rest of the components? Do they also have this capability?
> Thanks
> 
> Cristian
> 
> On 30/01/14 04:59, "Macdonald-Wallace, Matthew"
>  wrote:
> 
> >Hi Cristian,
> >
> >The functionality already exists within Openstack (certainly it's there
> >in Nova) it's just not very well documented (something I keep meaning
> >to
> >do!)
> >
> >Basically you need to add the following to your nova.conf file:
> >
> >log_config=/etc/nova/logging.conf
> >
> >And then create /etc/nova/logging.conf with the configuration you want
> >to use based on the Python Logging Module's "ini" configuration format.
> >
> >Hope that helps,
> >
> >Matt
> >
> >> -Original Message-
> >> From: Sanchez, Cristian A [mailto:cristian.a.sanc...@intel.com]
> >> Sent: 29 January 2014 17:57
> >> To: OpenStack Development Mailing List (not for usage questions)
> >> Subject: Re: [openstack-dev] Proposed Logging Standards
> >>
> >> Hi Matthew,
> >> I¹m interested to help in this switch to python logging framework for
> >>shipping to  logstash/etc. Are you working on a blueprint for this?
> >> Cheers,
> >>
> >> Cristian
> >>
> >> On 27/01/14 11:07, "Macdonald-Wallace, Matthew"
> >>  wrote:
> >>
> >> >Hi Sean,
> >> >
> >> >I'm currently working on moving away from the "built-in" logging to
> >> >use log_config= and the python logging framework so that
> >> >we can start shipping to logstash/sentry/.
> >> >
> >> >I'd be very interested in getting involved in this, especially from
> >> >a "why do we have log messages that are split across multiple lines"
> >> >perspective!
> >> >
> >> >Cheers,
> >> >
> >> >Matt
> >> >
> >> >P.S. FWIW, I'd also welcome details on what the "Audit" level gives
> >> >us that the others don't... :)
> >> >
> >> >> -Original Message-
> >> >> From: Sean Dague [mailto:s...@dague.net]
> >> >> Sent: 27 January 2014 13:08
> >> >> To: OpenStack Development Mailing List
> >> >> Subject: [openstack-dev] Proposed Logging Standards
> >> >>
> >> >> Back at the beginning of the cycle, I pushed for the idea of doing
> >> >>some log  harmonization, so that the OpenStack logs, across
> >> >>services, made sense.
> >> >>I've
> >> >> pushed a proposed changes to Nova and Keystone over the past
> >> >>couple of days.
> >> >>
> >> >> This is going to be a long process, so right now I want to just
> >> >>focus on making  INFO level sane, because as someone that spends a
> >> >>lot of time staring at logs in  test failures, I can tell you it
> >> >>currently isn't.
> >> >>
> >> >> https://wiki.openstack.org/wiki/LoggingStandards is a few things
> >> >>I've written  down so far, comments welcomed.
> >> >>
> >> >> We kind of need to solve this set of recommendations once and for
> >> >>all up front,  because negotiating each change, with each project,
> >> >>isn't going to work (e.g -
> >> >> https://review.openstack.org/#/c/69218/)
> >> >>
> >> >> What I'd like to find out now:
> >> >>
> >> >> 1) who's interested in this topic?
> >> >> 2) who's interested in helping flesh out the guidelines for
> >> >>various log levels?
> >> >> 3) who's interested in helping get these kinds of patches into
> >> >>various projects in  OpenStack?
> >> >> 4) which projects are interested in participating (i.e. interested
> >> >>in prioritizing  landing these kinds of UX improvements)
> >> >>
> >> >> This is going to be progressive and iterative. And will require
> >> >>lots of folks  involved.
> >> >>
> >> >> -Sean
> >> >>
> >> >> --
> >> >> Sean Dague
> >> >> Samsung Research America
> >> >> s...@dague.net / sean.da...@samsung.com http://dague.net
> >> >
> >> >___
> >> >OpenStack-dev mailing list
> >> >OpenStack-dev@lists.openstack.org
> >> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >> ___
> >> OpenStack-dev mailing list
> >> OpenStack-dev@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >___
> >OpenStack-dev mailing list
> >OpenStack-dev@lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposed Logging Standards

[Macdonald-Wallace, Matthew]

Hi Cristian,

The functionality already exists within Openstack (certainly it's there in 
Nova) it's just not very well documented (something I keep meaning to do!)

Basically you need to add the following to your nova.conf file:

log_config=/etc/nova/logging.conf

And then create /etc/nova/logging.conf with the configuration you want to use 
based on the Python Logging Module's "ini" configuration format.

Hope that helps,

Matt

> -Original Message-
> From: Sanchez, Cristian A [mailto:cristian.a.sanc...@intel.com]
> Sent: 29 January 2014 17:57
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] Proposed Logging Standards
> 
> Hi Matthew,
> I¹m interested to help in this switch to python logging framework for 
> shipping to
> logstash/etc. Are you working on a blueprint for this?
> Cheers,
> 
> Cristian
> 
> On 27/01/14 11:07, "Macdonald-Wallace, Matthew"
>  wrote:
> 
> >Hi Sean,
> >
> >I'm currently working on moving away from the "built-in" logging to use
> >log_config= and the python logging framework so that we can
> >start shipping to logstash/sentry/.
> >
> >I'd be very interested in getting involved in this, especially from a
> >"why do we have log messages that are split across multiple lines"
> >perspective!
> >
> >Cheers,
> >
> >Matt
> >
> >P.S. FWIW, I'd also welcome details on what the "Audit" level gives us
> >that the others don't... :)
> >
> >> -Original Message-
> >> From: Sean Dague [mailto:s...@dague.net]
> >> Sent: 27 January 2014 13:08
> >> To: OpenStack Development Mailing List
> >> Subject: [openstack-dev] Proposed Logging Standards
> >>
> >> Back at the beginning of the cycle, I pushed for the idea of doing
> >>some log  harmonization, so that the OpenStack logs, across services,
> >>made sense.
> >>I've
> >> pushed a proposed changes to Nova and Keystone over the past couple
> >>of days.
> >>
> >> This is going to be a long process, so right now I want to just focus
> >>on making  INFO level sane, because as someone that spends a lot of
> >>time staring at logs in  test failures, I can tell you it currently
> >>isn't.
> >>
> >> https://wiki.openstack.org/wiki/LoggingStandards is a few things I've
> >>written  down so far, comments welcomed.
> >>
> >> We kind of need to solve this set of recommendations once and for all
> >>up front,  because negotiating each change, with each project, isn't
> >>going to work (e.g -
> >> https://review.openstack.org/#/c/69218/)
> >>
> >> What I'd like to find out now:
> >>
> >> 1) who's interested in this topic?
> >> 2) who's interested in helping flesh out the guidelines for various
> >>log levels?
> >> 3) who's interested in helping get these kinds of patches into
> >>various projects in  OpenStack?
> >> 4) which projects are interested in participating (i.e. interested in
> >>prioritizing  landing these kinds of UX improvements)
> >>
> >> This is going to be progressive and iterative. And will require lots
> >>of folks  involved.
> >>
> >>-Sean
> >>
> >> --
> >> Sean Dague
> >> Samsung Research America
> >> s...@dague.net / sean.da...@samsung.com http://dague.net
> >
> >___
> >OpenStack-dev mailing list
> >OpenStack-dev@lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposed Logging Standards

[Macdonald-Wallace, Matthew]

> > I've also noticed just now that we appear to be "re-inventing" some parts of
> the logging framework (openstack.common.log.WriteableLogger for example
> appears to be a "catchall" when we should just be handing off to the default
> logger and letting the python logging framework decide what to do IMHO).
> 
> WriteableLogger exists for a very specific reason: eventlet. Eventlet assumes 
> a
> file object for logging, not a python logger.
> 
> I've proposed a change for that -
> https://github.com/eventlet/eventlet/pull/75 - but it's not yet upstream.

Thanks for clearing that up, makes a lot more sense now!

So when the change is merged upstream we can get rid of that in our code as 
well?

Matt
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposed Logging Standards

[Macdonald-Wallace, Matthew]

> -Original Message-
> From: Sean Dague [mailto:s...@dague.net]
> Sent: 27 January 2014 14:26
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] Proposed Logging Standards
> 
> On 01/27/2014 09:07 AM, Macdonald-Wallace, Matthew wrote:
> > Hi Sean,
> >
> > I'm currently working on moving away from the "built-in" logging to use
> log_config= and the python logging framework so that we can start
> shipping to logstash/sentry/.
> >
> > I'd be very interested in getting involved in this, especially from a "why 
> > do we
> have log messages that are split across multiple lines" perspective!
> 
> Do we have many that aren't either DEBUG or TRACE? I thought we were pretty
> clean there.
> 
> > Cheers,
> >
> > Matt
> >
> > P.S. FWIW, I'd also welcome details on what the "Audit" level gives us
> > that the others don't... :)
> 
> Well as far as I can tell the AUDIT level was a prior drive by contribution 
> that's
> not being actively maintained. Honestly, I think we should probably rip it 
> out,
> because I don't see any in tree tooling to use it, and it's horribly 
> inconsistent.
> 
>   -Sean

Just as an aside, AUDIT was introduced to the Nova code base as part of 
05ccbb75c45aa3c348162043495e1a3d279e5b06 however a "grep -r AUDIT *" (yes, I 
know, crude but it does work! :P ) across the nova codebase only returns the 
openstack.common.log libraries as having it listed in the code.

I don't know if other projects are making use of it, but if not then I agree 
that it should probably be removed from Oslo

Matt
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposed Logging Standards

[Macdonald-Wallace, Matthew]

> -Original Message-
> From: Sean Dague [mailto:s...@dague.net]
> Sent: 27 January 2014 14:26
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] Proposed Logging Standards
> 
> On 01/27/2014 09:07 AM, Macdonald-Wallace, Matthew wrote:
> > Hi Sean,
> >
> > I'm currently working on moving away from the "built-in" logging to use
> log_config= and the python logging framework so that we can start
> shipping to logstash/sentry/.
> >
> > I'd be very interested in getting involved in this, especially from a "why 
> > do we
> have log messages that are split across multiple lines" perspective!
> 
> Do we have many that aren't either DEBUG or TRACE? I thought we were pretty
> clean there.

True, most (all?!!) are DEBUG/TRACE and mainly from calling out to other 
clients (Neutron/Glance/Cinder/etc), but if you're sending DEBUG somewhere 
useful for future processing then trying to glue the split-lines back together 
again can be "interesting".

At the moment we are assuming that anything that doesn't start with a 
date-stamp is associated with the line above it.  This is probably OK for now, 
however if anything changes in future that negates this rule we won't catch it 
until it's too late!

> >
> > P.S. FWIW, I'd also welcome details on what the "Audit" level gives us
> > that the others don't... :)
> 
> Well as far as I can tell the AUDIT level was a prior drive by contribution 
> that's
> not being actively maintained. Honestly, I think we should probably rip it 
> out,
> because I don't see any in tree tooling to use it, and it's horribly 
> inconsistent.

+1 for this, I wondered if it was something to do with Ceilometer but I'm 
guessing probably not from your comment here.

I've also noticed just now that we appear to be "re-inventing" some parts of 
the logging framework (openstack.common.log.WriteableLogger for example appears 
to be a "catchall" when we should just be handing off to the default logger and 
letting the python logging framework decide what to do IMHO).

Cheers,

Matt
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposed Logging Standards

[Macdonald-Wallace, Matthew]

Hi Sean,

I'm currently working on moving away from the "built-in" logging to use 
log_config= and the python logging framework so that we can start 
shipping to logstash/sentry/.

I'd be very interested in getting involved in this, especially from a "why do 
we have log messages that are split across multiple lines" perspective!

Cheers,

Matt

P.S. FWIW, I'd also welcome details on what the "Audit" level gives us that the 
others don't... :)

> -Original Message-
> From: Sean Dague [mailto:s...@dague.net]
> Sent: 27 January 2014 13:08
> To: OpenStack Development Mailing List
> Subject: [openstack-dev] Proposed Logging Standards
> 
> Back at the beginning of the cycle, I pushed for the idea of doing some log
> harmonization, so that the OpenStack logs, across services, made sense. I've
> pushed a proposed changes to Nova and Keystone over the past couple of days.
> 
> This is going to be a long process, so right now I want to just focus on 
> making
> INFO level sane, because as someone that spends a lot of time staring at logs 
> in
> test failures, I can tell you it currently isn't.
> 
> https://wiki.openstack.org/wiki/LoggingStandards is a few things I've written
> down so far, comments welcomed.
> 
> We kind of need to solve this set of recommendations once and for all up 
> front,
> because negotiating each change, with each project, isn't going to work (e.g -
> https://review.openstack.org/#/c/69218/)
> 
> What I'd like to find out now:
> 
> 1) who's interested in this topic?
> 2) who's interested in helping flesh out the guidelines for various log 
> levels?
> 3) who's interested in helping get these kinds of patches into various 
> projects in
> OpenStack?
> 4) which projects are interested in participating (i.e. interested in 
> prioritizing
> landing these kinds of UX improvements)
> 
> This is going to be progressive and iterative. And will require lots of folks
> involved.
> 
>   -Sean
> 
> --
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Core Review Request 33296

[Macdonald-Wallace, Matthew]

Hi all,

Please can one of the core reviewers take a look at 
https://review.openstack.org/#/c/33296/ and either approve or reject it?

It has had two "+1's" from other developers and has also been passing Jenkins 
and Smokestack since June 28th whilst undergoing numerous rebases.

I realise that the time taken to complete reviews is something that the 
community is trying hard to improve - I'd love to see this change go in as part 
of that effort! :)

Thanks in advance,

Matt

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Making logging more modular

[Macdonald-Wallace, Matthew]

Thanks, that's good news... :)

Matt

From: Doug Hellmann [mailto:doug.hellm...@dreamhost.com]
Sent: 26 June 2013 15:20
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [nova] Making logging more modular



On Wed, Jun 26, 2013 at 9:18 AM, Macdonald-Wallace, Matthew 
mailto:matthew.macdonald-wall...@hp.com>> 
wrote:
Hi Doug,


The basic process as I see it would be that Openstack ships with the two 
logging modules by default (stderr and syslog) however adding a new logging 
mechanism (to sentry, graphite or whatever) would be as easy as dropping a new 
file into nova/openstack/common/log_engines/ and enabling it in nova.conf



If I have understood your comments below, this functionality already exists via 
Oslo?

Right. Oslo lets you provide a logging configuration file, and the Python 
standard library logging module reads that file and uses is to set up different 
log handlers and formatters. http://docs.python.org/2/library/logging.html




What we would like to see is the ability to configure multiple log destinations 
that can be used at the same time.  Often enterprises require the use of Syslog 
for compliance reasons, however it would be nice to be able to meet this 
requirement *and* enable developers to send "copies" of certain log events to 
other destinations if they need to.

Yes, this is definitely possible with the existing library. I think you'll just 
need to write the bits that plug in, and then document how to configure them in 
the logging config file. No changes should be needed in Oslo or Nova.

Doug



Hadoop was just an example that there may be other things in future that we 
require that we haven't thought of yet.  At that point, it would be far easier 
to drop some code into a directory and enable it in nova.conf (or where ever 
appropriate) than rewrite the logging section of the code.



Kind regards,



Matt

From: Doug Hellmann 
[mailto:doug.hellm...@dreamhost.com<mailto:doug.hellm...@dreamhost.com>]
Sent: 26 June 2013 14:04
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [nova] Making logging more modular



On Wed, Jun 26, 2013 at 6:31 AM, Macdonald-Wallace, Matthew 
mailto:matthew.macdonald-wall...@hp.com>> 
wrote:
Hi all,

One of the things we'd like to start thinking about is having the ability to 
log to multiple destinations at the same time.

This would include a move towards making the logging section of Nova more like 
the API and Network sections in that the end user would be able to include 
multiple logging "engines" simply by specifying them in the log file.

Our current use case is that we want to be able to log to SYSLOG at one level 
(DEBUG) and have errors reported to Sentry [0] at another (WARN|ERROR) without 
writing complex code that analyses syslog and then pushes to sentry.

In future, we may want to send all our logs that are WARN to one location 
(Hadoop or similar?), ERR to another location (Monitoring?) and store the last 
"n" hours of debug on local disk or some other crazy configuration - moving to 
modular logging would enable us to do all these things.

I've not created a blueprint yet for this because I thought it best to test the 
water first - is this something that others in the community would be 
interested in seeing/working on?

Python's logging module supports this sort of thing by specifying different log 
handlers for different named loggers and log levels. The Oslo log module 
exposes these configuration settings through the --log-config option, so it is 
possible for a deployer to use Python's built-in file configuration to 
completely override the way any OpenStack component logs.

Do you foresee needing to do any work beyond creating hadoop and/or sentry log 
handlers? Am I misunderstanding what you  intend about including logging 
"engines" in the configuration file?

Doug



Kind regards,

Matt

[0] http://www.getsentry.com/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Making logging more modular

[Macdonald-Wallace, Matthew]

Hi Doug,


The basic process as I see it would be that Openstack ships with the two 
logging modules by default (stderr and syslog) however adding a new logging 
mechanism (to sentry, graphite or whatever) would be as easy as dropping a new 
file into nova/openstack/common/log_engines/ and enabling it in nova.conf



If I have understood your comments below, this functionality already exists via 
Oslo?



What we would like to see is the ability to configure multiple log destinations 
that can be used at the same time.  Often enterprises require the use of Syslog 
for compliance reasons, however it would be nice to be able to meet this 
requirement *and* enable developers to send "copies" of certain log events to 
other destinations if they need to.



Hadoop was just an example that there may be other things in future that we 
require that we haven't thought of yet.  At that point, it would be far easier 
to drop some code into a directory and enable it in nova.conf (or where ever 
appropriate) than rewrite the logging section of the code.



Kind regards,



Matt

From: Doug Hellmann [mailto:doug.hellm...@dreamhost.com]
Sent: 26 June 2013 14:04
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [nova] Making logging more modular



On Wed, Jun 26, 2013 at 6:31 AM, Macdonald-Wallace, Matthew 
mailto:matthew.macdonald-wall...@hp.com>> 
wrote:
Hi all,

One of the things we'd like to start thinking about is having the ability to 
log to multiple destinations at the same time.

This would include a move towards making the logging section of Nova more like 
the API and Network sections in that the end user would be able to include 
multiple logging "engines" simply by specifying them in the log file.

Our current use case is that we want to be able to log to SYSLOG at one level 
(DEBUG) and have errors reported to Sentry [0] at another (WARN|ERROR) without 
writing complex code that analyses syslog and then pushes to sentry.

In future, we may want to send all our logs that are WARN to one location 
(Hadoop or similar?), ERR to another location (Monitoring?) and store the last 
"n" hours of debug on local disk or some other crazy configuration - moving to 
modular logging would enable us to do all these things.

I've not created a blueprint yet for this because I thought it best to test the 
water first - is this something that others in the community would be 
interested in seeing/working on?

Python's logging module supports this sort of thing by specifying different log 
handlers for different named loggers and log levels. The Oslo log module 
exposes these configuration settings through the --log-config option, so it is 
possible for a deployer to use Python's built-in file configuration to 
completely override the way any OpenStack component logs.

Do you foresee needing to do any work beyond creating hadoop and/or sentry log 
handlers? Am I misunderstanding what you  intend about including logging 
"engines" in the configuration file?

Doug



Kind regards,

Matt

[0] http://www.getsentry.com/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Making logging more modular

[Macdonald-Wallace, Matthew]

Hi all,

One of the things we'd like to start thinking about is having the ability to 
log to multiple destinations at the same time.

This would include a move towards making the logging section of Nova more like 
the API and Network sections in that the end user would be able to include 
multiple logging "engines" simply by specifying them in the log file.

Our current use case is that we want to be able to log to SYSLOG at one level 
(DEBUG) and have errors reported to Sentry [0] at another (WARN|ERROR) without 
writing complex code that analyses syslog and then pushes to sentry.

In future, we may want to send all our logs that are WARN to one location 
(Hadoop or similar?), ERR to another location (Monitoring?) and store the last 
"n" hours of debug on local disk or some other crazy configuration - moving to 
modular logging would enable us to do all these things.

I've not created a blueprint yet for this because I thought it best to test the 
water first - is this something that others in the community would be 
interested in seeing/working on?

Kind regards,

Matt

[0] http://www.getsentry.com/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev