Re: [openstack-dev] [TripleO] Spec Minimum Review Proposal
So given the increased complexity of a spec, why not make it 2 specs per week? Matt -Original Message- From: Ben Nemec [mailto:openst...@nemebean.com] Sent: 23 July 2014 14:21 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [TripleO] Spec Minimum Review Proposal For everyone's reference, the tripleo-specs stats can be found here: http://www.nemebean.com/reviewstats/tripleo-specs-30.txt Note that looking at the stats, over 30 days 1 review per week is only 4, which most of our cores are already doing anyway. I'm not sure codifying a requirement to do at least that is going to help much. To move the needle I'm thinking we would need at least 3 - most of our cores aren't meeting that today so it would actually require everyone to do more reviews. Spec reviews are difficult and tend to take a significant amount of time, so that would be a considerable increase in time commitments for cores. I'm not sure how I feel about that, although I'm probably biased because I'm not at 3 per week right now. :-) -Ben On 2014-07-22 15:18, Jay Dobies wrote: At the meetup today, the topic of our spec process came up. The general sentiment is that the process is still young and the hiccups are expected, but we do need to get better about making sure we're staying on top of them. As a first step, it was proposed to add 1 spec review a week to the existing 3 reviews per day requirement for cores. Additionally, we're going to start to capture and review the metrics on spec patches specifically during the weekly meeting. That should help bring to light how long reviews are sitting in the queue without being touched. What are everyone's feelings on adding a 1 spec review per week requirement for cores? Not surprisingly, I'm +1 for it :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] os-refresh-config run frequency
Any chance for getting it streamed or at least IRC'd for those of us who have an interest in this but can't attend? From: Robert Collins [robe...@robertcollins.net] Sent: 20 July 2014 20:30 To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency Sure. Put it in the agenda perhaps Tuesday morning On 20 Jul 2014 12:11, Chris Jones c...@tenshu.netmailto:c...@tenshu.net wrote: Hi I also have some strong concerns about this. Can we get round a table this week and hash it out? Cheers, Chris On 20 Jul 2014, at 14:51, Dan Prince dpri...@redhat.commailto:dpri...@redhat.com wrote: On Thu, 2014-07-17 at 15:54 +0100, Michael Kerrin wrote: On Thursday 26 June 2014 12:20:30 Clint Byrum wrote: Excerpts from Macdonald-Wallace, Matthew's message of 2014-06-26 04:13:31 -0700: Hi all, I've been working more and more with TripleO recently and whilst it does seem to solve a number of problems well, I have found a couple of idiosyncrasies that I feel would be easy to address. My primary concern lies in the fact that os-refresh-config does not run on every boot/reboot of a system. Surely a reboot *is* a configuration change and therefore we should ensure that the box has come up in the expected state with the correct config? This is easily fixed through the addition of an @reboot entry in /etc/crontab to run o-r-c or (less easily) by re-designing o-r-c to run as a service. My secondary concern is that through not running os-refresh-config on a regular basis by default (i.e. every 15 minutes or something in the same style as chef/cfengine/puppet), we leave ourselves exposed to someone trying to make a quick fix to a production node and taking that node offline the next time it reboots because the config was still left as broken owing to a lack of updates to HEAT (I'm thinking a quick change to allow root access via SSH during a major incident that is then left unchanged for months because no-one updated HEAT). There are a number of options to fix this including Modifying os-collect-config to auto-run os-refresh-config on a regular basis or setting os-refresh-config to be its own service running via upstart or similar that triggers every 15 minutes I'm sure there are other solutions to these problems, however I know from experience that claiming this is solved through education of users or (more severely!) via HR is not a sensible approach to take as by the time you realise that your configuration has been changed for the last 24 hours it's often too late! So I see two problems highlighted above. 1) We don't re-assert ephemeral state set by o-r-c scripts. You're right, and we've been talking about it for a while. The right thing to do is have os-collect-config re-run its command on boot. I don't think a cron job is the right way to go, we should just have a file in /var/run that is placed there only on a successful run of the command. If that file does not exist, then we run the command. I've just opened this bug in response: https://bugs.launchpad.net/os-collect-config/+bug/1334804 I have been looking into bug #1334804 and I have a review up to resolve it. I want to highlight something. Currently on a reboot we start all services via upstart (on debian anyways) and there have been quite a lot of issues around this - missing upstart scripts and timing issues. I don't know the issues on fedora. So with a fix to #1334804, on a reboot upstart will start all the services first (with potentially out-of-date configuration), then o-c-c will start o-r-c and will now configure all services and restart them or start them if upstart isn't configured properly. I would like to turn off all boot scripts for services we configure and leave all this to o-r-c. I think this will simplify things and put us in control of starting services. I believe that it will also narrow the gap between fedora and debian or debian and debian so what works on one should work on the other and make it easier for developers. I'm not sold on this approach. At the very least I think we want to make this optional because not all deployments may want to have o-r-c be the central service starting agent. So I'm opposed to this being our (only!) default... The job of o-r-c in this regard is to assert state... which to me means making sure that a service is configured correctly (config files, set to start on boot, and initially started). Requiring o-r-c to be the service starting agent (always) is beyond the scope of the o-r-c tool. If people want to use it in that mode I think having an *option* to do this is fine. I don't think it should be required though. Furthermore I don't think we should get into the habit of writing our elements in such a matter that things no longer start on boot without o-r-c in the mix. I do
Re: [openstack-dev] [TripleO] Proposal to add Jon Paul Sullivan and Alexis Lee to core review team
+1 from me. Matt -Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: 09 July 2014 16:52 To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [TripleO] Proposal to add Jon Paul Sullivan and Alexis Lee to core review team Hello! I've been looking at the statistics, and doing a bit of review of the reviewers, and I think we have an opportunity to expand the core reviewer team in TripleO. We absolutely need the help, and I think these two individuals are well positioned to do that. I would like to draw your attention to this page: http://russellbryant.net/openstack-stats/tripleo-reviewers-90.txt Specifically these two lines: +---+---++ | Reviewer | Reviews -2 -1 +1 +2 +A+/- % | Disagreements* | +---+---++ | jonpaul-sullivan | 1880 43 145 0 077.1% | 28 ( 14.9%) | | lxsli | 1860 23 163 0 087.6% | 27 ( 14.5%) | Note that they are right at the level we expect, 3 per work day. And I've looked through their reviews and code contributions: it is clear that they understand what we're trying to do in TripleO, and how it all works. I am a little dismayed at the slightly high disagreement rate, but looking through the disagreements, most of them were jp and lxsli being more demanding of submitters, so I am less dismayed. So, I propose that we add jonpaul-sullivan and lxsli to the TripleO core reviewer team. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] os-refresh-config run frequency
FWIW, I've just registered https://blueprints.launchpad.net/tripleo/+spec/re-assert-system-state and I'm about to start work on the spec. Matt -Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: 27 June 2014 17:01 To: openstack-dev Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency Excerpts from Macdonald-Wallace, Matthew's message of 2014-06-27 00:14:49 -0700: Hi Clint, -Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: 26 June 2014 20:21 To: openstack-dev Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency So I see two problems highlighted above. 1) We don't re-assert ephemeral state set by o-r-c scripts. You're right, and we've been talking about it for a while. The right thing to do is have os-collect- config re-run its command on boot. I don't think a cron job is the right way to go, we should just have a file in /var/run that is placed there only on a successful run of the command. If that file does not exist, then we run the command. I've just opened this bug in response: https://bugs.launchpad.net/os-collect-config/+bug/1334804 Cool, I'm more than happy for this to be done elsewhere, I'm glad that people are in agreement with me on the concept and that work has already started on this. I'll add some notes to the bug if needed later on today. 2) We don't re-assert any state on a regular basis. So one reason we haven't focused on this, is that we have a stretch goal of running with a readonly root partition. It's gotten lost in a lot of the craziness of just get it working, but with rebuilds blowing away root now, leading to anything not on the state drive (/mnt currently), there's a good chance that this will work relatively well. Now, since people get root, they can always override the readonly root and make changes. golemwe hates thiss!/golem. I'm open to ideas, however, os-refresh-config is definitely not the place to solve this. It is intended as a non-resident command to be called when it is time to assert state. os-collect-config is intended to gather configurations, and expose them to a command that it runs, and thus should be the mechanism by which os- refresh-config is run. I'd like to keep this conversation separate from one in which we discuss more mechanisms to make os-refresh-config robust. There are a bunch of things we can do, but I think we should focus just on how do we re-assert state?. OK, that's fair enough. Because we're able to say right now that it is only for running when config changes, we can wave our hands and say it's ok that we restart everything on every run. As Jan alluded to, that won't work so well if we run it every 20 minutes. Agreed, and chatting with Jan and a couple of others yesterday we came to the conclusion that whatever we do here it will require tweaking of a number of elements to safely restart services. So, I wonder if we can introduce a config version into os-collect-config. Basically os-collect-config would keep a version along with its cache. Whenever a new version is detected, os-collect-config would set a value in the environment that informs the command this is a new version of config. From that, scripts can do things like this: if [ -n $OS_CONFIG_NEW_VERSION ] ; then service X restart else if !service X status ; then service X start fi This would lay the groundwork for future abilities to compare old/new so we can take shortcuts by diffing the two config versions. For instance if we look at old vs. new and we don't see any of the keys we care about changed, we can skip restarting. I like this approach - does this require a new spec? If so, I'll start an etherpad to collect thoughts on it before writing it up for approval. I think this should be a tripleo spec. If you're volunteering write it, hooray \o/. It will require several work items. Off the top of my head: - Add version awareness to os-collect-config - Add version awareness to all os-refresh-config scripts that do disruptive things. - Add periodic command run to os-collect-config Let's call it 're-assert-system-state'. Sound good? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] os-refresh-config run frequency
And the spec is now up at https://review.openstack.org/104524 for everyone to pull apart... ;) Matt -Original Message- From: Macdonald-Wallace, Matthew Sent: 03 July 2014 11:18 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency FWIW, I've just registered https://blueprints.launchpad.net/tripleo/+spec/re- assert-system-state and I'm about to start work on the spec. Matt -Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: 27 June 2014 17:01 To: openstack-dev Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency Excerpts from Macdonald-Wallace, Matthew's message of 2014-06-27 00:14:49 -0700: Hi Clint, -Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: 26 June 2014 20:21 To: openstack-dev Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency So I see two problems highlighted above. 1) We don't re-assert ephemeral state set by o-r-c scripts. You're right, and we've been talking about it for a while. The right thing to do is have os-collect- config re-run its command on boot. I don't think a cron job is the right way to go, we should just have a file in /var/run that is placed there only on a successful run of the command. If that file does not exist, then we run the command. I've just opened this bug in response: https://bugs.launchpad.net/os-collect-config/+bug/1334804 Cool, I'm more than happy for this to be done elsewhere, I'm glad that people are in agreement with me on the concept and that work has already started on this. I'll add some notes to the bug if needed later on today. 2) We don't re-assert any state on a regular basis. So one reason we haven't focused on this, is that we have a stretch goal of running with a readonly root partition. It's gotten lost in a lot of the craziness of just get it working, but with rebuilds blowing away root now, leading to anything not on the state drive (/mnt currently), there's a good chance that this will work relatively well. Now, since people get root, they can always override the readonly root and make changes. golemwe hates thiss!/golem. I'm open to ideas, however, os-refresh-config is definitely not the place to solve this. It is intended as a non-resident command to be called when it is time to assert state. os-collect-config is intended to gather configurations, and expose them to a command that it runs, and thus should be the mechanism by which os- refresh-config is run. I'd like to keep this conversation separate from one in which we discuss more mechanisms to make os-refresh-config robust. There are a bunch of things we can do, but I think we should focus just on how do we re-assert state?. OK, that's fair enough. Because we're able to say right now that it is only for running when config changes, we can wave our hands and say it's ok that we restart everything on every run. As Jan alluded to, that won't work so well if we run it every 20 minutes. Agreed, and chatting with Jan and a couple of others yesterday we came to the conclusion that whatever we do here it will require tweaking of a number of elements to safely restart services. So, I wonder if we can introduce a config version into os-collect-config. Basically os-collect-config would keep a version along with its cache. Whenever a new version is detected, os-collect-config would set a value in the environment that informs the command this is a new version of config. From that, scripts can do things like this: if [ -n $OS_CONFIG_NEW_VERSION ] ; then service X restart else if !service X status ; then service X start fi This would lay the groundwork for future abilities to compare old/new so we can take shortcuts by diffing the two config versions. For instance if we look at old vs. new and we don't see any of the keys we care about changed, we can skip restarting. I like this approach - does this require a new spec? If so, I'll start an etherpad to collect thoughts on it before writing it up for approval. I think this should be a tripleo spec. If you're volunteering write it, hooray \o/. It will require several work items. Off the top of my head: - Add version awareness to os-collect-config - Add version awareness to all os-refresh-config scripts that do disruptive things. - Add periodic command run to os-collect-config Let's call it 're-assert-system-state'. Sound good? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo
Re: [openstack-dev] [TripleO] os-refresh-config run frequency
Hi Clint, -Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: 26 June 2014 20:21 To: openstack-dev Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency So I see two problems highlighted above. 1) We don't re-assert ephemeral state set by o-r-c scripts. You're right, and we've been talking about it for a while. The right thing to do is have os-collect- config re-run its command on boot. I don't think a cron job is the right way to go, we should just have a file in /var/run that is placed there only on a successful run of the command. If that file does not exist, then we run the command. I've just opened this bug in response: https://bugs.launchpad.net/os-collect-config/+bug/1334804 Cool, I'm more than happy for this to be done elsewhere, I'm glad that people are in agreement with me on the concept and that work has already started on this. I'll add some notes to the bug if needed later on today. 2) We don't re-assert any state on a regular basis. So one reason we haven't focused on this, is that we have a stretch goal of running with a readonly root partition. It's gotten lost in a lot of the craziness of just get it working, but with rebuilds blowing away root now, leading to anything not on the state drive (/mnt currently), there's a good chance that this will work relatively well. Now, since people get root, they can always override the readonly root and make changes. golemwe hates thiss!/golem. I'm open to ideas, however, os-refresh-config is definitely not the place to solve this. It is intended as a non-resident command to be called when it is time to assert state. os-collect-config is intended to gather configurations, and expose them to a command that it runs, and thus should be the mechanism by which os- refresh-config is run. I'd like to keep this conversation separate from one in which we discuss more mechanisms to make os-refresh-config robust. There are a bunch of things we can do, but I think we should focus just on how do we re-assert state?. OK, that's fair enough. Because we're able to say right now that it is only for running when config changes, we can wave our hands and say it's ok that we restart everything on every run. As Jan alluded to, that won't work so well if we run it every 20 minutes. Agreed, and chatting with Jan and a couple of others yesterday we came to the conclusion that whatever we do here it will require tweaking of a number of elements to safely restart services. So, I wonder if we can introduce a config version into os-collect-config. Basically os-collect-config would keep a version along with its cache. Whenever a new version is detected, os-collect-config would set a value in the environment that informs the command this is a new version of config. From that, scripts can do things like this: if [ -n $OS_CONFIG_NEW_VERSION ] ; then service X restart else if !service X status ; then service X start fi This would lay the groundwork for future abilities to compare old/new so we can take shortcuts by diffing the two config versions. For instance if we look at old vs. new and we don't see any of the keys we care about changed, we can skip restarting. I like this approach - does this require a new spec? If so, I'll start an etherpad to collect thoughts on it before writing it up for approval. Matt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] os-refresh-config run frequency
Hi Chris, -Original Message- From: Chris Jones [mailto:c...@tenshu.net] Sent: 26 June 2014 20:38 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency Hi Given... On 26 Jun 2014, at 20:20, Clint Byrum cl...@fewbar.com wrote: we should just have a file in /var/run that and... I think we should focus just on how do we re-assert state?. ... for the reboot case, we could have os-collect-config check for the presence of the /var/run file when it starts, if it doesn't find it, unconditionally call o-r-c and then write out the file. Given that we're starting o-c-c on boot, this seems like a fairly simple way to get o-r-c to run on boot (and one that could be trivially disabled by configuration or just dumbly pre-creating the /var/run file). Sounds good. Whenever a new version is detected, os-collect-config would set a value in the environment that informs the command this is a new version of I like the idea of exposing the fact that a new config version has arrived, to o-r-c scripts, but... if !service X status ; then service X start ... I always worry when I see suggestions to have periodic state-assertion tasks take care of starting services that are not running, but in this case I will try to calm my nerves with the knowledge that service(1) is almost certainly talking to a modern init which is perfectly capable of supervising daemons :D Heh, we could always insist on supervisord as a dependency for all elements that run a service... ;) /troll Matt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] os-refresh-config run frequency
-Original Message- From: Chris Jones [mailto:c...@tenshu.net] Sent: 26 June 2014 20:58 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [TripleO] os-refresh-config run frequency Hi On 26 Jun 2014, at 20:55, Clint Byrum cl...@fewbar.com wrote: Let's not focus on the else, but the primary condition which is If we changed, restart. The point is, if we didn't change, try not to be disruptive. Agreed :) +1 Matt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [TripleO] os-refresh-config run frequency
Hi all, I've been working more and more with TripleO recently and whilst it does seem to solve a number of problems well, I have found a couple of idiosyncrasies that I feel would be easy to address. My primary concern lies in the fact that os-refresh-config does not run on every boot/reboot of a system. Surely a reboot *is* a configuration change and therefore we should ensure that the box has come up in the expected state with the correct config? This is easily fixed through the addition of an @reboot entry in /etc/crontab to run o-r-c or (less easily) by re-designing o-r-c to run as a service. My secondary concern is that through not running os-refresh-config on a regular basis by default (i.e. every 15 minutes or something in the same style as chef/cfengine/puppet), we leave ourselves exposed to someone trying to make a quick fix to a production node and taking that node offline the next time it reboots because the config was still left as broken owing to a lack of updates to HEAT (I'm thinking a quick change to allow root access via SSH during a major incident that is then left unchanged for months because no-one updated HEAT). There are a number of options to fix this including Modifying os-collect-config to auto-run os-refresh-config on a regular basis or setting os-refresh-config to be its own service running via upstart or similar that triggers every 15 minutes I'm sure there are other solutions to these problems, however I know from experience that claiming this is solved through education of users or (more severely!) via HR is not a sensible approach to take as by the time you realise that your configuration has been changed for the last 24 hours it's often too late! I'd welcome thoughts on the above, Kind regards, Matt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Reviews - we need your help!
FWIW, I’ve tried to make a useful dashboard for this using Sean Dague’s gerrit-dash-creator [0]. Short URL is http://bit.ly/1l4DLFS long url is: https://review.openstack.org/#/dashboard/?foreach=%28project%3Aopenstack%2Ftripleo-incubator+OR+project%3Aopenstack%2Ftripleo-image-elements+OR+project%3Aopenstack%2Ftripleo-heat-templates+OR+project%3Aopenstack%2Ftripleo-specs+OR+project%3Aopenstack%2Fos-apply-config+OR+project%3Aopenstack%2Fos-collect-config+OR+project%3Aopenstack%2Fos-refresh-config+OR+project%3Aopenstack%2Fdiskimage-builder%29+NOT+owner%3Aself+NOT+label%3AWorkflow%3C%3D-1+label%3AVerified%3E%3D1%252cjenkins+NOT+label%3ACode-Review%3E%3D-2%252cself+branch%3Amaster+status%3Aopentitle=TripleO+ReviewsYour+are+a+reviewer%2C+but+haven%27t+voted+in+the+current+revision=reviewer%3AselfPassed+Jenkins%2C+No+Negative+Feedback=NOT+label%3ACode-Review%3C%3D-1+limit%3A100Changes+with+no+code+review+in+the+last+48hrs=NOT+label%3ACode-Review%3C%3D2+age%3A48hChanges+with+no+code+review+in+the+last+5+days=NOT+label%3ACode-Review%3C%3D2+age%3A5dChanges+with+no+code+review+in+the+last+7+days=NOT+label%3ACode-Review%3C%3D2+age%3A7dSome+adjustment+required+%28-1+only%29=label%3ACode-Review%3D-1+NOT+label%3ACode-Review%3D-2+limit%3A100Dead+Specs+%28-2%29=label%3ACode-Review%3C%3D-2 I’ll add it to my fork and submit a PR if people think it useful. Matt [0] https://github.com/sdague/gerrit-dash-creator From: James Polley [mailto:j...@jamezpolley.com] Sent: 12 June 2014 06:08 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [TripleO] Reviews - we need your help! During yesterday's IRC meeting, we realized that our review stats are starting to slip again. Just after summit, our stats were starting to improve. In the 2014-05-20 meeting, the TripleO Stats since the last revision without -1 or -2[1] looked like this: 1rd quartile wait time: 1 days, 1 hours, 11 minutes Median wait time: 6 days, 9 hours, 49 minutes 3rd quartile wait time: 13 days, 5 hours, 46 minutes As of yesterdays meeting, we have: 1rd quartile wait time: 4 days, 23 hours, 19 minutes Median wait time: 7 days, 22 hours, 8 minutes 3rd quartile wait time: 13 days, 19 hours, 17 minutes This really hurts our velocity, and is especially hard on people making their first commit, as it can take them almost a full work week before they even get their first feedback. To get things moving, we need everyone to make a special effort to do a few reviews every day. It would be most helpful if you can look for older reviews without a -1 or -2 and help those reviews get over the line. If you find reviews that are just waiting for a simple fix - typo or syntax fixes, simple code fixes, or a simple rebase - it would be even more helpful if you could take a few minutes to make those patches, rather than just leaving the review waiting for the attention of the original submitter. Please keep in mind that these stats are based on all of our projects, not just tripleo-incubator. To save you heading to the wiki, here's a handy link that shows you all open code reviews in all our projects: bit.ly/1hQco1Nhttp://bit.ly/1hQco1N If you'd prefer the long version: https://review.openstack.org/#/q/status:open+%28project:openstack/tripleo-incubator+OR+project:openstack/tuskar+OR+project:openstack/tuskar-ui+OR+project:openstack-infra/tripleo-ci+OR+project:openstack/os-apply-config+OR+project:openstack/os-collect-config+OR+project:openstack/os-refresh-config+OR+project:openstack/os-cloud-config+OR+project:openstack/tripleo-image-elements+OR+project:openstack/tripleo-heat-templates+OR+project:openstack/diskimage-builder+OR+project:openstack/python-tuskarclient+OR+project:openstack/tripleo-specs%29,n,zhttps://review.openstack.org/#/q/status:open+%28project:openstack/tripleo-incubator+OR+project:openstack/tuskar+OR+project:openstack/tuskar-ui+OR+project:openstack-infra/tripleo-ci+OR+project:openstack/os-apply-config+OR+project:openstack/os-collect-config+OR+project:openstack/os-refr https://wiki.openstack.org/wiki/TripleO#Bug_Triage has links to reviews for individual projects as well as more information about how we triage bugs. [1] http://www.nemebean.com/reviewstats/tripleo-open.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tripleo] Location of Monitoring Checks
Hi all, I have a patch in flight at the moment [0] to install check_mk server and compliment the already merged installation of check_mk agent [1] so my thoughts are now turning to how we would recommend adding new service checks. The concept behind check_mk makes this really simple to do. You just place a script that outputs status_code check_name performance_data message into the agent's local directory (/usr/lib/check_mk_agent/local on Ubuntu for example) and it will be picked up the next time an inventory of the system is run. There are two ways that we can recommend doing this: 1) We ask users to update the check_mk_agent T-I-E every time they wish to add a new check 2) We ask users to distribute checks from their own T-I-E into the correct location In my opinion, requiring an update to check_mk_agent for every new check is the wrong way of doing this as it means that all systems get all checks regardless of function. Far more preferable would be option 2, however I'm open to other ideas, especially if they mean that organisations using this don't have to go through the review process if they have checks they wish to keep behind the firewall for IP/Licensing reasons. Thanks in advance for any help people can give on this, Matt [0] https://review.openstack.org/#/c/87226/ [1] https://review.openstack.org/#/c/81485/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tripleo] Reviews wanted for new TripleO elements
Apologies to both. I have been asking in IRC but with little luck and had missed that original email. I'll stick to IRC... :) Matt -Original Message- From: Ben Nemec [mailto:openst...@nemebean.com] Sent: 21 April 2014 16:59 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Tripleo] Reviews wanted for new TripleO elements Please don't make review requests on the list. Details here: http://lists.openstack.org/pipermail/openstack-dev/2013- September/015264.html Thanks. -Ben On 04/20/2014 02:44 PM, Macdonald-Wallace, Matthew wrote: Hi all, Can I please ask for some reviews on the following: https://review.openstack.org/#/c/87226/ - Install checkmk_agent https://review.openstack.org/#/c/87223/ - Install icinga cgi interface I already have a souple of +1s and jenkins is happy, all I need is +2 and +A! :) Thanks, Matt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Tripleo] Reviews wanted for new TripleO elements
Hi all, Can I please ask for some reviews on the following: https://review.openstack.org/#/c/87226/ - Install checkmk_agent https://review.openstack.org/#/c/87223/ - Install icinga cgi interface I already have a souple of +1s and jenkins is happy, all I need is +2 and +A! :) Thanks, Matt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposed Logging Standards
-Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 31 January 2014 12:29 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Proposed Logging Standards On 01/30/2014 07:15 PM, John Dickinson wrote: 1) Every log message is one line (ends with \n) and the log fields are space-delineated. eg (`log_line = ' '.join(urllib.quote(x) for x in log_fields_list)`) +1 for this - multiple lines (even in in DEBUG mode!) are a PITA to handle with most log analyser software. 2) The only definition of a log format is the prefix and the message is a set of fields defined by the service actually doing the logging. So, actually, most of my concern at this point wasn't the line format. It was the concern about when projects were calling the loggers, and what kind of information should be logged at each level. Given that most projects are using the oslo defaults today, much of the line format is handled. I think that if you have concerns on that front, it's probably a different conversation with the oslo team. I do agree we should standards a little more on project (i.e. logger name), because in most projects this is just defaulting to module. Which is fine for debug level, but not very user friendly at ops levels. I'd just love to see the ability in the python logger to include the application name, not just the class/module that created the log message (it's in 3.something but I don't think we can justify a switch to Python 3 just based on logging!): datetime LEVEL PID PROGRAM-NAME (i.e. nova-compute) module stuff more_stuff even_more_stuff At the moment, all of the above is possible except for the PROGRAM_NAME part. Is there anything we can do to add this to the context or similar? Matt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposed Logging Standards
-Original Message- From: Ben Nemec [mailto:openst...@nemebean.com] Sent: 31 January 2014 16:01 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] Proposed Logging Standards I'd just love to see the ability in the python logger to include the application name, not just the class/module that created the log message (it's in 3.something but I don't think we can justify a switch to Python 3 just based on logging!): datetime LEVEL PID PROGRAM-NAME (i.e. nova-compute) module stuff more_stuff even_more_stuff At the moment, all of the above is possible except for the PROGRAM_NAME part. Is there anything we can do to add this to the context or similar? Matt Wish granted. :-) YAY! \o/ https://github.com/openstack/oslo- incubator/commit/8c3046b78dca8eae1d911e3421b5938c19f20c37 The plan is to turn that on by default as soon as we can get through the deprecation process for the existing format. Awesome, good to hear. Thanks, Matt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposed Logging Standards
Hi Cristian, The functionality already exists within Openstack (certainly it's there in Nova) it's just not very well documented (something I keep meaning to do!) Basically you need to add the following to your nova.conf file: log_config=/etc/nova/logging.conf And then create /etc/nova/logging.conf with the configuration you want to use based on the Python Logging Module's ini configuration format. Hope that helps, Matt -Original Message- From: Sanchez, Cristian A [mailto:cristian.a.sanc...@intel.com] Sent: 29 January 2014 17:57 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Proposed Logging Standards Hi Matthew, I¹m interested to help in this switch to python logging framework for shipping to logstash/etc. Are you working on a blueprint for this? Cheers, Cristian On 27/01/14 11:07, Macdonald-Wallace, Matthew matthew.macdonald-wall...@hp.com wrote: Hi Sean, I'm currently working on moving away from the built-in logging to use log_config=filename and the python logging framework so that we can start shipping to logstash/sentry/insert other useful tool here. I'd be very interested in getting involved in this, especially from a why do we have log messages that are split across multiple lines perspective! Cheers, Matt P.S. FWIW, I'd also welcome details on what the Audit level gives us that the others don't... :) -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 27 January 2014 13:08 To: OpenStack Development Mailing List Subject: [openstack-dev] Proposed Logging Standards Back at the beginning of the cycle, I pushed for the idea of doing some log harmonization, so that the OpenStack logs, across services, made sense. I've pushed a proposed changes to Nova and Keystone over the past couple of days. This is going to be a long process, so right now I want to just focus on making INFO level sane, because as someone that spends a lot of time staring at logs in test failures, I can tell you it currently isn't. https://wiki.openstack.org/wiki/LoggingStandards is a few things I've written down so far, comments welcomed. We kind of need to solve this set of recommendations once and for all up front, because negotiating each change, with each project, isn't going to work (e.g - https://review.openstack.org/#/c/69218/) What I'd like to find out now: 1) who's interested in this topic? 2) who's interested in helping flesh out the guidelines for various log levels? 3) who's interested in helping get these kinds of patches into various projects in OpenStack? 4) which projects are interested in participating (i.e. interested in prioritizing landing these kinds of UX improvements) This is going to be progressive and iterative. And will require lots of folks involved. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposed Logging Standards
No idea, I only really work on Nova, but as this is in Oslo I expect so! Matt -Original Message- From: Sanchez, Cristian A [mailto:cristian.a.sanc...@intel.com] Sent: 30 January 2014 13:44 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Proposed Logging Standards Hi Matt, What about the rest of the components? Do they also have this capability? Thanks Cristian On 30/01/14 04:59, Macdonald-Wallace, Matthew matthew.macdonald-wall...@hp.com wrote: Hi Cristian, The functionality already exists within Openstack (certainly it's there in Nova) it's just not very well documented (something I keep meaning to do!) Basically you need to add the following to your nova.conf file: log_config=/etc/nova/logging.conf And then create /etc/nova/logging.conf with the configuration you want to use based on the Python Logging Module's ini configuration format. Hope that helps, Matt -Original Message- From: Sanchez, Cristian A [mailto:cristian.a.sanc...@intel.com] Sent: 29 January 2014 17:57 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Proposed Logging Standards Hi Matthew, I¹m interested to help in this switch to python logging framework for shipping to logstash/etc. Are you working on a blueprint for this? Cheers, Cristian On 27/01/14 11:07, Macdonald-Wallace, Matthew matthew.macdonald-wall...@hp.com wrote: Hi Sean, I'm currently working on moving away from the built-in logging to use log_config=filename and the python logging framework so that we can start shipping to logstash/sentry/insert other useful tool here. I'd be very interested in getting involved in this, especially from a why do we have log messages that are split across multiple lines perspective! Cheers, Matt P.S. FWIW, I'd also welcome details on what the Audit level gives us that the others don't... :) -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 27 January 2014 13:08 To: OpenStack Development Mailing List Subject: [openstack-dev] Proposed Logging Standards Back at the beginning of the cycle, I pushed for the idea of doing some log harmonization, so that the OpenStack logs, across services, made sense. I've pushed a proposed changes to Nova and Keystone over the past couple of days. This is going to be a long process, so right now I want to just focus on making INFO level sane, because as someone that spends a lot of time staring at logs in test failures, I can tell you it currently isn't. https://wiki.openstack.org/wiki/LoggingStandards is a few things I've written down so far, comments welcomed. We kind of need to solve this set of recommendations once and for all up front, because negotiating each change, with each project, isn't going to work (e.g - https://review.openstack.org/#/c/69218/) What I'd like to find out now: 1) who's interested in this topic? 2) who's interested in helping flesh out the guidelines for various log levels? 3) who's interested in helping get these kinds of patches into various projects in OpenStack? 4) which projects are interested in participating (i.e. interested in prioritizing landing these kinds of UX improvements) This is going to be progressive and iterative. And will require lots of folks involved. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposed Logging Standards
Hi Sean, I'm currently working on moving away from the built-in logging to use log_config=filename and the python logging framework so that we can start shipping to logstash/sentry/insert other useful tool here. I'd be very interested in getting involved in this, especially from a why do we have log messages that are split across multiple lines perspective! Cheers, Matt P.S. FWIW, I'd also welcome details on what the Audit level gives us that the others don't... :) -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 27 January 2014 13:08 To: OpenStack Development Mailing List Subject: [openstack-dev] Proposed Logging Standards Back at the beginning of the cycle, I pushed for the idea of doing some log harmonization, so that the OpenStack logs, across services, made sense. I've pushed a proposed changes to Nova and Keystone over the past couple of days. This is going to be a long process, so right now I want to just focus on making INFO level sane, because as someone that spends a lot of time staring at logs in test failures, I can tell you it currently isn't. https://wiki.openstack.org/wiki/LoggingStandards is a few things I've written down so far, comments welcomed. We kind of need to solve this set of recommendations once and for all up front, because negotiating each change, with each project, isn't going to work (e.g - https://review.openstack.org/#/c/69218/) What I'd like to find out now: 1) who's interested in this topic? 2) who's interested in helping flesh out the guidelines for various log levels? 3) who's interested in helping get these kinds of patches into various projects in OpenStack? 4) which projects are interested in participating (i.e. interested in prioritizing landing these kinds of UX improvements) This is going to be progressive and iterative. And will require lots of folks involved. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposed Logging Standards
I've also noticed just now that we appear to be re-inventing some parts of the logging framework (openstack.common.log.WriteableLogger for example appears to be a catchall when we should just be handing off to the default logger and letting the python logging framework decide what to do IMHO). WriteableLogger exists for a very specific reason: eventlet. Eventlet assumes a file object for logging, not a python logger. I've proposed a change for that - https://github.com/eventlet/eventlet/pull/75 - but it's not yet upstream. Thanks for clearing that up, makes a lot more sense now! So when the change is merged upstream we can get rid of that in our code as well? Matt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposed Logging Standards
-Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: 27 January 2014 14:26 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Proposed Logging Standards On 01/27/2014 09:07 AM, Macdonald-Wallace, Matthew wrote: Hi Sean, I'm currently working on moving away from the built-in logging to use log_config=filename and the python logging framework so that we can start shipping to logstash/sentry/insert other useful tool here. I'd be very interested in getting involved in this, especially from a why do we have log messages that are split across multiple lines perspective! Do we have many that aren't either DEBUG or TRACE? I thought we were pretty clean there. Cheers, Matt P.S. FWIW, I'd also welcome details on what the Audit level gives us that the others don't... :) Well as far as I can tell the AUDIT level was a prior drive by contribution that's not being actively maintained. Honestly, I think we should probably rip it out, because I don't see any in tree tooling to use it, and it's horribly inconsistent. -Sean Just as an aside, AUDIT was introduced to the Nova code base as part of 05ccbb75c45aa3c348162043495e1a3d279e5b06 however a grep -r AUDIT * (yes, I know, crude but it does work! :P ) across the nova codebase only returns the openstack.common.log libraries as having it listed in the code. I don't know if other projects are making use of it, but if not then I agree that it should probably be removed from Oslo Matt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
