Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack
I'd +1 Clint on this. I believe that the only right way to handle SIGHUP for process running in foreground is to terminate. -- Best regards, Oleg Gelbukh On Fri, Dec 20, 2013 at 10:54 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Sean Dague's message of 2013-12-19 16:33:12 -0800: So a few people had been reporting recently that unstack no longer stops nova processes, which I only got around to looking at today. It turns out the new common.service stack from oslo takes SIGHUP and treats it as a restart. Which isn't wrong, but is new, and is incompatible with screen (the way we use it). Because we use -X stuff, the resulting -X quit sends SIGHUP to the child processes. So the question is, are we definitely in a state now where nova services can and do want to support SIGHUP as restart? If so, is there interest in being able to disable that behavior at start time, so we can continue with a screen based approach as well? If not, we'll need to figure out another way to approach the shutdown in devstack. Which is fine, just work that wasn't expected. Perhaps if the process is running in the foreground, as it does in devstack, it should still terminate on SIGHUP rather than restart. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack
On Fri, Dec 20, 2013 at 1:54 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Sean Dague's message of 2013-12-19 16:33:12 -0800: So a few people had been reporting recently that unstack no longer stops nova processes, which I only got around to looking at today. It turns out the new common.service stack from oslo takes SIGHUP and treats it as a restart. Which isn't wrong, but is new, and is incompatible with screen (the way we use it). Because we use -X stuff, the resulting -X quit sends SIGHUP to the child processes. So the question is, are we definitely in a state now where nova services can and do want to support SIGHUP as restart? If so, is there interest in being able to disable that behavior at start time, so we can continue with a screen based approach as well? If not, we'll need to figure out another way to approach the shutdown in devstack. Which is fine, just work that wasn't expected. Perhaps if the process is running in the foreground, as it does in devstack, it should still terminate on SIGHUP rather than restart. It looks like the changes to ServiceLauncher.wait() that introduced this behavior are related to https://blueprints.launchpad.net/oslo/+spec/cfg-reload-config-files where we wanted a service to be able to re-read its configuration files on a signal. HUP seems appropriate for a production environment, but probably not for development. I opened https://bugs.launchpad.net/oslo/+bug/1263122 to track the fix. Doug ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack
On 12/20/2013 09:55 AM, Doug Hellmann wrote: On Fri, Dec 20, 2013 at 1:54 AM, Clint Byrum cl...@fewbar.com mailto:cl...@fewbar.com wrote: Excerpts from Sean Dague's message of 2013-12-19 16:33:12 -0800: So a few people had been reporting recently that unstack no longer stops nova processes, which I only got around to looking at today. It turns out the new common.service stack from oslo takes SIGHUP and treats it as a restart. Which isn't wrong, but is new, and is incompatible with screen (the way we use it). Because we use -X stuff, the resulting -X quit sends SIGHUP to the child processes. So the question is, are we definitely in a state now where nova services can and do want to support SIGHUP as restart? If so, is there interest in being able to disable that behavior at start time, so we can continue with a screen based approach as well? If not, we'll need to figure out another way to approach the shutdown in devstack. Which is fine, just work that wasn't expected. Perhaps if the process is running in the foreground, as it does in devstack, it should still terminate on SIGHUP rather than restart. It looks like the changes to ServiceLauncher.wait() that introduced this behavior are related to https://blueprints.launchpad.net/oslo/+spec/cfg-reload-config-files where we wanted a service to be able to re-read its configuration files on a signal. HUP seems appropriate for a production environment, but probably not for development. I opened https://bugs.launchpad.net/oslo/+bug/1263122 to track the fix. So as Clint said, SIGHUP is only appropriate to do that *if* the process is daemonized. If it's in the foreground it's not. So that logic needs to be better. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack
On 12/20/2013 09:59 AM, Sean Dague wrote: snip So as Clint said, SIGHUP is only appropriate to do that *if* the process is daemonized. If it's in the foreground it's not. So that logic needs to be better. This is basically a blocker for adding any upgrade testing from something later than havana. Grenade upstream is still functioning because the service code wasn't merged into nova until after havana was cut. However there is a desire to do more interesting upgrade patterns, and without the ability to shutdown nova services on master in the gate, that's going to hit us pretty hard. So I'd like to get this fixed soon. As digging us out of this later is going to be way more expensive. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack
On 12/20/2013 10:56 AM, Sean Dague wrote: On 12/20/2013 09:59 AM, Sean Dague wrote: snip So as Clint said, SIGHUP is only appropriate to do that *if* the process is daemonized. If it's in the foreground it's not. So that logic needs to be better. This is basically a blocker for adding any upgrade testing from something later than havana. Grenade upstream is still functioning because the service code wasn't merged into nova until after havana was cut. However there is a desire to do more interesting upgrade patterns, and without the ability to shutdown nova services on master in the gate, that's going to hit us pretty hard. So I'd like to get this fixed soon. As digging us out of this later is going to be way more expensive. Work around here for review - https://review.openstack.org/#/c/63444/ -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack
On Fri, Dec 20, 2013 at 11:22 AM, Sean Dague s...@dague.net wrote: On 12/20/2013 10:56 AM, Sean Dague wrote: On 12/20/2013 09:59 AM, Sean Dague wrote: snip So as Clint said, SIGHUP is only appropriate to do that *if* the process is daemonized. If it's in the foreground it's not. So that logic needs to be better. This is basically a blocker for adding any upgrade testing from something later than havana. Grenade upstream is still functioning because the service code wasn't merged into nova until after havana was cut. However there is a desire to do more interesting upgrade patterns, and without the ability to shutdown nova services on master in the gate, that's going to hit us pretty hard. So I'd like to get this fixed soon. As digging us out of this later is going to be way more expensive. Work around here for review - https://review.openstack.org/#/c/63444/ That fix has landed in oslo, and I'm working on a patch to copy it into nova now. Doug -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack
Ok so this is interesting. I think this new feature will have caused bugs in anything using this idiom: def foo(bar=CONF): because that is only evaluated at import time - any later reevaluation of the config settings won't propagate into code. (This is why we recently avoided that idiom in Ironic, but part of that review was looking at where else it was used). So - something for reviewers to watch out for - default parameter values from config settings in daemons is no longer 'safe' (it never really was, but now it's clearly always unsafe). -Rob On 21 December 2013 09:09, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Fri, Dec 20, 2013 at 11:22 AM, Sean Dague s...@dague.net wrote: On 12/20/2013 10:56 AM, Sean Dague wrote: On 12/20/2013 09:59 AM, Sean Dague wrote: snip So as Clint said, SIGHUP is only appropriate to do that *if* the process is daemonized. If it's in the foreground it's not. So that logic needs to be better. This is basically a blocker for adding any upgrade testing from something later than havana. Grenade upstream is still functioning because the service code wasn't merged into nova until after havana was cut. However there is a desire to do more interesting upgrade patterns, and without the ability to shutdown nova services on master in the gate, that's going to hit us pretty hard. So I'd like to get this fixed soon. As digging us out of this later is going to be way more expensive. Work around here for review - https://review.openstack.org/#/c/63444/ That fix has landed in oslo, and I'm working on a patch to copy it into nova now. Doug -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
