Re: [openstack-dev] Do all OpenStack daemons support sd_notify?

[Thomas Goirand]

On 12/16/2014 11:59 AM, Alan Pevec wrote:
> There's one issue with Type=notify that Dan Prince discovered and
> where Type=simple works better for his use-case:
> if service startup takes more than DefaultTimeoutStartSec (90s by
> default) and systemd does not get notification from the service, it
> will consider it failed and try to restart it if Restart is defined in
> the service unit file.
> I tried to fix that by disabling timeout (example in Nova package
> https://review.gerrithub.io/13054 ) but then systemctl start blocks
> until notification is received.
> Copying Dan's review comment: "This still isn't quite right. It is
> better in that the systemctl doesn't think the service fails... rather
> it just seems to hang indefinately on 'systemctl start
> openstack-nova-compute', as in I never get my terminal back.
> My test case is:
> 1) Stop Nova compute. 2) Stop RabbitMQ. 3) Try to start Nova compute
> via systemctl.
> Could the RabbitMQ retry loop be preventing the notification?
> "
> 
> Current implementation in oslo service sends notification only just
> before entering the wait loop, because at that point all
> initialization should be done and service ready to serve. Does anyone
> have a suggestion for the better place where to notify service
> readiness?
> Or should just Dan's test-case step 3) be modified as:
> 3) Start Nova compute via systemctl start ... &  (i.e. in the background) ?
> 
> 
> Cheers,
> Alan

What you describe above looks like a defect in the implementation. Of
course, waiting for more than 90s should be considered as failed, and I
wouldn't want in any case to have this timeout increased. Failed
attempts to connect to Rabbit shouldn't, IMO, be the cause for not
sending the notify signal.

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Do all OpenStack daemons support sd_notify?

[Alan Pevec]

Thanks for responding, does this make it longest by time duration
thread ever? :)

2015-10-15 22:44 GMT+02:00 Thomas Goirand :
> On 12/16/2014 11:59 AM, Alan Pevec wrote:
...
>> Current implementation in oslo service sends notification only just
>> before entering the wait loop, because at that point all
>> initialization should be done and service ready to serve. Does anyone
>> have a suggestion for the better place where to notify service
>> readiness?
> What you describe above looks like a defect in the implementation. Of
> course, waiting for more than 90s should be considered as failed, and I
> wouldn't want in any case to have this timeout increased. Failed
> attempts to connect to Rabbit shouldn't, IMO, be the cause for not
> sending the notify signal.

But if service requires message bus for normal operations, it is not
ready to server requests, is it?

Cheers,
Alan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Do all OpenStack daemons support sd_notify?

[Thomas Goirand]

On 12/15/2014 04:21 PM, Ihar Hrachyshka wrote:
> On 14/12/14 09:45, Thomas Goirand wrote:
>> Hi,
> 
>> As I am slowing fixing all systemd issues for the daemons of
>> OpenStack in Debian (and hopefully, have this ready before the
>> freeze of Jessie), I was wondering what kind of Type= directive to
>> put on the systemd .service files. I have noticed that in Fedora,
>> there's Type=notify. So my question is:
> 
>> Do all OpenStack daemons, as a rule, support the DBus sd_notify
>> thing? Should I always use Type=notify for systemd .service files?
>> Can this be called a general rule with no exception?
> 
> (I will talk about neutron only.)
> 
> I guess Type=notify is supposed to be used with daemons that use
> Service class from oslo-incubator that provides systemd notification
> mechanism, or call to systemd.notify_once() otherwise.
> 
> In terms of Neutron, neutron-server process is doing it, metadata
> agent also seems to do it, while OVS agent seems to not. So it really
> should depend on each service and the way it's implemented. You cannot
> just assume that every Neutron service reports back to systemd.
> 
> In terms of Fedora, we have Type=notify for neutron-server service only.
> 
> BTW now that more distributions are interested in shipping unit files
> for services, should we upstream them and ship the same thing in all
> interested distributions?

In Debian & Ubuntu, we use a system which, from a sysv-rc init script,
generates startup files for systemd, sysv-rc, and upstart. So if you
where shipping .service files, I'm not sure I'd use them (though they
may serve as good documentation...).

Cheers,

Thomas Goirand (zigo)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Do all OpenStack daemons support sd_notify?

[Alan Pevec]

2014-12-15 17:00 GMT+01:00 Clint Byrum cl...@fewbar.com:
 Excerpts from Ihar Hrachyshka's message of 2014-12-15 07:21:04 -0800:
 I guess Type=notify is supposed to be used with daemons that use
 Service class from oslo-incubator that provides systemd notification
 mechanism, or call to systemd.notify_once() otherwise.
...
 BTW now that more distributions are interested in shipping unit files
 for services, should we upstream them and ship the same thing in all
 interested distributions?

 Since we can expect the five currently implemented OS's in TripleO to all
 have systemd by default soon (Debian, Fedora, openSUSE, RHEL, Ubuntu),
 it would make a lot of sense for us to make the systemd unit files that
 TripleO generates set Type=notify wherever possible. So hopefully we can
 actually make such a guarantee upstream sometime in the not-so-distant
 future, especially since our CI will run two of the more distinct forks,
 Ubuntu and Fedora.

There's one issue with Type=notify that Dan Prince discovered and
where Type=simple works better for his use-case:
if service startup takes more than DefaultTimeoutStartSec (90s by
default) and systemd does not get notification from the service, it
will consider it failed and try to restart it if Restart is defined in
the service unit file.
I tried to fix that by disabling timeout (example in Nova package
https://review.gerrithub.io/13054 ) but then systemctl start blocks
until notification is received.
Copying Dan's review comment: This still isn't quite right. It is
better in that the systemctl doesn't think the service fails... rather
it just seems to hang indefinately on 'systemctl start
openstack-nova-compute', as in I never get my terminal back.
My test case is:
1) Stop Nova compute. 2) Stop RabbitMQ. 3) Try to start Nova compute
via systemctl.
Could the RabbitMQ retry loop be preventing the notification?


Current implementation in oslo service sends notification only just
before entering the wait loop, because at that point all
initialization should be done and service ready to serve. Does anyone
have a suggestion for the better place where to notify service
readiness?
Or should just Dan's test-case step 3) be modified as:
3) Start Nova compute via systemctl start ...   (i.e. in the background) ?


Cheers,
Alan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Do all OpenStack daemons support sd_notify?

[Ihar Hrachyshka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 14/12/14 09:45, Thomas Goirand wrote:
 Hi,
 
 As I am slowing fixing all systemd issues for the daemons of
 OpenStack in Debian (and hopefully, have this ready before the
 freeze of Jessie), I was wondering what kind of Type= directive to
 put on the systemd .service files. I have noticed that in Fedora,
 there's Type=notify. So my question is:
 
 Do all OpenStack daemons, as a rule, support the DBus sd_notify
 thing? Should I always use Type=notify for systemd .service files?
 Can this be called a general rule with no exception?

(I will talk about neutron only.)

I guess Type=notify is supposed to be used with daemons that use
Service class from oslo-incubator that provides systemd notification
mechanism, or call to systemd.notify_once() otherwise.

In terms of Neutron, neutron-server process is doing it, metadata
agent also seems to do it, while OVS agent seems to not. So it really
should depend on each service and the way it's implemented. You cannot
just assume that every Neutron service reports back to systemd.

In terms of Fedora, we have Type=notify for neutron-server service only.

BTW now that more distributions are interested in shipping unit files
for services, should we upstream them and ship the same thing in all
interested distributions?

 
 Cheers,
 
 Thomas Goirand (zigo)
 
 ___ OpenStack-dev
 mailing list OpenStack-dev@lists.openstack.org 
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iQEcBAEBCgAGBQJUjvxgAAoJEC5aWaUY1u57N1gH/RsYPqmGpyoZ8fe8CwXcnz+R
Rvfo7FHpcEZ9+Idvr9qitoPhKtGjzwgJC27EIQ6NCvgZZT462f+/jYHlxW0dX5Cz
Fm9Zg/Hv50ukDOC1nT3nfDKH8uMwuPMrQsfRuXTGKhwqsfgnFfExozydgVeC2XFw
WB9B3tBblp+7PRzaGyN9Bpe3gQnHUm3lyXaziK+wLbf7NTROzATlVCZ4xpPWu/5C
ArfzwXICp9Dk5Juy75mwYwh37gw26w0VWfvPzn2WjkSVHKymNVn9GRdflVOrV3fq
wnhu08e/wup8XF1/eKkWUJyF+hEsN5E0kO2x5CvavvMS3HSTm3Viuhz5tKC6ZAs=
=WiBi
-END PGP SIGNATURE-

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Do all OpenStack daemons support sd_notify?

[Clint Byrum]

Excerpts from Ihar Hrachyshka's message of 2014-12-15 07:21:04 -0800:
 Hash: SHA512
 
 On 14/12/14 09:45, Thomas Goirand wrote:
  Hi,
  
  As I am slowing fixing all systemd issues for the daemons of
  OpenStack in Debian (and hopefully, have this ready before the
  freeze of Jessie), I was wondering what kind of Type= directive to
  put on the systemd .service files. I have noticed that in Fedora,
  there's Type=notify. So my question is:
  
  Do all OpenStack daemons, as a rule, support the DBus sd_notify
  thing? Should I always use Type=notify for systemd .service files?
  Can this be called a general rule with no exception?
 
 (I will talk about neutron only.)
 
 I guess Type=notify is supposed to be used with daemons that use
 Service class from oslo-incubator that provides systemd notification
 mechanism, or call to systemd.notify_once() otherwise.
 
 In terms of Neutron, neutron-server process is doing it, metadata
 agent also seems to do it, while OVS agent seems to not. So it really
 should depend on each service and the way it's implemented. You cannot
 just assume that every Neutron service reports back to systemd.
 
 In terms of Fedora, we have Type=notify for neutron-server service only.
 
 BTW now that more distributions are interested in shipping unit files
 for services, should we upstream them and ship the same thing in all
 interested distributions?
 

Since we can expect the five currently implemented OS's in TripleO to all
have systemd by default soon (Debian, Fedora, openSUSE, RHEL, Ubuntu),
it would make a lot of sense for us to make the systemd unit files that
TripleO generates set Type=notify wherever possible. So hopefully we can
actually make such a guarantee upstream sometime in the not-so-distant
future, especially since our CI will run two of the more distinct forks,
Ubuntu and Fedora.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev