Re: [Freeipa-devel] [PATCH] 324 Add fallback for httpd restarts
Martin Kosek wrote: On 10/18/2012 04:36 PM, Rob Crittenden wrote: Martin Kosek wrote: On 10/18/2012 02:47 PM, Rob Crittenden wrote: Martin Kosek wrote: Attaching a script I used to reproduce the issue on machine with sysV (RHEL 6.4 in my case). With the patch applied, httpd restarts correctly fallback-ed. If you think that the wait is not enough, I can add a more complicated procedure, like this one: wait_time = 5 retries = 3 for x in xrange(retries): try: sleep(wait_time) http.stop() sleep(wait_time) http.start() except CalledProcessError: wait_time = wait_time * 2 continue break httpd init script on sysV based platforms cannot guarantee that two consecutive httpd service restarts succeed when run in a small time distance. Add fallback procedure that adds additional waiting time after such failed restart attempt, and then try to stop and start the service again. https://fedorahosted.org/freeipa/ticket/2965 Should we attempt to retrieve a file to ensure that the service is up? The ipa.crt would be a candidate for this. rob Hm, this looks over-engineered from my POV. We already check that ports are open, right? Well, this is assuming that sysV is going to return an error when httpd doesn't start. It should at least call service status to make sure the service is operational. rob Actually, we generally already do that as a part of start() process, you will see it when running my test script: DEBUG: stderr= DEBUG: args=/sbin/service httpd restart DEBUG: stdout=Stopping httpd: [FAILED] Starting httpd:[FAILED] DEBUG: stderr=(98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down Unable to open logs DEBUG: httpd restart failed, try to stopstart again DEBUG: args=/sbin/service httpd stop DEBUG: stdout=Stopping httpd: [ OK ] DEBUG: stderr= DEBUG: args=/sbin/service httpd start DEBUG: stdout=Starting httpd: [ OK ] DEBUG: stderr= DEBUG: args=/sbin/service httpd status DEBUG: stdout=httpd dead but subsys locked But I just realized, that my change in httpinstance.py is redundant, it uses platform service to do the restart which has the fallback already, i.e. modification there is enough. Updated patch attached. Martin ack, pushed to master and ipa-3-0 ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
[Freeipa-devel] [PATCH] 324 Add fallback for httpd restarts
Attaching a script I used to reproduce the issue on machine with sysV (RHEL 6.4 in my case). With the patch applied, httpd restarts correctly fallback-ed. If you think that the wait is not enough, I can add a more complicated procedure, like this one: wait_time = 5 retries = 3 for x in xrange(retries): try: sleep(wait_time) http.stop() sleep(wait_time) http.start() except CalledProcessError: wait_time = wait_time * 2 continue break httpd init script on sysV based platforms cannot guarantee that two consecutive httpd service restarts succeed when run in a small time distance. Add fallback procedure that adds additional waiting time after such failed restart attempt, and then try to stop and start the service again. https://fedorahosted.org/freeipa/ticket/2965 -- Martin Kosek mko...@redhat.com Senior Software Engineer - Identity Management Team Red Hat Inc. From 660717208270cea523cf9d2be655542b38782aad Mon Sep 17 00:00:00 2001 From: Martin Kosek mko...@redhat.com Date: Thu, 18 Oct 2012 14:06:06 +0200 Subject: [PATCH] Add fallback for httpd restarts httpd init script on sysV based platforms cannot guarantee that two consecutive httpd service restarts succeed when run in a small time distance. Add fallback procedure that adds additional waiting time after such failed restart attempt, and then try to stop and start the service again. https://fedorahosted.org/freeipa/ticket/2965 --- ipapython/platform/redhat.py | 17 + ipaserver/install/httpinstance.py | 13 + 2 files changed, 30 insertions(+) diff --git a/ipapython/platform/redhat.py b/ipapython/platform/redhat.py index 3f35cfcc9607bd0c9a05659d936a9903ec198f1b..3551c28410ceeabfc1064ac79e86dc7ee40dd8c3 100644 --- a/ipapython/platform/redhat.py +++ b/ipapython/platform/redhat.py @@ -25,9 +25,11 @@ import stat import sys import socket import stat +import time from ipapython import ipautil from ipapython.platform import base +from ipapython.ipa_log_manager import root_logger from ipalib import api # All what we allow exporting directly from this module @@ -115,6 +117,19 @@ class RedHatSSHService(RedHatService): def get_config_dir(self, instance_name=): return '/etc/ssh' +class RedHatHTTPDService(RedHatService): +def restart(self, instance_name=, capture_output=True, wait=True): +try: +super(RedHatHTTPDService, self).restart(instance_name, capture_output, wait) +except ipautil.CalledProcessError: +# http may have issues with binding to ports, try to fallback +# https://bugzilla.redhat.com/show_bug.cgi?id=845405 +root_logger.debug(%s restart failed, try to stopstart again, self.service_name) +time.sleep(5) +self.stop(instance_name, capture_output) +time.sleep(5) +self.start(instance_name, capture_output, wait) + class RedHatAuthConfig(base.AuthConfig): AuthConfig class implements system-independent interface to configure @@ -145,6 +160,8 @@ class RedHatAuthConfig(base.AuthConfig): def redhat_service(name): if name == 'sshd': return RedHatSSHService(name) +elif name == 'httpd': +return RedHatHTTPDService(name) return RedHatService(name) class RedHatServices(base.KnownServices): diff --git a/ipaserver/install/httpinstance.py b/ipaserver/install/httpinstance.py index 29077367dc17b6df0618c4740fad445edc43e790..6d9c76da9900972aba2ed5c5ae85710566ed3509 100644 --- a/ipaserver/install/httpinstance.py +++ b/ipaserver/install/httpinstance.py @@ -23,6 +23,7 @@ import tempfile from ipapython.ipa_log_manager import * import pwd import shutil +import time import service import certs @@ -393,3 +394,15 @@ class HTTPInstance(service.Service): if not running is None and running: self.start() + +def restart(self, instance_name=, capture_output=True, wait=True): +try: +super(HTTPInstance, self).restart(instance_name, capture_output, wait) +except ipautil.CalledProcessError: +# http may have issues with binding to ports, try to fallback +# https://bugzilla.redhat.com/show_bug.cgi?id=845405 +root_logger.debug(%s restart failed, try to stopstart again, self.service_name) +time.sleep(5) +self.stop(instance_name, capture_output) +time.sleep(5) +self.start(instance_name, capture_output, wait) -- 1.7.11.7 #!/usr/bin/python import ipaserver.install.dsinstance import ipaserver.install.httpinstance from ipapython.ipa_log_manager import * from ipapython import services as ipaservices import time standard_logging_setup(debug=True, console_format='%(levelname)s: %(message)s') print print *** Test HTTPInstance restart print http = ipaserver.install.httpinstance.HTTPInstance() for x in xrange(2): http.restart() # wait before next test time.sleep(3) print
Re: [Freeipa-devel] [PATCH] 324 Add fallback for httpd restarts
Martin Kosek wrote: Attaching a script I used to reproduce the issue on machine with sysV (RHEL 6.4 in my case). With the patch applied, httpd restarts correctly fallback-ed. If you think that the wait is not enough, I can add a more complicated procedure, like this one: wait_time = 5 retries = 3 for x in xrange(retries): try: sleep(wait_time) http.stop() sleep(wait_time) http.start() except CalledProcessError: wait_time = wait_time * 2 continue break httpd init script on sysV based platforms cannot guarantee that two consecutive httpd service restarts succeed when run in a small time distance. Add fallback procedure that adds additional waiting time after such failed restart attempt, and then try to stop and start the service again. https://fedorahosted.org/freeipa/ticket/2965 Should we attempt to retrieve a file to ensure that the service is up? The ipa.crt would be a candidate for this. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] 324 Add fallback for httpd restarts
On 10/18/2012 02:47 PM, Rob Crittenden wrote: Martin Kosek wrote: Attaching a script I used to reproduce the issue on machine with sysV (RHEL 6.4 in my case). With the patch applied, httpd restarts correctly fallback-ed. If you think that the wait is not enough, I can add a more complicated procedure, like this one: wait_time = 5 retries = 3 for x in xrange(retries): try: sleep(wait_time) http.stop() sleep(wait_time) http.start() except CalledProcessError: wait_time = wait_time * 2 continue break httpd init script on sysV based platforms cannot guarantee that two consecutive httpd service restarts succeed when run in a small time distance. Add fallback procedure that adds additional waiting time after such failed restart attempt, and then try to stop and start the service again. https://fedorahosted.org/freeipa/ticket/2965 Should we attempt to retrieve a file to ensure that the service is up? The ipa.crt would be a candidate for this. rob Hm, this looks over-engineered from my POV. We already check that ports are open, right? Martin ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] 324 Add fallback for httpd restarts
Martin Kosek wrote: On 10/18/2012 02:47 PM, Rob Crittenden wrote: Martin Kosek wrote: Attaching a script I used to reproduce the issue on machine with sysV (RHEL 6.4 in my case). With the patch applied, httpd restarts correctly fallback-ed. If you think that the wait is not enough, I can add a more complicated procedure, like this one: wait_time = 5 retries = 3 for x in xrange(retries): try: sleep(wait_time) http.stop() sleep(wait_time) http.start() except CalledProcessError: wait_time = wait_time * 2 continue break httpd init script on sysV based platforms cannot guarantee that two consecutive httpd service restarts succeed when run in a small time distance. Add fallback procedure that adds additional waiting time after such failed restart attempt, and then try to stop and start the service again. https://fedorahosted.org/freeipa/ticket/2965 Should we attempt to retrieve a file to ensure that the service is up? The ipa.crt would be a candidate for this. rob Hm, this looks over-engineered from my POV. We already check that ports are open, right? Well, this is assuming that sysV is going to return an error when httpd doesn't start. It should at least call service status to make sure the service is operational. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] 324 Add fallback for httpd restarts
On 10/18/2012 04:36 PM, Rob Crittenden wrote: Martin Kosek wrote: On 10/18/2012 02:47 PM, Rob Crittenden wrote: Martin Kosek wrote: Attaching a script I used to reproduce the issue on machine with sysV (RHEL 6.4 in my case). With the patch applied, httpd restarts correctly fallback-ed. If you think that the wait is not enough, I can add a more complicated procedure, like this one: wait_time = 5 retries = 3 for x in xrange(retries): try: sleep(wait_time) http.stop() sleep(wait_time) http.start() except CalledProcessError: wait_time = wait_time * 2 continue break httpd init script on sysV based platforms cannot guarantee that two consecutive httpd service restarts succeed when run in a small time distance. Add fallback procedure that adds additional waiting time after such failed restart attempt, and then try to stop and start the service again. https://fedorahosted.org/freeipa/ticket/2965 Should we attempt to retrieve a file to ensure that the service is up? The ipa.crt would be a candidate for this. rob Hm, this looks over-engineered from my POV. We already check that ports are open, right? Well, this is assuming that sysV is going to return an error when httpd doesn't start. It should at least call service status to make sure the service is operational. rob Actually, we generally already do that as a part of start() process, you will see it when running my test script: DEBUG: stderr= DEBUG: args=/sbin/service httpd restart DEBUG: stdout=Stopping httpd: [FAILED] Starting httpd:[FAILED] DEBUG: stderr=(98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down Unable to open logs DEBUG: httpd restart failed, try to stopstart again DEBUG: args=/sbin/service httpd stop DEBUG: stdout=Stopping httpd: [ OK ] DEBUG: stderr= DEBUG: args=/sbin/service httpd start DEBUG: stdout=Starting httpd: [ OK ] DEBUG: stderr= DEBUG: args=/sbin/service httpd status DEBUG: stdout=httpd dead but subsys locked But I just realized, that my change in httpinstance.py is redundant, it uses platform service to do the restart which has the fallback already, i.e. modification there is enough. Updated patch attached. Martin From 4275126bf5fd3422da9fb0536725b809fae9a6bd Mon Sep 17 00:00:00 2001 From: Martin Kosek mko...@redhat.com Date: Thu, 18 Oct 2012 16:50:08 +0200 Subject: [PATCH] Add fallback for httpd restarts on sysV platforms httpd init script on sysV based platforms cannot guarantee that two consecutive httpd service restarts succeed when run in a small time distance. Add fallback procedure that adds additional waiting time after such failed restart attempt, and then try to stop and start the service again. https://fedorahosted.org/freeipa/ticket/2965 --- ipapython/platform/redhat.py | 17 + 1 file changed, 17 insertions(+) diff --git a/ipapython/platform/redhat.py b/ipapython/platform/redhat.py index 3f35cfcc9607bd0c9a05659d936a9903ec198f1b..3551c28410ceeabfc1064ac79e86dc7ee40dd8c3 100644 --- a/ipapython/platform/redhat.py +++ b/ipapython/platform/redhat.py @@ -25,9 +25,11 @@ import stat import sys import socket import stat +import time from ipapython import ipautil from ipapython.platform import base +from ipapython.ipa_log_manager import root_logger from ipalib import api # All what we allow exporting directly from this module @@ -115,6 +117,19 @@ class RedHatSSHService(RedHatService): def get_config_dir(self, instance_name=): return '/etc/ssh' +class RedHatHTTPDService(RedHatService): +def restart(self, instance_name=, capture_output=True, wait=True): +try: +super(RedHatHTTPDService, self).restart(instance_name, capture_output, wait) +except ipautil.CalledProcessError: +# http may have issues with binding to ports, try to fallback +# https://bugzilla.redhat.com/show_bug.cgi?id=845405 +root_logger.debug(%s restart failed, try to stopstart again, self.service_name) +time.sleep(5) +self.stop(instance_name, capture_output) +time.sleep(5) +self.start(instance_name, capture_output, wait) + class RedHatAuthConfig(base.AuthConfig): AuthConfig class implements system-independent interface to configure @@ -145,6 +160,8 @@ class RedHatAuthConfig(base.AuthConfig): def redhat_service(name): if name == 'sshd': return RedHatSSHService(name) +elif name == 'httpd': +return RedHatHTTPDService(name) return RedHatService(name) class RedHatServices(base.KnownServices): -- 1.7.11.7
