Re: [request-tracker-maintainers] Bug#595054: request-tracker3.8: Race condition between RT3.8+apache2 and MySQL when booting by insserv
On Tue, Aug 31, 2010 at 11:50:31PM +0200, Salvo Tomaselli wrote: On Tuesday 31 August 2010 22:39:18 Dominic Hargreaves wrote: * Arrange for database servers to start before Apache I wouldn't like that. We would be removing possible parallelism (in a far far away future when we would be using this) at boot. In other words, although the notion of a sequencing is unlikely to result in functional problems (eg the database server failing to start up because it needs to talk to Apache) this doesn't go fo possible quantative problems (slower boot time) which the parallel booting scheme is trying to address. I suppose I hadn't fully considered this aspect. I agree that we should avoid slowing down the boot process as a workaround for a buggy applicaton. * Arrange for RT to be more robust when a connection to the database fails That sounds better. Sounds to me that we assume the SQL server used is going to be on localhost. It's quite a strong assumption and a connection can always fail, so the application shouldn't crash. Yes. I should have made more clear that I wouldn't intend to disregard a fix in the application; it was a question of the possibility of a pragmatic fix in the short term which I wanted to discuss. The original email was talking about a manual restart of apache. I didn't try it myself but... does the apache server terminates because of a failed connection to a database from a web application running on apache itself? _If_ this is the case, this behavior is absolutely wrong and should be considered a bug without trying to look for workarounds. Yes. Perhaps there is also a separate bug to be filed against Apache, although that tight binding between a mod_perl application and Apache itself means that perhaps this would be considered NOTABUG, and something which the application should be relied upon to get right. Thanks for your feedback. Dominic. -- Dominic Hargreaves | http://www.larted.org.uk/~dom/ PGP key 5178E2A5 from the.earth.li (keyserver,web,email) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100901180038.gp26...@urchin.earth.li
Re: [request-tracker-maintainers] Bug#595054: request-tracker3.8: Race condition between RT3.8+apache2 and MySQL when booting by insserv
I appear to have utterly failed to proof-read this before sending. I shouln't attempt mailing list posts at the end of a work day. Apologies. On Wed, Sep 01, 2010 at 07:00:38PM +0100, Dominic Hargreaves wrote: In other words, although the notion of a sequencing is unlikely ^^ sequencing the database server to start before the web server to result in functional problems (eg the database server failing to start up because it needs to talk to Apache) this doesn't go fo possible quantative ^^^ it will cause problems (slower boot time) which the parallel booting scheme is trying to address. I suppose I hadn't fully considered this aspect. I agree that we should avoid slowing down the boot process as a workaround for a buggy applicaton. Yes. I should have made more clear that I wouldn't intend to disregard a don't fix in the application; it was a question of the possibility of a pragmatic ^^ a fix in the short term which I wanted to discuss. -- Dominic Hargreaves | http://www.larted.org.uk/~dom/ PGP key 5178E2A5 from the.earth.li (keyserver,web,email) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100901182740.gr26...@urchin.earth.li
Re: [request-tracker-maintainers] Bug#595054: request-tracker3.8: Race condition between RT3.8+apache2 and MySQL when booting by insserv
On Tue, Aug 31, 2010 at 04:06:28PM -0500, Matt Zagrabelny wrote: [...] Okay, there are two possible resolutions to this problem which spring to mind: * Arrange for database servers to start before Apache I'm not sure if this is feasible, but it seems to me that it's likely to be generally correct, given the layering implicit in database + web applications. * Arrange for RT to be more robust when a connection to the database fails This is certainly something worth exploring, but is likely to be a rather involved fix; not something I relish at this stage of the release cycle (diverging from upstream). Then again, perhaps changing Apache or MySQL (and the other database servers) would be more disruptive :) I've added the CC because I'm not sure where to start discussing whether the Apache/MySQL ordering can be changed. I had a look at Policy (which doesn't seem to mention dependency based booting at all; surely an inconsistency if it is to be enabled for squeeze?) and http://wiki.debian.org/LSBInitScripts which leads me to think that Apache needs something like Should-Start: $database_server, but this is mainly guessing. I would appreciate input from those familiar with this part of the Debian infrastructure. Perhaps for /etc/init.d/mysql-server (or whatever it is called) # X-Start-Before:apache2 # X-Stop-After: apache2 I would think that it would be innocuous if apache2 was not installed. Is there a way to reference the virtual package httpd in the LSB headers? Well, these are valid comments, but I dislike these solutions, and here's why: RT can be served by different HTTP servers (as an example, I've seen a request in BTS for not depending on apache as several people are serving RT using lighttpd) and, in turn, Apache is a very generic HTTP server. Also, RT can use different database backends, and MySQL is nothing special to RT in this regard; again, in turn, MySQL is a generic DBMS. This means neither MySQL nor Apache, as packaged in Debian, should be concerned about each other via their init-scripts, for instance, the idea that apache2 init script should hint insserv that it should be started after DB server is flawed: what about people serving purely static content with their apache instance? What is really needed, is a way to hint insserv (or whatever system will come to replace it, be it upstart or systemd or whatever trendy thing at the time for squeeze+N) about peculiar inter-package dependencies on a package by package basis without hardcoding this to init scripts. I can imagine some sort of /etc/insserv.d, in which a package could place a file with some explicit rule about how it wants to start up. Then the debconf for RT could construct such a file based on what the user is selected as the HTTP server and as the DB server. That way, in a normal setup the insserv would pick whatever boot order in wants; when an RT is happened to be configured to run with apache2 and mysql, its debconf could hint insserv that apache2 now depends on mysql booted first to start successfully. Should the user switch her DB backend to pg, the explicit rule would be replaced; same thing if she opts to change the HTTP server. Unfortunately, I'm a relative old-timer and I'm facing a system like insserv for the first time, so I lack expertise with it. Hence it would be cool if some folks related to insserv could comment on the idea presented above. And yes, the idea that RT could use a DB server running on another host, and hence RT should not prevent apache from starting if that server is not available, is of course valid. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100901222004.gc2...@localhost.localdomain
Re: [request-tracker-maintainers] Bug#595054: request-tracker3.8: Race condition between RT3.8+apache2 and MySQL when booting by insserv
On Thursday 02 September 2010 00:20:04 Konstantin Khomoutov wrote: come to replace it, be it upstart or systemd or whatever trendy thing at the time for squeeze+N) about peculiar inter-package dependencies on a package by package basis without hardcoding this to init scripts. I can imagine some sort of /etc/insserv.d, in which a package could place a file with some explicit rule about how it wants to start up. Then the debconf for RT could construct such a file based on what the user is selected as the HTTP server and as the DB server. That way, in a normal setup the insserv would pick whatever boot order in wants; when an RT is happened to be configured to run with apache2 and mysql, its debconf could hint insserv that apache2 now depends on mysql booted first to start successfully. Should the user switch her DB backend to pg, the explicit rule would be replaced; same thing if she opts to change the HTTP server. This sounds very interesting (but i am afraid i am not related to insserv) but consider this situation: mysql-server wants the email server to be running before running, and the email server is configured to authenticate users on mysql, so it requires mysql to be running when it starts. This issue should be handled as well and it would not be trivial i guess. I am not saying here that the idea in itself isn't good, but that it is very complex to implement and this time could rather be spent fixing the bug rather than building up complex ways to avoid it. And yes, the idea that RT could use a DB server running on another host, and hence RT should not prevent apache from starting if that server is not available, is of course valid. Bye -- Salvo Tomaselli signature.asc Description: This is a digitally signed message part.
Re: [request-tracker-maintainers] Bug#595054: request-tracker3.8: Race condition between RT3.8+apache2 and MySQL when booting by insserv
On Tue, Aug 31, 2010 at 10:17:34PM +0400, Konstantin Khomoutov wrote:
I'm using RT 3.8 with apache2 via mod_perl, MySQL is used as a database
backend. When the server is booted using insserv, apache2 starts long
before MySQL and for some reason some bit of RT tries to access the
MySQL server, times out and fails; this prevents apache2 from starting.
[snip logs]
DBI connect('dbname=rtdb;host=localhost','rtuser',...) failed: Can't connect
to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at
/usr/share/perl5/DBIx/SearchBuilder/Handle.pm line 106
[Tue Aug 31 21:40:18 2010] [error] Connect Failed Can't connect to local
MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)\n at
/usr/share/request-tracker3.8/lib/RT.pm line 217\nCompilation failed in
require at (eval 2) line 1.\n
[Tue Aug 31 21:40:18 2010] [error] Can't load Perl file:
/usr/share/request-tracker3.8/libexec/webmux.pl for server
rt-admin.domain007.com:0, exiting...
If I then start apache2 by hand, it starts OK (it only complains about
the RTAddressRegexp config variable being unset which is harmless I suppose).
[snip details]
I suspect the included /etc/request-tracker3.8/apache2-modperl2.conf
file might be a culprit as otherwise I can't imagine what other code
could call RT internals and make it access the DB backend.
Okay, there are two possible resolutions to this problem which spring
to mind:
* Arrange for database servers to start before Apache
I'm not sure if this is feasible, but it seems to me that it's likely
to be generally correct, given the layering implicit in database + web
applications.
* Arrange for RT to be more robust when a connection to the database fails
This is certainly something worth exploring, but is likely to be a rather
involved fix; not something I relish at this stage of the release cycle
(diverging from upstream). Then again, perhaps changing Apache or MySQL
(and the other database servers) would be more disruptive :)
I've added the CC because I'm not sure where to start discussing
whether the Apache/MySQL ordering can be changed. I had a look at
Policy (which doesn't seem to mention dependency based booting at all;
surely an inconsistency if it is to be enabled for squeeze?) and
http://wiki.debian.org/LSBInitScripts which leads me to think that
Apache needs something like Should-Start: $database_server, but this
is mainly guessing. I would appreciate input from those familiar with
this part of the Debian infrastructure.
Dominic.
--
Dominic Hargreaves | http://www.larted.org.uk/~dom/
PGP key 5178E2A5 from the.earth.li (keyserver,web,email)
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100831203917.gk26...@urchin.earth.li
Re: [request-tracker-maintainers] Bug#595054: request-tracker3.8: Race condition between RT3.8+apache2 and MySQL when booting by insserv
On Tue, Aug 31, 2010 at 3:39 PM, Dominic Hargreaves d...@earth.li wrote:
On Tue, Aug 31, 2010 at 10:17:34PM +0400, Konstantin Khomoutov wrote:
I'm using RT 3.8 with apache2 via mod_perl, MySQL is used as a database
backend. When the server is booted using insserv, apache2 starts long
before MySQL and for some reason some bit of RT tries to access the
MySQL server, times out and fails; this prevents apache2 from starting.
[snip logs]
DBI connect('dbname=rtdb;host=localhost','rtuser',...) failed: Can't connect
to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at
/usr/share/perl5/DBIx/SearchBuilder/Handle.pm line 106
[Tue Aug 31 21:40:18 2010] [error] Connect Failed Can't connect to local
MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)\n at
/usr/share/request-tracker3.8/lib/RT.pm line 217\nCompilation failed in
require at (eval 2) line 1.\n
[Tue Aug 31 21:40:18 2010] [error] Can't load Perl file:
/usr/share/request-tracker3.8/libexec/webmux.pl for server
rt-admin.domain007.com:0, exiting...
If I then start apache2 by hand, it starts OK (it only complains about
the RTAddressRegexp config variable being unset which is harmless I suppose).
[snip details]
I suspect the included /etc/request-tracker3.8/apache2-modperl2.conf
file might be a culprit as otherwise I can't imagine what other code
could call RT internals and make it access the DB backend.
Okay, there are two possible resolutions to this problem which spring
to mind:
* Arrange for database servers to start before Apache
I'm not sure if this is feasible, but it seems to me that it's likely
to be generally correct, given the layering implicit in database + web
applications.
* Arrange for RT to be more robust when a connection to the database fails
This is certainly something worth exploring, but is likely to be a rather
involved fix; not something I relish at this stage of the release cycle
(diverging from upstream). Then again, perhaps changing Apache or MySQL
(and the other database servers) would be more disruptive :)
I've added the CC because I'm not sure where to start discussing
whether the Apache/MySQL ordering can be changed. I had a look at
Policy (which doesn't seem to mention dependency based booting at all;
surely an inconsistency if it is to be enabled for squeeze?) and
http://wiki.debian.org/LSBInitScripts which leads me to think that
Apache needs something like Should-Start: $database_server, but this
is mainly guessing. I would appreciate input from those familiar with
this part of the Debian infrastructure.
Perhaps for /etc/init.d/mysql-server (or whatever it is called)
# X-Start-Before:apache2
# X-Stop-After: apache2
I would think that it would be innocuous if apache2 was not installed.
Is there a way to reference the virtual package httpd in the LSB
headers?
-matt zagrabelny
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/aanlkti=l7r_sjw5k-zagks0juyy3m1p=gc4ledyfm...@mail.gmail.com
Re: [request-tracker-maintainers] Bug#595054: request-tracker3.8: Race condition between RT3.8+apache2 and MySQL when booting by insserv
On Tuesday 31 August 2010 22:39:18 Dominic Hargreaves wrote: * Arrange for database servers to start before Apache I wouldn't like that. We would be removing possible parallelism (in a far far away future when we would be using this) at boot. * Arrange for RT to be more robust when a connection to the database fails That sounds better. Sounds to me that we assume the SQL server used is going to be on localhost. It's quite a strong assumption and a connection can always fail, so the application shouldn't crash. The original email was talking about a manual restart of apache. I didn't try it myself but... does the apache server terminates because of a failed connection to a database from a web application running on apache itself? _If_ this is the case, this behavior is absolutely wrong and should be considered a bug without trying to look for workarounds. Bye -- Salvo Tomaselli signature.asc Description: This is a digitally signed message part.
