Re: automatically restarting dying daemons?
Will Trillich [EMAIL PROTECTED] writes: problem: xinetd, after working just fine and dandy for weeks at a time, gets dozens of unexpected signal (source unknown) and gives up the ghost. questions: 1) what's the best way (e.g. debian way) to monitor active daemons and restart them when necessary? maybe some utility already exists for this? or /proc/something? or `ps ax`? 2) how can i track down the source of the signals specific to this case and make it stop? xinetd chugs along nicely for the most part, and then -- poof! -- it dies a sudden death: Give it to init. This kind of stuff is what init is for. man init. man 5 inittab. Use the respawn action. -- Johan KULLSTAM -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
On Thu, Jul 01, 2004 at 02:31:44PM -0500, Will Trillich wrote: | On Thu, Jul 01 at 05:59PM +0100, Thomas Adam wrote: | --- Will Trillich [EMAIL PROTECTED] wrote: | #!/bin/bash | /etc/init.d/some-daemon-here restart | | Better to use 'invoke-rc.d' here: | | invoke-rc.d script restart [...] | and WHY is invoke-rc.d * better than /etc/init.d/*? and is that | reason still applicable for daemon configs? I didn't know about invoke-rc.d before, but a quick read of the (short) manpage explains : invoke-rc.d is a generic interface to execute System V style init script /etc/init.d/name actions, obeying runlevel constraints as well as any local policies set by the system administrator. [...] invoke-rc.d introduces the concept of a policy layer which is used to verify if an init script should be run or not, or if something else should be done instead. This layer has various uses, the most immedi ate ones being avoiding that package upgrades start daemons outofrun level, and that a package starts or stops daemons while inside a chroot jail. Interesting. | == | | btw -- restartd seems to be just the item we're looking for. | it's a bit terse, too, but it monitors already-running items and | lets you specify a command to resurrect (or terminate) | accordingly. not too advanced, and needs better documentation, | but it works just fine -- at least, for us. Simple and effective where a proper solution is lacking. I learned of it and used it to restart the jabberd aim transport when it would crash. (obviously the correct solution is code fixes to prevent crashes) -D -- Religion that God our Father accepts as pure and faultless is this: to look after orphans and widows in their distress and to keep oneself from being polluted by the world. James 1:27 www: http://dman13.dyndns.org/~dman/jabber: [EMAIL PROTECTED] signature.asc Description: Digital signature
Re: automatically restarting dying daemons?
On Thu, Jul 01, 2004 at 02:46:06PM -0500, Will Trillich wrote: | On Wed, Jun 30 at 06:25PM -0400, Derrick 'dman' Hudson wrote: | On Wed, Jun 30, 2004 at 04:34:06PM -0500, Will Trillich wrote: | | problem: xinetd, after working just fine and dandy for weeks at | | a time, gets dozens of unexpected signal (source unknown) | | and gives up the ghost. | | | | questions: | | 1) what's the best way (e.g. debian way) to monitor active | | daemons and restart them when necessary? maybe some | | utility already exists for this? or /proc/something? | | or `ps ax`? | | restartd. | | aha. not available for woody, but it's available for sarge... | | the logging is odd (stdout, even with /etc/init.d/restartd | restart? Really? I don't know, I haven't actually used it in a while. ISTR it reporting to some log file, though. | is this thing finished?) No software is ever finished ;-). It may not be mature, though. [...] | descriptors 0, 1, 2 are pts/0! for a daemon? | plus, whatever it does restart (according to configs, of course) | winds up with file descriptors open to /var/run/restartd... File a bug report, and even a patch! :-) You're right that a deamon shouldn't have stdin/stdout/stderr open, and the child process ought not to inherit any file descriptors. Fortunately the child, in this case, is probably just a small shell script to restart a daemon, and that daemon will be better behaved and not leave the descriptors open. | thanks for the pointer! You're welcome. -D -- After you install Microsoft Windows XP, you have the option to create user accounts. If you create user accounts, by default, they will have an account type of administrator with no password. -- bugtraq www: http://dman13.dyndns.org/~dman/jabber: [EMAIL PROTECTED] signature.asc Description: Digital signature
Re: automatically restarting dying daemons?
On Wed, Jun 30, 2004 at 10:43:54PM -0500, Will Trillich wrote: | On Wed, Jun 30 at 06:25PM -0400, Derrick 'dman' Hudson wrote: | On Wed, Jun 30, 2004 at 04:34:06PM -0500, Will Trillich wrote: | | problem: xinetd, after working just fine and dandy for weeks at | | a time, gets dozens of unexpected signal (source unknown) | | and gives up the ghost. | | | | questions: | | 1) what's the best way (e.g. debian way) to monitor active | | daemons and restart them when necessary? maybe some | | utility already exists for this? or /proc/something? | | or `ps ax`? | | restartd. | | hmm. this sounds promising... | | $ apt-cache search restart | sort [...] $ apt-cache policy restartd restartd: Installed: 0.1.a-3 Candidate: 0.1.a-3 Version Table: *** 0.1.a-3 0 990 http://http.us.debian.org sarge/main Packages 80 http://http.us.debian.org sid/main Packages 100 /var/lib/dpkg/status Oh, sorry, it's not in woody. I tend to forget those sort of things since I've been using a testing and unstable combination for a long time. -D -- One OS to rule them all, one OS to find them, One OS to bring them all and in the darkness bind them, In the Land of Redmond, where the Shadows lie. www: http://dman13.dyndns.org/~dman/jabber: [EMAIL PROTECTED] signature.asc Description: Digital signature
Re: automatically restarting dying daemons?
On Thu, Jul 01 at 10:55AM -0400, Derrick 'dman' Hudson wrote: $ apt-cache policy restartd restartd: Installed: 0.1.a-3 Candidate: 0.1.a-3 Version Table: *** 0.1.a-3 0 990 http://http.us.debian.org sarge/main Packages 80 http://http.us.debian.org sid/main Packages 100 /var/lib/dpkg/status Oh, sorry, it's not in woody. I tend to forget those sort of things since I've been using a testing and unstable combination for a long time. never knew about the policy thing before. cool! :) we're about to instantiate a new server anyhow, and it'll be running sarge, so this may be the way to go. thanks for the pointers... -- I use Debian/GNU Linux version 3.0; Linux boss 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST 2002 i586 unknown DEBIAN NEWBIE TIP #110 from Dimitri Maziuk [EMAIL PROTECTED] : Here's how to TUNNEL SECURE X11 CONNECTIONS THROUGH SSH: on the client, do this: local-client# export DISPLAY=:0.0 local-client# ssh -X server then once you're logged in at the server, do: remote-server# netscape The environment created at the server will include the DISPLAY variable, so netscape (or whatever) will dialogue with the client machine. (See man ssh for more.) Also see http://newbieDoc.sourceForge.net/ ... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
On Wed, Jun 30 at 11:20PM -0500, Jacob S. wrote: On Wed, 30 Jun 2004 22:43:54 -0500 Will Trillich [EMAIL PROTECTED] wrote: at http://backports.org, i search for restartd and get Sorry, no packages found. At the risk of starting a flamewar about whether djb's tools are a good way to do things or not... :-) Have you looked at daemontools? apt-cache show daemontools-installer, apt-cache show svtools. The sole purpose of daemontools is to make sure a program keeps running properly. I have successfully used it on occasion when I was working with a program that was known for crashing, but didn't consider the program important enough to make it run dependably. daemontools worked great. the documentation is a bit terse at http://cr.yp.to/ -- can the run script be #!/bin/bash /etc/init.d/some-daemon-here restart which is effectively a start-some-thing and quick return... or does it need to be the non-returning call to the daemon itself, so that the daemon is a child of the supervise process? if so, ick. -- I use Debian/GNU Linux version 3.0; Linux boss 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST 2002 i586 unknown DEBIAN NEWBIE TIP #44 from Will Trillich [EMAIL PROTECTED] : Ever think you're reading OUTDATED DOCUMENTATION? Check the last-revised-date: if it's more than a few years ago, then there's probably something more recent out there. It may be under a whole different name, so it'll take perseverance and determination on your part. Be alert -- you'll find it! Also see http://newbieDoc.sourceForge.net/ ... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
On Thu, Jul 01 at 08:58AM +0800, John Summerfield wrote:
Mark Ferlatte wrote:
Will Trillich said on Wed, Jun 30, 2004 at 04:34:06PM -0500:
questions:
1) what's the best way (e.g. debian way) to monitor active
daemons and restart them when necessary? maybe some
utility already exists for this? or /proc/something?
or `ps ax`?
monit can do this.
As can webmin.
webmin would be promising if we already had all that overhead
running. (plus i've seen it have problems -- for ecsample,
apache-lib.pl is missing in a few installations i've seen, and
it borks the html interface when a piece like that is absent.)
plus, the webmin code itself looks like it's right out of the
seventies. hoo boy!
--
I use Debian/GNU Linux version 3.0;
Linux boss 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST 2002 i586 unknown
DEBIAN NEWBIE TIP #69 from Will Trillich [EMAIL PROTECTED]
:
Preparing to UPGRADE POSTGRESQL? If you have a second machine on
your network that you can tinker with, do your upgrade there,
first: once tested, you can just have your current applications
link to the remote database through the network:
psql -h 192.168.2.17 myDB
or in perl,
$dbh = DBI-connect('dbi:Pg:dbname=myDB;host=192.168.2.17');
(You may need to tweak your 'host-based access' settings in
/etc/postgresql/pg_hba.conf, first.) Once you're satisfied that
all is well, upgrade your main server. No down time!
See man psql and man DBD::Pg for details.
Also see http://newbieDoc.sourceForge.net/ ...
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
--- Will Trillich [EMAIL PROTECTED] wrote: #!/bin/bash /etc/init.d/some-daemon-here restart Better to use 'invoke-rc.d' here: invoke-rc.d script restart which is effectively a start-some-thing and quick return... or does it need to be the non-returning call to the daemon itself, so that the daemon is a child of the supervise process? if so, ick. I'm not quite sure what you're asking here. There's no need to background that (rather pointless) wrapper script, since the daemon will fork if its own accord. -- Thomas Adam = The Linux Weekend Mechanic -- http://linuxgazette.net TAG Editor -- http://linuxgazette.net shrug We'll just save up your sins, Thomas, and punish you for all of them at once when you get better. The experience will probably kill you. :) -- Benjamin A. Okopnik (Linux Gazette Technical Editor) ___ALL-NEW Yahoo! Messenger - so many all-new ways to express yourself http://uk.messenger.yahoo.com -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
Will Trillich said on Wed, Jun 30, 2004 at 10:38:51PM -0500: On Wed, Jun 30 at 03:43PM -0700, Mark Ferlatte wrote: Will Trillich said on Wed, Jun 30, 2004 at 04:34:06PM -0500: questions: 1) what's the best way (e.g. debian way) to monitor active daemons and restart them when necessary? maybe some utility already exists for this? or /proc/something? or `ps ax`? monit can do this. so let's go find monit... Suck. As with restartd, monit is also only in testing/unstable. But the package name is `monit'... mon is something else. M pgplCo4leMuEO.pgp Description: PGP signature
Re: automatically restarting dying daemons?
On Thu, Jul 01 at 05:59PM +0100, Thomas Adam wrote: --- Will Trillich [EMAIL PROTECTED] wrote: #!/bin/bash /etc/init.d/some-daemon-here restart Better to use 'invoke-rc.d' here: invoke-rc.d script restart for daemon-config-file-settings, i'm more comfortable specifying the whole path. less chance of intervention or misdirection based on $PATH mungings... is invoke-rc.d similar to the service function on other distros? (sarge already has a _service for bash to facilitate command-line word completion... and i understand that the service function/script/alias is on its way.) and WHY is invoke-rc.d * better than /etc/init.d/*? and is that reason still applicable for daemon configs? (don't you stick with full path names in your cron jobs?) which is effectively a start-some-thing and quick return... or does it need to be the non-returning call to the daemon itself, so that the daemon is a child of the supervise process? if so, ick. I'm not quite sure what you're asking here. There's no need to background that (rather pointless) wrapper script, since the daemon will fork if its own accord. right. here's my question: according to the not-so-very-loquacious documentation at http://cr.yp.to/ -- supervise switches to the directory named s and starts ./run. It restarts ./run if ./run exits. It pauses for a second after starting ./run, so that it does not loop too quickly if ./run exits immediately. looks like it's expecting ./run to be a long-duration process, such as a database server itself, for example, that service will then restart when the thing dies. it's NOT expecting (if i get the gist of the english there) that this will be a quickie start a daemon somewhere else because 1) there seems to be no facility for checking for a daemon process, only the ./run process (i.e. child processes of supervise) 2) when ./run exits immediately, supervise will re-launch it (after waiting for a whole second) so our daemons will get re-initialized sixty times a minute, and for something like a database server, that's very much badness unless i misunderstand, this seems to be a run-and-monitor home grown programs and scripts, do your system daemon resurrection elsewhere... no? == btw -- restartd seems to be just the item we're looking for. it's a bit terse, too, but it monitors already-running items and lets you specify a command to resurrect (or terminate) accordingly. not too advanced, and needs better documentation, but it works just fine -- at least, for us. -- I use Debian/GNU Linux version 3.0; Linux boss 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST 2002 i586 unknown DEBIAN NEWBIE TIP #14 from Will Trillich [EMAIL PROTECTED] : What's a RUNLEVEL? It's simply a big-time setting group; runlevel 2 might have a full-blown web server plus X running, and runlevel 3 might be ssh-only, for secure logins. Check /etc/inittab (and /etc/rcRUNLEVEL.d/*) for details on how yours are set up. And try man runlevel. Also see http://newbieDoc.sourceForge.net/ ... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
On Wed, Jun 30 at 06:25PM -0400, Derrick 'dman' Hudson wrote: On Wed, Jun 30, 2004 at 04:34:06PM -0500, Will Trillich wrote: | problem: xinetd, after working just fine and dandy for weeks at | a time, gets dozens of unexpected signal (source unknown) | and gives up the ghost. | | questions: | 1) what's the best way (e.g. debian way) to monitor active |daemons and restart them when necessary? maybe some |utility already exists for this? or /proc/something? |or `ps ax`? restartd. aha. not available for woody, but it's available for sarge... the logging is odd (stdout, even with /etc/init.d/restartd restart? is this thing finished?) but it does what we want it to do. # lsof | grep ^restartd restartd 12689root cwdDIR3,1 4096 15387 /etc/webmin restartd 12689root rtdDIR3,1 4096 2 / restartd 12689root txtREG3,6 9008 65286 /usr/sbin/restartd restartd 12689root memREG3,190152 46147 /lib/ld-2.3.2.so restartd 12689root memREG3,1 1243856 46185 /lib/libc-2.3.2.so restartd 12689root0u CHR 136,0 2 /dev/pts/0 restartd 12689root1u CHR 136,0 2 /dev/pts/0 restartd 12689root2u CHR 136,0 2 /dev/pts/0 restartd 12689root3u unix 0xcb92b330 199392 socket descriptors 0, 1, 2 are pts/0! for a daemon? # lsof | grep pts/ bash 5179will0u CHR 136,0 2 /dev/pts/0 bash 5179will1u CHR 136,0 2 /dev/pts/0 bash 5179will2u CHR 136,0 2 /dev/pts/0 bash 5179will 255u CHR 136,0 2 /dev/pts/0 bash 5310root0u CHR 136,0 2 /dev/pts/0 bash 5310root1u CHR 136,0 2 /dev/pts/0 bash 5310root2u CHR 136,0 2 /dev/pts/0 bash 5310root 255u CHR 136,0 2 /dev/pts/0 restartd 12689root0u CHR 136,0 2 /dev/pts/0 restartd 12689root1u CHR 136,0 2 /dev/pts/0 restartd 12689root2u CHR 136,0 2 /dev/pts/0 lsof 13050root0u CHR 136,0 2 /dev/pts/0 lsof 13050root2u CHR 136,0 2 /dev/pts/0 grep 13051root1u CHR 136,0 2 /dev/pts/0 grep 13051root2u CHR 136,0 2 /dev/pts/0 lsof and grep are running at my terminal; so is bash... but restartd was launched as a daemon! eesh. [hmm -- must look into the /etc/init.d/restartd script to make sure it's properly launched there hmm] plus, whatever it does restart (according to configs, of course) winds up with file descriptors open to /var/run/restartd... # lsof | grep run/restartd spamd 12752root4w REG3,5 382294355 /var/run/restartd postmaste 12901postgres4w REG3,5 382294355 /var/run/restartd postmaste 12906postgres4w REG3,5 382294355 /var/run/restartd postmaste 12908postgres4w REG3,5 382294355 /var/run/restartd named 13013bind4w REG3,5 382294355 /var/run/restartd named 13014bind4w REG3,5 382294355 /var/run/restartd named 13015bind4w REG3,5 382294355 /var/run/restartd named 13016bind4w REG3,5 382294355 /var/run/restartd named 13017bind4w REG3,5 382294355 /var/run/restartd weird. but operational. thanks for the pointer! -- I use Debian/GNU Linux version 3.0; Linux boss 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST 2002 i586 unknown DEBIAN NEWBIE TIP #62 from Will Trillich [EMAIL PROTECTED] : Wouldn't it be nice to SEE YOUR TABS WHILE YOU EDIT? With Vim, you can do this with :set listchars=tab:+-,trail:$ :set list and format them via :highlight NonText (See :help listchars and :help highlight for more info.) Put them in your ~/.vimrc if you decide you like that setup. Also see http://newbieDoc.sourceForge.net/ ... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
--- Will Trillich [EMAIL PROTECTED] wrote: for daemon-config-file-settings, i'm more comfortable specifying the whole path. less chance of intervention or misdirection based on $PATH mungings... /etc/init.d is not in $PATH, and as such scripts are run as root anyway, invoke-rc.d is perfect still. is invoke-rc.d similar to the service function on other distros? (sarge already has a _service for bash to facilitate command-line word completion... and i understand that the service function/script/alias is on its way.) It's a little similar, yes. 1) there seems to be no facility for checking for a daemon process, only the ./run process (i.e. child processes of supervise) If that is the case, then the script (and overall design) is very broken, and I would avoid it. unless i misunderstand, this seems to be a run-and-monitor home grown programs and scripts, do your system daemon resurrection elsewhere... no? monit has already been suggested along with 'daemontools'. btw -- restartd seems to be just the item we're looking for. it's a bit terse, too, but it monitors already-running items and lets you specify a command to resurrect (or terminate) accordingly. not too advanced, and needs better documentation, but it works just fine -- at least, for us. -- Thomas Adam = The Linux Weekend Mechanic -- http://linuxgazette.net TAG Editor -- http://linuxgazette.net shrug We'll just save up your sins, Thomas, and punish you for all of them at once when you get better. The experience will probably kill you. :) -- Benjamin A. Okopnik (Linux Gazette Technical Editor) ___ALL-NEW Yahoo! Messenger - so many all-new ways to express yourself http://uk.messenger.yahoo.com -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
On Thu, Jul 01 at 09:31PM +0100, Thomas Adam wrote: --- Will Trillich [EMAIL PROTECTED] wrote: for daemon-config-file-settings, i'm more comfortable specifying the whole path. less chance of intervention or misdirection based on $PATH mungings... /etc/init.d is not in $PATH, and as such scripts are run as root anyway, invoke-rc.d is perfect still. you probably already know this, being the expert du jour and everything, but just in case: when a command specification starts with a slash, it's an absolute reference, no uncertainties about it; if it does NOT start with a slash, then your environmental variable $PATH is called upon to supply likely directories to scan, looking for an executable by the name you specified. (if you have perl, say, in both /usr/local/bin and /usr/bin you'll never see the one in /usr/bin.) the trouble, of course, is that script kiddies can find ways to munge your $PATH; you might think you're asking for ls or more in their standard /bin/* location, but in fact the black-hats can prepend your $PATH with a directory of their own making, which runs a fake ls or more which can do worse things yet. so in system scripts, it's good to 1) specify exact, full, absolute paths, and 2) set your own $PATH variable, and finally 3) specify exact, full, absolute paths anyhow. using invoke-rc.d in a system/daemon script is as dangerous as using ls or more -- without a full path. and invoking it with a full path is better than calling /etc/init.d/* scripts directly ... in what way? is invoke-rc.d similar to the service function on other distros? (sarge already has a _service for bash to facilitate command-line word completion... and i understand that the service function/script/alias is on its way.) It's a little similar, yes. a little? how little? is this invoke-rc.d something we understand, or something we repeat? [re: daemontools--] 1) there seems to be no facility for checking for a daemon process, only the ./run process (i.e. child processes of supervise) If that is the case, then the script (and overall design) is very broken, and I would avoid it. i would, too. and since it does seem the case, i do. unless i misunderstand, this seems to be a run-and-monitor home grown programs and scripts, do your system daemon resurrection elsewhere... no? monit has already been suggested along with 'daemontools'. and daemontools was actually the subject under discussion. unless monit has something ingenious to offer, we'll be staying with restartd for now. -- I use Debian/GNU Linux version 3.0; Linux boss 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST 2002 i586 unknown DEBIAN NEWBIE TIP #87 from Joost Kooij [EMAIL PROTECTED] : Did you CHMOD -R / and destroy your permissions? Bad dog! If you have access to a newly-installed Debian machine, run this script there, and copy the resulting script to the box with the bad permissions; run it, and all should be back to normal: find / -regex '/\(mnt\|proc\|tmp\)/.*' -prune -or \ -not -type l -not -type s \ -printf 'chown %u.%g %p\nchmod %m %p\n' \ fixperms.sh Also see http://newbieDoc.sourceForge.net/ ... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
--- Will Trillich [EMAIL PROTECTED] wrote: starts with a slash, it's an absolute reference, no uncertainties about it; if it does NOT start with a slash, then your environmental variable $PATH is called upon to supply likely directories to scan, looking for an executable by the name you specified. (if you have perl, say, in both /usr/local/bin and /usr/bin you'll never see the one in /usr/bin.) Yes, and? :) This has nothing to do with script files. invoke-rc.d already knows its starting place -- it has been told it already. $PATH is only used for binary locations. [..snip..] -- Thomas Adam = The Linux Weekend Mechanic -- http://linuxgazette.net TAG Editor -- http://linuxgazette.net shrug We'll just save up your sins, Thomas, and punish you for all of them at once when you get better. The experience will probably kill you. :) -- Benjamin A. Okopnik (Linux Gazette Technical Editor) ___ALL-NEW Yahoo! Messenger - so many all-new ways to express yourself http://uk.messenger.yahoo.com -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
automatically restarting dying daemons?
problem: xinetd, after working just fine and dandy for weeks at
a time, gets dozens of unexpected signal (source unknown)
and gives up the ghost.
questions:
1) what's the best way (e.g. debian way) to monitor active
daemons and restart them when necessary? maybe some
utility already exists for this? or /proc/something?
or `ps ax`?
2) how can i track down the source of the signals specific
to this case and make it stop?
xinetd chugs along nicely for the most part, and then -- poof!
-- it dies a sudden death:
[EMAIL PROTECTED] cd /var/log
[EMAIL PROTECTED] grep xinetd daemon.log
Jun 30 13:39:13 boss xinetd[21953]: {general_handler} (21953) Unexpected signal: 11
(Segmentation fault)
Jun 30 13:39:13 boss xinetd[21953]: {bad_signal} Received 10 signals in 1 seconds.
Exiting...
Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11
(Segmentation fault)
Jun 30 13:39:13 boss xinetd[4873]: Resetting...
Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11
(Segmentation fault)
Jun 30 13:39:13 boss xinetd[4873]: Resetting...
Jun 30 13:39:25 boss xinetd[21964]: {general_handler} (21964) Unexpected signal: 11
(Segmentation fault)
Jun 30 13:39:25 boss xinetd[21964]: {bad_signal} Received 10 signals in 1 seconds.
Exiting...
Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11
(Segmentation fault)
Jun 30 13:39:25 boss xinetd[4873]: Resetting...
Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11
(Segmentation fault)
Jun 30 13:39:25 boss xinetd[4873]: Resetting...
Jun 30 13:41:27 boss xinetd[21990]: {general_handler} (21990) Unexpected signal: 11
(Segmentation fault)
Jun 30 13:41:27 boss xinetd[21990]: {bad_signal} Received 10 signals in 1 seconds.
Exiting...
Jun 30 13:41:27 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11
(Segmentation fault)
Jun 30 13:41:27 boss xinetd[4873]: Resetting...
Jun 30 13:41:27 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11
(Segmentation fault)
Jun 30 13:41:27 boss xinetd[4873]: {general_handler} (4873) Unexpected signal: 11
(Segmentation fault)
Jun 30 13:41:27 boss xinetd[4873]: {bad_signal} Received 50 bad signals. Exiting...
Jun 30 15:51:56 boss xinetd[23062]: xinetd Version 2.3.4 started with libwrap loadavg
options compiled in.
Jun 30 15:51:56 boss xinetd[23062]: Started working: 8 available services
so that shows when xinetd died. here's all the activity at
that time plus-or-minus a second or two (13:39:12 - :14 and
13:39:24 - :28):
[EMAIL PROTECTED] find . -type f \
| xargs grep 'Jun 30 13:39' \
| egrep ':1[234] |:2[45678] '
./syslog:Jun 30 13:39:13 boss xinetd[21953]: {general_handler} (21953) Unexpected
signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:13 boss last message repeated 9 times
./syslog:Jun 30 13:39:13 boss xinetd[21953]: {bad_signal} Received 10 signals in 1
seconds. Exiting...
./syslog:Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected
signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:13 boss last message repeated 9 times
./syslog:Jun 30 13:39:13 boss xinetd[4873]: Resetting...
./syslog:Jun 30 13:39:13 boss postgres[21954]: [1] DEBUG: pq_recvbuf: recv() failed:
Connection reset by peer
./syslog:Jun 30 13:39:13 boss postgres[21954]: [2] DEBUG: incomplete startup packet
./syslog:Jun 30 13:39:13 boss -f[21958]: (v4.0.4) Unable to get canonical name of
client 127.12.21.44: Unknown host (1) [pop_init.c:1075]
./syslog:Jun 30 13:39:13 boss xinetd[4873]: {general_handler} (4873) Unexpected
signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:13 boss last message repeated 8 times
./syslog:Jun 30 13:39:13 boss xinetd[4873]: Resetting...
./syslog:Jun 30 13:39:13 boss -f[21958]: (null) at 127.12.21.44 (127.12.21.44): -ERR
POP EOF or I/O Error [popper.c:820]
./syslog:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at
127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./syslog:Jun 30 13:39:13 boss -f[21958]: I/O error flushing output to client at
127.12.21.44 [127.12.21.44]: Operation not permitted (1) [pop_send.c:689]
./syslog:Jun 30 13:39:25 boss postgres[21962]: [1] DEBUG: pq_recvbuf: recv() failed:
Connection reset by peer
./syslog:Jun 30 13:39:25 boss postgres[21962]: [2] DEBUG: incomplete startup packet
./syslog:Jun 30 13:39:25 boss xinetd[21964]: {general_handler} (21964) Unexpected
signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:25 boss last message repeated 9 times
./syslog:Jun 30 13:39:25 boss xinetd[21964]: {bad_signal} Received 10 signals in 1
seconds. Exiting...
./syslog:Jun 30 13:39:25 boss spamd[371]: connection from localhost [127.0.0.1] at
port 4706
./syslog:Jun 30 13:39:25 boss xinetd[4873]: {general_handler} (4873) Unexpected
signal: 11 (Segmentation fault)
./syslog:Jun 30 13:39:25 boss last message
Re: automatically restarting dying daemons?
On Wed, Jun 30, 2004 at 04:34:06PM -0500, Will Trillich wrote: | problem: xinetd, after working just fine and dandy for weeks at | a time, gets dozens of unexpected signal (source unknown) | and gives up the ghost. | | questions: | 1) what's the best way (e.g. debian way) to monitor active | daemons and restart them when necessary? maybe some | utility already exists for this? or /proc/something? | or `ps ax`? restartd. | 2) how can i track down the source of the signals specific | to this case and make it stop? I don't know. Maybe start with the source code and see what sort of conditions cause that error to be reported? The only thing that jumps out at me from the logs I snipped is a lot of daemons are reporting unusual problems at that time. Maybe some hardware issue on the machine (failing memory or data corruption on some bus)? -D -- The heart is deceitful above all things and beyond cure. Who can understand it? I the Lord search the heart and examine the mind, to reward a man according to his conduct, according to what his deeds deserve. Jeremiah 17:9-10 www: http://dman13.dyndns.org/~dman/jabber: [EMAIL PROTECTED] signature.asc Description: Digital signature
Re: automatically restarting dying daemons?
Will Trillich said on Wed, Jun 30, 2004 at 04:34:06PM -0500: questions: 1) what's the best way (e.g. debian way) to monitor active daemons and restart them when necessary? maybe some utility already exists for this? or /proc/something? or `ps ax`? monit can do this. M pgppzZ9x1tzKA.pgp Description: PGP signature
Re: automatically restarting dying daemons?
Mark Ferlatte wrote: Will Trillich said on Wed, Jun 30, 2004 at 04:34:06PM -0500: questions: 1) what's the best way (e.g. debian way) to monitor active daemons and restart them when necessary? maybe some utility already exists for this? or /proc/something? or `ps ax`? monit can do this. As can webmin. -- Cheers John -- spambait [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
On Wed, Jun 30 at 03:43PM -0700, Mark Ferlatte wrote: Will Trillich said on Wed, Jun 30, 2004 at 04:34:06PM -0500: questions: 1) what's the best way (e.g. debian way) to monitor active daemons and restart them when necessary? maybe some utility already exists for this? or /proc/something? or `ps ax`? monit can do this. so let's go find monit... # dpkg -l monit No packages found matching monit. # dpkg -S monit imagemagick: /usr/share/doc/imagemagick/html/www/api/monitor.html xlibs: /usr/X11R6/include/X11/bitmaps/monitor.xbm xlibs: /usr/X11R6/include/X11/pixmaps/monitor.xpm svgatextmode: /usr/share/doc/svgatextmode/monitor-timings.howto.gz # apt-cache search monit | wc -l 226 # apt-cache search monit | grep monit | wc -l 81 # apt-cache search monit | grep monit | sort | pager aha! it's mon... # apt-cache show mon Package: mon Priority: extra Section: admin Installed-Size: 800 Maintainer: Roderick Schertler [EMAIL PROTECTED] Architecture: i386 Version: 0.99.2-2 Depends: perl, libmon-perl (= 0.10), libtime-period-perl, libtime-hires-perl, libc6 (= 2.2.3-7) Suggests: fping, libauthen-pam-perl, libfilesys-diskspace-perl, libnet-perl, libnet-dns-perl, libnet-ldap-perl, libnet-telnet-perl, libsnmp-perl, libstatistics-descriptive-perl Filename: pool/main/m/mon/mon_0.99.2-2_i386.deb Size: 175370 MD5sum: c98fe7752c129eae0ef3edcd75747276 Description: monitor hosts/services/whatever and alert about problems mon is a tool for monitoring the availability of services. Services may be network-related, environmental conditions, or anything that can be tested with software. If a service is unavailable mon can tell you with syslog, email, your pager or a script of your choice. You can control who gets each alert based on the time of day or day of week, and you can control how often an existing problem is re-alerted. . More information can be found at http://www.kernel.org/software/mon/. so let's try it-- # apt-get install mon # man mon # man moncmd nosing through the manpages for mon and moncmd, it looks like this will check (monitor) running daemons and send a flare when things go bad. so far so good... but what happens when the daemon that's to receive the flare (i.e. email -- e.g. exim in this case) is dead? is there some facility for when daemon Q dies, run this script that i missed? -- I use Debian/GNU Linux version 3.0; Linux boss 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST 2002 i586 unknown DEBIAN NEWBIE TIP #16 from Will Trillich [EMAIL PROTECTED] : Why are *.rpm (RED HAT PACKAGES) considered spawn of Satan? Because the Debian package system is a lot more sophisticated than the one Red Hat uses; lots more inter-dependency information is built in to a *.deb package. If you bypass that with an *.rpm file, you're taking chances with your system. Try to apt-get install debian-only packages if possible. (Also check out the alien package if you must.) Also see http://newbieDoc.sourceForge.net/ ... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
On Wed, Jun 30 at 06:25PM -0400, Derrick 'dman' Hudson wrote: On Wed, Jun 30, 2004 at 04:34:06PM -0500, Will Trillich wrote: | problem: xinetd, after working just fine and dandy for weeks at | a time, gets dozens of unexpected signal (source unknown) | and gives up the ghost. | | questions: | 1) what's the best way (e.g. debian way) to monitor active |daemons and restart them when necessary? maybe some |utility already exists for this? or /proc/something? |or `ps ax`? restartd. hmm. this sounds promising... $ apt-cache search restart | sort daemontools-installer - Installer package for building daemontools binary package firestarter - gtk program for managing and observing your firewall. freefont - Freeware font selection for X11 gentoo - A fully GUI configurable X file manager using GTK+ jesred - A redirector for Squid oss-preserve - Program to save/restore OSS mixer settings run - Watch programs and restart them if they die scsiadd - Add or remove SCSI devices by rescanning the bus. snmptrapfmt - A configurable snmp trap handler daemon for snmpd. xpacman - Basic Pacman zope-zshell - Zshell present a command line interface to zope xpacman? not quite what i had in mind. wokka wokka. :) $ apt-cache search restartd | sort $ dpkg -S restart debhelper: /usr/share/debhelper/autoscripts/postinst-init-norestart-invoke debhelper: /usr/share/debhelper/autoscripts/prerm-init-norestart debhelper: /usr/share/debhelper/autoscripts/prerm-init-norestart-invoke debhelper: /usr/share/debhelper/autoscripts/postinst-init-norestart $ dpkg -S restartd dpkg: *restartd* not found. at http://backports.org, i search for restartd and get Sorry, no packages found. :( -- I use Debian/GNU Linux version 3.0; Linux boss 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST 2002 i586 unknown DEBIAN NEWBIE TIP #21 from Will Trillich [EMAIL PROTECTED] : Looking to configure your Debian NETWORK SETTINGS? Look at the file /etc/network/interfaces (try man interfaces for more info). Then ifup -a to reload your settings, and ifconfig to display them. (Also check out apt-get install ipmasq!) Also see http://newbieDoc.sourceForge.net/ ... -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: automatically restarting dying daemons?
On Wed, 30 Jun 2004 22:43:54 -0500 Will Trillich [EMAIL PROTECTED] wrote: On Wed, Jun 30 at 06:25PM -0400, Derrick 'dman' Hudson wrote: On Wed, Jun 30, 2004 at 04:34:06PM -0500, Will Trillich wrote: | problem: xinetd, after working just fine and dandy for weeks at | a time, gets dozens of unexpected signal (source unknown) | and gives up the ghost. | | questions: | 1) what's the best way (e.g. debian way) to monitor active | daemons and restart them when necessary? maybe some | utility already exists for this? or /proc/something? | or `ps ax`? restartd. hmm. this sounds promising... snip xpacman? not quite what i had in mind. wokka wokka. :) snip at http://backports.org, i search for restartd and get Sorry, no packages found. At the risk of starting a flamewar about whether djb's tools are a good way to do things or not... :-) Have you looked at daemontools? apt-cache show daemontools-installer, apt-cache show svtools. The sole purpose of daemontools is to make sure a program keeps running properly. I have successfully used it on occasion when I was working with a program that was known for crashing, but didn't consider the program important enough to make it run dependably. daemontools worked great. HTH, Jacob -- GnuPG Key: 1024D/16377135 Random .signature #32: I prefer an OS made by programmers that needs marketing than an OS made by marketing that needs programmers... http://www.linux.org -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
