Send inn-workers mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/inn-workers
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of inn-workers digest..."
Today's Topics:
1. Re: innwatch forks without reason (Julien ?LIE)
2. Re: concurrency (Nick Edwards)
3. Re: innwatch forks without reason (Julien ?LIE)
----------------------------------------------------------------------
Message: 1
Date: Sat, 13 Sep 2014 14:54:04 +0200
From: Julien ?LIE <[email protected]>
To: [email protected]
Cc: Lauri Tirkkonen <[email protected]>
Subject: Re: innwatch forks without reason
Message-ID: <[email protected]>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hi Lauri,
>> innwatch apparently creates a child process only for sleeping and then
>> waits on that process. The forked-off process is not killed by 'rc.news
>> stop' (only its parent is), and will only die after it's done sleeping.
>> If running under SMF on illumos/Solaris, this causes the service to
>> likely drop into maintenance state (since not all processes die within
>> timeout).
>
> Thanks for your bug report and also your patch.
Is the subprocess "sleep 600" properly killed with your patch?
I see that it is still running on my system (but maybe it is not seen by
illumos/Solaris -- does it only look at innwatch?)
--
Julien ?LIE
? Quousque tandem ? ? (Cic?ron)
------------------------------
Message: 2
Date: Sun, 14 Sep 2014 02:13:27 +1000
From: Nick Edwards <[email protected]>
To: Julien ?LIE <[email protected]>
Cc: [email protected]
Subject: Re: concurrency
Message-ID:
<CAMD-=VLttv6h=jzfsruxy1m5aws4rtkbxmk5lbphg6e1+ix...@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
On 9/13/14, Julien ?LIE <[email protected]> wrote:
> Hi Nick,
>
>>>> So far, inn working well, but I need to do two things, firstly, limit
>>>> max user concurrency (8) at any given time, the closest I can find is
>>>> running innd with -H -X which does not sound appropriate,
>>>
>>> I believe you should do that with a Perl or Python auth hook.
>>
>> Oh, really? wow, hard to imagine inn's been around for so long without
>> basic features
>
> I bet the reason is that it is very easy to do what you want to achieve
> with the very efficient Perl hooks facilities INN provides.
>
perl and efficient should never be used together when it comes to
system resource usage :)
especially like this, when you have on average 3100 concurrent users
on a server, I'd hate to see the load on the box if I had to run that
script :-)
> After a few searches on Google, I found out a bit of code written by
> Fr?d?ric Senault:
> http://news.lacave.net/inn/auth_rate.pl
>
> For your needs, this code should be run at access time (instead of auth
> time).
>
> He basically checks user concurrency this way:
>
> my $oc = 0;
>
> open(P, '/bin/ps -ax |') && do {
> while(<P>) {
> if(/nnrpd: (\S*)/) {
> if($1 eq $attributes{hostname}) {
> $oc++;
> }
> }
> }
> close(P);
> };
>
> if($oc > $MAX) {
> sleep(2);
> return ( '502', 'Too many connections from your host' );
> }
>
>
>
>> my manager has already questioned why it has taken so long, he's
>> very relaxed with time management, else I would have been ordered to
>> dump inn 5 days ago, but even his patience is running dry, all these
>> hassles when with dnews its simple as adding into access.conf
> [...]
>> inn is so so so much more complex and although free
>
> I confess INN is not "newbie-friendly" as you also say but on the other
> hand, you can customize it and achieve pretty interesting and powerful
> things with its hooks facilities (in Perl or Python).
> Well, of course it implies that you know such languages, though they are
> not in fact complex (see the previous Perl code).
>
>
>
> Regarding the other dnews extensions you mention (quota, input/output
> bandwidth speed...), and even the one discussed here, I see in our
> inn-workers archives that we talked about them:
> https://lists.isc.org/pipermail/inn-workers/2003-September/012063.html
> https://lists.isc.org/pipermail/inn-workers/2000-July/003386.html
>
> It appears that the best way to do that is the use of xinetd (or a
> similar utility). You then get a few handy features for controlling
> access at the TCP level and also get the advantage of any further
> improvements in those programs.
> "No need to reinvent the wheel." :)
spawning from xinetd or tcpserver is far from optimum with so many
users (we learnt that in old days with qmail, vpopmail, and so on,
dnews re-invented that wheel because it clearly needed to be, and
given the load on that 32bit server rarely exceeds above .70 it
clearly does it nicely, and if it can, no reason inn cant.
>
>> Thank you everyone for your help, I certainly learned a lot in this
>> past week, even if it turns out to be of no immediate use to us here,
>> it might be if I ever change jobs in an IPv6 only world.
>
> Good luck with your projects! I am a bit sorry that INN did not fit
> your needs. If only you had used the straight-forward adequate program
> (like xinetd) to fulfil your expectations on user tracking and limiting,
> you would have had a more pleasant experience!
>
as above, more pleasant, I dont think so with that many users, it
would be worse than imaginable. Oh and please dont say split the load
of more servers, again, if DNews can run so smoothly on one crappy
dl380g5 with 2 GB ram (with 35 TB DAS array), why would we waste
thousands more to run cluster of inn boxes, that would get me fired
:-)
PS your google foo must be far greater than mine, 99/100 hits ended up
beibg for holiday inns, the upside to all that wasted googling was, I
know have a new destination for my next holiday LOL
Nik
------------------------------
Message: 3
Date: Sat, 13 Sep 2014 19:04:44 +0200
From: Julien ?LIE <[email protected]>
To: [email protected]
Cc: Lauri Tirkkonen <[email protected]>
Subject: Re: innwatch forks without reason
Message-ID: <[email protected]>
Content-Type: text/plain; charset=ISO-8859-1
Hi again Lauri,
>>> innwatch apparently creates a child process only for sleeping and then
>>> waits on that process. The forked-off process is not killed by 'rc.news
>>> stop' (only its parent is), and will only die after it's done sleeping.
>>> If running under SMF on illumos/Solaris, this causes the service to
>>> likely drop into maintenance state (since not all processes die within
>>> timeout).
>>
>> Thanks for your bug report and also your patch.
>
> Is the subprocess "sleep 600" properly killed with your patch?
> I see that it is still running on my system (but maybe it is not seen by
> illumos/Solaris -- does it only look at innwatch?)
Here is a new suggestion of patch to deal with the problem:
* add a new "-i" flag to innwatch to specify how many seconds it should sleep
when started;
* add a new "-i" flag to cnfsstat to specify how many seconds it should sleep
when started;
* when innwatch is killed, also kill the forked-off process it uses when
sleeping.
This allows to fix the issue you reported, and also to fix a similar issue
if stopping INN within 60 seconds after its start.
Here are the 3 files to patch.
Could you please test these changes on your illumos/Solaris and tell me if
they also fix your issue?
Thanks!
--- rc.news 2014-09-07 16:24:59.000000000 +0200
+++ /home/news/bin/rc.news 2014-09-13 18:46:35.000000000 +0200
@@ -32,7 +32,7 @@
# Stop innwatch (if running)
if [ -f $WATCHPID ]; then
kill `cat $WATCHPID`
- rm -f $WATCHPID
+ rm -f $WATCHPID ${LOCKS}/LOCK.innwatch
fi
printf "Stopping innd: "
@@ -163,12 +163,12 @@
# Gee, looks like lisp, doesn't it?
${DOINNWATCH} && {
echo "Scheduled start of ${INNWATCH}."
- ( sleep 60 ; ${INNWATCH} ) > /dev/null &
+ ${INNWATCH} -i 60 > /dev/null &
}
${DOCNFSSTAT} && {
echo "Scheduled start of cnfsstat."
- ( sleep 60 ; ${PATHBIN}/cnfsstat -s -l -P ) > /dev/null &
+ ${PATHBIN}/cnfsstat -i 60 -s -l -P > /dev/null &
}
RMFILE=${MOST_LOGS}/expire.rm
--- innwatch 2014-09-07 16:24:59.000000000 +0200
+++ /home/news/bin/innwatch 2014-09-13 18:23:29.000000000 +0200
@@ -17,6 +17,9 @@
PROGNAME=innwatch
LOCK=${LOCKS}/LOCK.${PROGNAME}
DAILY=${LOCKS}/LOCK.news.daily
+INNWATCHINITIALSLEEPTIME=1
+CHILDPID=
+
## Where to put the timestamp file (directory and filename).
TIMESTAMP=${LOCKS}/${PROGNAME}.time
@@ -41,14 +44,21 @@
shift
;;
X-f*)
- FILE=`expr "$1" : '-s\(.*\)'`
+ FILE=`expr "$1" : '-f\(.*\)'`
;;
+ X-i)
+ INNWATCHINITIALSLEEPTIME=$2
+ shift
+ ;;
+ X-i*)
+ INNWATCHINITIALSLEEPTIME=`expr "$1" : '-i\(.*\)'`
+ ;;
X-l)
LOGFILE=$2
shift
;;
X-l*)
- LOGFILE=`expr "$1" : '-s\(.*\)'`
+ LOGFILE=`expr "$1" : '-l\(.*\)'`
;;
X-t)
INNWATCHSLEEPTIME=$2
@@ -86,7 +96,7 @@
exit 0
}
-trap 'rm -f ${LOCK} ${WATCHPID} ; exit 1' 1 3 15
+trap 'rm -f ${LOCK} ${WATCHPID} ; test -n "${CHILDPID}" && kill ${CHILDPID} ;
exit 1' 1 3 15
echo "$$" > ${WATCHPID}
## The reason why we turned innd off, and its, and our current state.
@@ -102,7 +112,9 @@
## We need to remember the process ID of innd, in case one exits.
## But we need to wait for innd to start before we can do that.
while PID=`cat ${SERVERPID} 2>/dev/null`; test -z "${PID}"; do
- sleep ${INNWATCHSLEEPTIME}
+ sleep ${INNWATCHINITIALSLEEPTIME} & CHILDPID=$!
+ wait
+ CHILDPID=
done
trap '(
@@ -126,8 +138,10 @@
NEXTSLEEP=1
HASEXITED=false
-while sleep ${NEXTSLEEP}
+while { sleep ${NEXTSLEEP} & CHILDPID=$! ; }
do
+ wait
+ CHILDPID=
NEXTSLEEP=${INNWATCHSLEEPTIME}
## If news.daily is running, idle: we don't want to change the
--- cnfsstat 2014-09-07 16:24:11.000000000 +0200
+++ /home/news/bin/cnfsstat 2014-09-13 18:45:18.000000000 +0200
@@ -29,12 +29,13 @@
Summary tool for CNFS
Usage:
- $0 [-ahpPsv] [-c class] [-l [seconds]] [-m buffer]
+ $0 [-ahpPsv] [-c class] [-i seconds] [-l [seconds]] [-m buffer]
If called without args, does a one-time status of all CNFS buffers
-a: print the age of the oldest article in the cycbuff
-c class: print out status of CNFS buffers in that class
-h: this information
+ -i seconds: initial sleep of that many seconds at startup
-l [seconds]: loop like vmstat, default seconds = 600
-m buffer: print out information suitable for MRTG
-p: print out an MRTG config file
@@ -54,7 +55,7 @@
my %opt = (c=>\$oclass, m=>\$obuffer);
Getopt::Long::config('no_ignore_case');
GetOptions(\%opt,
- "-a", "-c=s", "-h", "-l:i", "-m=s",
+ "-a", "-c=s", "-h", "-i=i", "-l:i", "-m=s",
"-p", "-P", "-s", "-v");
usage() if $opt{'h'};
@@ -96,6 +97,13 @@
mrtg($obuffer) if $obuffer;
mrtg_config() if $opt{'p'};
+if(defined($opt{'i'}) && $opt{'i'} > 0) {
+ sleep($opt{'i'});
+ if (!$use_syslog) {
+ print STDOUT "$opt{'i'} seconds later:\n";
+ }
+}
+
START:
That's all (man pages and changelog will also be updated).
--
Julien ?LIE
? Quousque tandem ? ? (Cic?ron)
------------------------------
_______________________________________________
inn-workers mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/inn-workers
End of inn-workers Digest, Vol 64, Issue 10
*******************************************
