Update of /cvsroot/mailman/mailman/contrib
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv13692/contrib
Modified Files:
Tag: Release_2_1-maint
README.mmdsr mmdsr
Log Message:
Update Brad Knowles' mailman daily status report.
Index: README.mmdsr
===================================================================
RCS file: /cvsroot/mailman/mailman/contrib/README.mmdsr,v
retrieving revision 1.1.2.1
retrieving revision 1.1.2.2
diff -u -d -r1.1.2.1 -r1.1.2.2
--- README.mmdsr 30 Dec 2005 18:58:15 -0000 1.1.2.1
+++ README.mmdsr 29 Jan 2006 12:55:26 -0000 1.1.2.2
@@ -5,31 +5,41 @@
<http://sourceforge.net/tracker/index.php?func=detail&aid=1123383&group_id=103&atid=300103>
which see for possible patches and other enhancements.
-It is intended that there will be a 'cleaner', more complete readme file in
-the future. In the interim, here are Brad's original comments.
+Here goes the original mmdsr.readme by Brad ...
-I quickly whacked together a Daily Status Report script for Mailman (using
-Bourne shell, not Python ;), and thought that other folks might be interested
-in seeing it.
+========================================================================
+This is a basic Bourne shell script that I quickly hacked together for
+my own purposes, designed to be fired off at 23:59 every night, going
+through a variety of Mailman log files looking for entries specific
+to that date, summarizing the activities, and indicating problems or
+certain types of activity that might be of interest to someone trying
+to administer the server.
-The basic concept is a program that gets fired off at 23:59 every night, and
-goes through a variety of log files looking for entries specific to that date,
-and indicating problems or certain types of activity that might be of interest
-to someone trying to administer the server. It also does an "ls -la" of
-/usr/local/mailman/qfiles/*, so that you can see what is in the queue at the
-time of the running of the script.
+It also does an "ls -la" of /usr/local/mailman/qfiles/*, so that you
+can see what is in the queue at the time of the execution of the script.
-My concept was that this daily report would get e-mailed to the admin, or
-posted to a "reports" mailing list, where they could be archived and kept for
-future reference.
+This daily report will get e-mailed to the admin, or posted to a "reports"
+mailing list, where they can be archived and kept for future reference.
+If you don't define an address where the output e-mail should be sent,
+it will instead be printed to the standard output (thus allowing you to
+do something else with it).
-The script does not (yet) do any statistics calculations, although it should
-be relatively easy to hack together some basic stats using awk, sort, etc....
-Anyway, I thought I'd share it and let folks take a look at it, and if anyone
-has any recommended improvements, we can incorporate those and share them back
-out with everyone.
+Once I'd gone through a few revisions of my own on this tool, I
+thought that I would release the code to the public and get comments
+and suggestions from others in the Mailman community. This program is
+currently being used actively on the mail servers for python.org (where
+the mailman-users and other Mailman-related mailing lists are hosted),
+as well as many others.
-The code is written under a BSD-style license, so if you don't want to
-contribute any changes back to me, that's okay. Of course, I would prefer that
-you did, but I leave the choice up to you.
+Note that this script needs to be configured once to know where standard
+commands are located, where log files are kept, etc... (see the top
+500 lines or so of the script), but after that you don't need to feed
+it any input, or capture the output to be sent anywhere. This script
+takes care of all of that. All you should need to do is to call this
+script from a cron job at 23:59 (local time) every night.
+
+
+When looking at this script, perhaps during configuration, please keep
+in mind that it is heavily commented at the top, and everything should
+hopefully be self-evident.
Index: mmdsr
===================================================================
RCS file: /cvsroot/mailman/mailman/contrib/mmdsr,v
retrieving revision 1.1.2.2
retrieving revision 1.1.2.3
diff -u -d -r1.1.2.2 -r1.1.2.3
--- mmdsr 25 Dec 2005 03:21:56 -0000 1.1.2.2
+++ mmdsr 29 Jan 2006 12:55:26 -0000 1.1.2.3
@@ -112,6 +112,38 @@
# Adrian discovered that there was a log file format
# change between Mailman 2.1.5 and 2.1.6, which broke
# hourly statistics.
+#
+# 0.0.13 Update by Brad Knowles <[EMAIL PROTECTED]>
+# Updated on: Mon Dec 26 05:54:27 CET 2005
+# Bugs found by Tom G. Christensen (tgc99):
+# > ps output on solaris is full of whitespace but a further
+# > echo get's rid of it.
+# > The lines in the smtp log are sometimes broken up by a
+# > newline (right before the msgid) which throws of the
+# > summary. Piping it through sed first will rejoin the broken
+# > lines.
+# > Use $AWK instead of awk.
+# Enhancements from Mark Sapiro:
+# > The vette log summary lists posts held for moderation
+# > individually under "Other Errors:". The following patch
+# > (watch out for wrapped lines) summarizes them by list instead.
+#
+# 0.0.14 Update by Brad Knowles <[EMAIL PROTECTED]>
+# Updated on: Thu Dec 29 08:17:38 CET 2005
+# Added code to check /usr/local/mailman/data for
+# moderation hold queue
+#
+# 0.0.15 Update by Brad Knowles <[EMAIL PROTECTED]>
+# Updated on: Thu Jan 26 02:39:38 CET 2006
+# Tweaked display of summary data in the "smtp-failure" log,
+# and "other" category in the "vette" log, so as to reduce
+# the spewage when things go wonky, mostly by removing
+# unique-ifying data like message-id or sender address.
+# Also tweaked moderation queue information, to tell us
+# how many Python pickle files are in the data directory,
+# as opposed to just doing a directory listing and skipping
+# the files in the middle if there are too many -- now we
+# know how many are being skipped.
###############################################################################
# Set up locations of standard commands, directories, etc....
@@ -156,9 +188,19 @@
LOGDIR="/usr/local/mailman/logs"
###############################################################################
+# Mailman data directory
+###############################################################################
+DATADIR="/usr/local/mailman/data"
+
+###############################################################################
# Maximum number of subdirectory entries to display in report
###############################################################################
-MAX_LINES=20
+MAX_QUEUE_LINES=20
+
+###############################################################################
+# Maximum number of moderation queue pickle files to display in report
+###############################################################################
+MAX_DATA_LINES=100
###############################################################################
# Mailman Log files to check for errors.
@@ -182,8 +224,8 @@
# Specify sender address for report. Not used if recipient list is empty.
###############################################################################
-SENDER="[EMAIL PROTECTED]"
-RCPTS="[EMAIL PROTECTED],[EMAIL PROTECTED]"
+SENDER="[EMAIL PROTECTED]"
+RCPTS="[EMAIL PROTECTED]"
###############################################################################
# If you run this program in cron at 23:59:00, you need to sleep sixty
@@ -203,7 +245,8 @@
# there is an easier cross-platform way to do it, please let me know.
###############################################################################
-MYUID=`$PS -o user -p $$ | $TAIL -1`
+GRABUID=`$PS -o user -p $$ | $TAIL -1`
+MYUID=`echo $GRABUID`
RUNAS="mailman"
###############################################################################
@@ -254,7 +297,7 @@
$TOUCH $TMPLOG
echo "Log file: $LOG" >> $TMP
echo "==============================" >> $TMP
- $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >> $TMPLOG
+ $SED -e :a -e '$!N;s/\n //;ta' -e 'P;D' $LOGDIR/$LOG | $GREP -si "^$DAY
[0-9][0-9:]* $YEAR" >> $TMPLOG
if [ -f "$LOGDIR/${LOG}" ] ; then
@@ -264,7 +307,7 @@
echo "Hourly Summary of Posts" >> $TMP
echo "-----------------------" >> $TMP
- $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.*$//' $TMPLOG |
$UNIQ -c | $SORT -n +1 | awk '{ printf( "%8d %02d:00-%02d:59\n", $1, $2, $2 )
}' >> $TMP
+ $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.*$//' $TMPLOG |
$UNIQ -c | $SORT -n +1 | $AWK '{ printf( "%8d %02d:00-%02d:59\n", $1, $2, $2 )
}' >> $TMP
echo "" >> $TMP
echo "Post Count by List" >> $TMP
@@ -295,7 +338,7 @@
echo "" >> $TMP
echo "Hourly Summary of Messages Sent" >> $TMP
echo "-------------------------------" >> $TMP
- $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.* for / /' -e 's/
recips,.*$//' $TMPLOG | awk '{ val=int($1); sum[val]+=$2 } END { for (i=0;
i<24; i++) { printf "%8d %02d:00-%02d:59\n", sum[i], i, i } }' >> $TMP
+ $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e 's/:.* for / /' -e 's/
recips,.*$//' $TMPLOG | $AWK '{ val=int($1); sum[val]+=$2 } END { for (i=0;
i<24; i++) { printf "%8d %02d:00-%02d:59\n", sum[i], i, i } }' >> $TMP
else
@@ -416,9 +459,19 @@
$GREP -i 'Posting to a moderated newsgroup' $TMPLOG | $AWK '{ print
$6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
echo "" >> $TMP
+ echo "Post to moderated list (by list):" >> $TMP
+ echo "------------------------------" >> $TMP
+ $GREP -i 'Post to moderated list' $TMPLOG | $AWK '{ print $6 }' |
$SORT | $UNIQ -c | $SORT -nr >> $TMP
+
+ echo "" >> $TMP
echo "Other Errors:" >> $TMP
echo "------------------------------" >> $TMP
- $EGREP -vi '(Post by non-member|suspicious header|message
approved|Discarded posting|bulk message discarded|junk message
discarded|Message has implicit destination|Posting to a moderated
newsgroup|Message discarded, msgid)' $TMPLOG | $SED 's/^.* ([0-9]*) //' | $SORT
| $UNIQ -c | $SORT -nr >> $TMP
+MYUID=`$PS -o user -p $$ | $TAIL -1`
+ $EGREP -vi '(Post by non-member|suspicious header|message
approved|Discarded posting|bulk message discarded|junk message
discarded|Message has implicit destination|Posting to a moderated
newsgroup|Post to moderated list|Message discarded, msgid)' $TMPLOG | $SED -e
's/^.* ([0-9]*) //' -e 's/, message-id=<[^> ]*>:/:/' | $SORT | $UNIQ -c | $SORT
-nr >> $TMP
+
+ elif [ "${LOG}" = "smtp-failure" ] ; then
+
+ $SED 's/^.* ([0-9]*) //' $TMPLOG | $SED 's/delivery to [^@ [EMAIL
PROTECTED]@ ]* failed with code/delivery failed with code/g' | $SORT | $UNIQ -c
| $SORT -nr >> $TMP
else
@@ -446,7 +499,7 @@
$RM -f $TMPLOG
$TOUCH $TMPLOG
$LS -la $DIR >> $TMPLOG
- LINES=`wc -l < $TMPLOG`
+ LINES=`$WC -l < $TMPLOG`
LINES=`expr $LINES - 3`
echo "Subdirectory: $DIR" >> $TMP
@@ -454,12 +507,12 @@
echo "Entries: $LINES" >> $TMP
echo "------------------------------" >> $TMP
- if [ $LINES -le $MAX_LINES ] ; then
+ if [ $LINES -le $MAX_QUEUE_LINES ] ; then
$CAT $TMPLOG >> $TMP
else
$HEAD $TMPLOG >> $TMP
echo "" >> $TMP
- echo " More than $MAX_LINES total entries, skipping ..." >> $TMP
+ echo " More than $MAX_QUEUE_LINES total entries, skipping ..." >> $TMP
echo "" >> $TMP
$TAIL $TMPLOG >> $TMP
fi
@@ -467,6 +520,45 @@
done
+echo "******************************" >> $TMP
+echo "Data Directory Contents" >> $TMP
+echo " (Python pickle files only)" >> $TMP
+echo "******************************" >> $TMP
+
+cd $DATADIR
+
+$RM -f $TMPLOG
+$TOUCH $TMPLOG
+$LS -la | $GREP -i '\.pck$' >> $TMPLOG
+LINES=`$WC -l < $TMPLOG`
+
+echo "Python pickle files: $LINES" >> $TMP
+echo "==============================" >> $TMP
+echo "" >> $TMP
+
+if [ $LINES -le $MAX_DATA_LINES ] ; then
+ $CAT $TMPLOG >> $TMP
+else
+ $HEAD $TMPLOG >> $TMP
+ echo "" >> $TMP
+ echo " More than $MAX_DATA_LINES total entries, skipping ..." >> $TMP
+ echo "" >> $TMP
+ $TAIL $TMPLOG >> $TMP
+fi
+echo "" >> $TMP
+
+echo "******************************" >> $TMP
+echo "Moderation Queue Summary" >> $TMP
+echo "******************************" >> $TMP
+echo "" >> $TMP
+
+$AWK '{ print $NF }' $TMPLOG | $GREP -i '^heldmsg-' | $SED -e 's/^heldmsg-//'
-e 's/-[0-9]*\.pck$//' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+echo "" >> $TMP
+
+###############################################################################
+# Okay, we're done with all the code stuff, now handle the output
+###############################################################################
+
if [ "${RCPTS}x" != "x" ] ; then
$SENDMAIL -t -f$SENDER < $TMP
else
_______________________________________________
Mailman-checkins mailing list
[email protected]
Unsubscribe:
http://mail.python.org/mailman/options/mailman-checkins/archive%40jab.org
