This sounds like a useful tool, is this something you can make available :-)
I'm not seeing anything obvious here
do you see anything processed on any of the actions inside the ruleset once the
problem starts?
when faced with problems like this in the past, I've sometimes resorted to
writing enough of the log to know what's what in between each action (and before
the first action in the ruleset), what will let me trace which action last
processed the last message that we saw.
David Lang
On Fri, 4 May 2018, Dave Caplinger via rsyslog wrote:
Date: Fri, 4 May 2018 22:09:11 +0000
From: Dave Caplinger via rsyslog <[email protected]>
To: rsyslog-users <[email protected]>
Cc: Dave Caplinger <[email protected]>
Subject: Re: [rsyslog] Queue stops writing to disk
...apologies if this is a re-post; initial reply was rejected because my email
address changed...
I have an [internal] tool to process and summarize pstats logs; it shows no
drops, fails, suspends, disk-assist usage, etc.
I was kind of expecting something obvious to pop out like a queue size to max
out and drops to begin, or an action that starts failing or suspending, but
since it was only 50 minutes of logs that may not be apparent yet.
Here's the output if you're interested:
Host: monvsyslog.na.lzb.hq
name logs /sec peak
bytes Mbit/s decomp_bytes ratio Mbit/s
imptcp(*/1515/IPv4) 962
60135 0.00
imptcp(*/1520/IPv4) 252571 140 166
86437962 0.37
imptcp(*/1521/IPv4) 133051 73 76
48402395 0.21
imptcp(*/1522/IPv4) 1373564 762 759
293498465 1.24
imptcp(*/1523/IPv4) 22807 12 12
4079930 0.02
imptcp(*/1524/IPv4) 606663 336 303
213386180 0.90
imptcp(*/1525/IPv4) 82425 45 76
20961108 0.09
imptcp(*/1526/IPv4) 20939065 11626 10995
1893232427 8.02
imptcp(*/1531/IPv4) 6323791 2107 4096
2399324276 6.10
imptcp(*/1532/IPv4) 491268 272 250
210879164 0.89
imptcp(*/1533/IPv4) 86258 47 48
29658282 0.13
imptcp(*/514/IPv4) 367957 204 181
50378823 0.21
imudp(*:1514) 3232143 1077 916
imudp(*:1515) 351663 117 102
imudp(*:1516) 946 1
imudp(*:1530) 175
imudp(*:1534) 6814 2 2
imudp(*:514) 77
imuxsock
name logs full drop_full
drop_nf maxqsize
1531-forward-sumo 1366192
1207
Net-1514 3232142
726
Net-1522 1373564
1117
Net-1524 606663
8231
Net-1526 20939064
36982
Net-1531 6323670
176705
fwd-udpspoof-1530 175
3
fwd-udpspoof-1534 6814
10
fwd-udpspoof-splunk-1514 3232143
577
io-work-q 1969225
2
main 1801589
54544
name processed failed suspended
resumed
1531-forward-sumo 1366193
Net-0514-LoginReport 550
Net-0514-authpriv 30803
Net-0514-bash_history 18
Net-0514-cron 4954
Net-0514-hazelcast 24
Net-0514-info-mail-authpriv 26369
Net-0514-jboss 130904
Net-0514-local1 379
Net-0514-local4 38
Net-0514-local5 17
Net-0514-local6 67
Net-0514-mail 305
Net-1514-FWCH 77
Net-1514-omfile 3232139
Net-1515-omfile 352625
Net-1516-omfile 946
Net-1520-rslog_ohs_oblog1log 9
Net-1520-rslog_ohs_oblog2log 252562
Net-1521-rslog_oam_accesslog1 1771
Net-1521-rslog_oam_accesslog2 857
Net-1521-rslog_oam_adminserverlog 12
Net-1521-rslog_oam_domlog 17998
Net-1521-rslog_oam_polmgr1log 4
Net-1521-rslog_oam_polmgr2diaglog 8277
Net-1521-rslog_oam_polmgr2log 7
Net-1521-rslog_oam_server1diaglog 76921
Net-1521-rslog_oam_server1oamlog 37
Net-1521-rslog_oam_server1omsmdiaglog 2079
Net-1521-rslog_oam_server1omsmlog 32
Net-1521-rslog_oam_server1omsmout 7410
Net-1521-rslog_oam_server1out 2272
Net-1521-rslog_oam_server2diaglog 1688
Net-1521-rslog_oam_server2oamlog 12
Net-1521-rslog_oam_server2omsmdiaglog 802
Net-1521-rslog_oam_server2omsmlog 21
Net-1521-rslog_oam_server2omsmout 2850
Net-1521-rslog_oam_server2out 1725
Net-1521-rslog_oam_server2polmgrout 8276
Net-1522-rslog_bi_server1accesslog 1666
Net-1522-rslog_bi_server1bidiaglog 344
Net-1522-rslog_bi_server1biserverlog 84662
Net-1522-rslog_bi_server1out 23329
Net-1522-rslog_bi_server2accesslog 1764
Net-1522-rslog_bi_server2bidiaglog 348
Net-1522-rslog_bi_server2biserverlog 83963
Net-1522-rslog_bi_server2out 23459
Net-1522-rslog_oim_server1accesslog 3486
Net-1522-rslog_oim_server1adminlog 25
Net-1522-rslog_oim_server1diaglog 197677
Net-1522-rslog_oim_server1oimdomlog 444700
Net-1522-rslog_oim_server1out 142167
Net-1522-rslog_oim_server1serverlog 5387
Net-1522-rslog_oim_server2accesslog 3591
Net-1522-rslog_oim_server2diaglog 198761
Net-1522-rslog_oim_server2out 144734
Net-1522-rslog_oim_server2serverlog 5363
Net-1522-rslog_soa_server1accesslog 1539
Net-1522-rslog_soa_server1diaglog 2124
Net-1522-rslog_soa_server1log 250
Net-1522-rslog_soa_server1out 115
Net-1522-rslog_soa_server2accesslog 1625
Net-1522-rslog_soa_server2diaglog 2124
Net-1522-rslog_soa_server2log 246
Net-1522-rslog_soa_server2out 115
Net-1523-rslog_oud_access 22687
Net-1523-rslog_oud_errors 120
Net-1524-sitauthwls1acclog 420
Net-1524-sitauthwls1authlog 3007
Net-1524-sitauthwls1outlog 4631
Net-1524-sitauthwls2acclog 420
Net-1524-sitauthwls2authlog 3046
Net-1524-sitauthwls2authout 4759
Net-1524-sitbatchserver1acclog 872
Net-1524-sitbatchserver1log 28038
Net-1524-sitbatchserver1out 101275
Net-1524-sitbatchserver2acclog 994
Net-1524-sitbatchserver2log 27921
Net-1524-sitbatchserver2out 68463
Net-1524-sitbatchserver3acclog 1118
Net-1524-sitbatchserver3log 27847
Net-1524-sitbatchserver3out 73538
Net-1524-sitbatchserver4acclog 868
Net-1524-sitbatchserver4log 28026
Net-1524-sitbatchserver4out 76680
Net-1524-sitdvwls1accesslog 378
Net-1524-sitdvwls1serverout 3454
Net-1524-sitdvwls2accesslog 882
Net-1524-sitdvwls2serverout 7150
Net-1524-sitwls1accesslog 5343
Net-1524-sitwls1acclog 1287
Net-1524-sitwls1serverlog 11874
Net-1524-sitwls1serverout 19801
Net-1524-sitwls2accesslog 4276
Net-1524-sitwls2acclog 1167
Net-1524-sitwls2serverlog 11201
Net-1524-sitwls2serverout 27430
Net-1524-sitwls3accesslog 682
Net-1524-sitwls3serverlog 392
Net-1524-sitwls3serverout 18885
Net-1524-sitwls4accesslog 673
Net-1524-sitwls4serverlog 1249
Net-1524-sitwls4serverout 3734
Net-1524-sitwlsa1srvlog 8457
Net-1524-sitwlsa1srvout 22491
Net-1524-sitwlsa2srvlog 3237
Net-1524-sitwlsa2srvout 697
Net-1525-rslog_wcp_admindiaglog 16
Net-1525-rslog_wcp_adminlog 23
Net-1525-rslog_wcp_domainlog 26811
Net-1525-rslog_wcp_ucm1out 896
Net-1525-rslog_wcp_ucm2out 784
Net-1525-rslog_wcp_ucmserver1log 30
Net-1525-rslog_wcp_ucmserver2log 24
Net-1525-rslog_wcp_wcportlet1log 38
Net-1525-rslog_wcp_wcportlet2log 27
Net-1525-rslog_wcp_wcspaces1accesslog 4876
Net-1525-rslog_wcp_wcspaces1diaglog 11499
Net-1525-rslog_wcp_wcspaces1log 8508
Net-1525-rslog_wcp_wcspaces1out 14148
Net-1525-rslog_wcp_wcspaces2accesslog 3649
Net-1525-rslog_wcp_wcspaces2diaglog 4663
Net-1525-rslog_wcp_wcspaces2log 196
Net-1525-rslog_wcp_wcspaces2out 6237
Net-1526-rslog_osb1_acclog 7784
Net-1526-rslog_osb1_srv1log 83
Net-1526-rslog_osb1_srv1out 1712
Net-1526-rslog_osb2_acclog 7648
Net-1526-rslog_osb2_srv1log 87
Net-1526-rslog_osb2_srv1out 1283
Net-1526-rslog_osb_admin_domlog 60
Net-1526-rslog_osb_admin_srvlog 8
Net-1526-rslog_soadpd2_wsm2log 7
Net-1526-rslog_soadpd2_wsmacclog 112
Net-1526-rslog_soadv1_soa1acc 789
Net-1526-rslog_soadv1_soa1diag 6965
Net-1526-rslog_soadv1_soa1log 367
Net-1526-rslog_soadv1_soa1out 7092
Net-1526-rslog_soadv2_soa2diag 226
Net-1526-rslog_soadv2_soa2log 221
Net-1526-rslog_soadv_adminlog 10
Net-1526-rslog_soadv_bamacc1log 113
Net-1526-rslog_soadv_bamdiag1log 312
Net-1526-rslog_soadv_bamserver1log 18
Net-1526-rslog_soadv_wsmacc2log 113
Net-1526-rslog_soadv_wsmserv2log 6
Net-1526-rslog_soapd1_bam1acclog 224
Net-1526-rslog_soapd1_bam1diaglog 588
Net-1526-rslog_soapd1_bam1log 14
Net-1526-rslog_soapd1_soa1acc 332
Net-1526-rslog_soapd1_soa1diag 70973
Net-1526-rslog_soapd1_soa1log 318
Net-1526-rslog_soapd1_soa1out 20664549
Net-1526-rslog_soapd2_bam2acclog 112
Net-1526-rslog_soapd2_bam2diaglog 504
Net-1526-rslog_soapd2_bam2log 15
Net-1526-rslog_soapd2_soa2acc 402
Net-1526-rslog_soapd2_soa2diag 103176
Net-1526-rslog_soapd2_soa2log 817
Net-1526-rslog_soapd2_soa2out 4126
Net-1526-rslog_soapd_admin1log 28
Net-1526-rslog_soapy1_bam1out 9531
Net-1526-rslog_soapy1_soa1acc 162
Net-1526-rslog_soapy1_soa1diag 4108
Net-1526-rslog_soapy1_soa1log 319
Net-1526-rslog_soapy2_bam2out 1216
Net-1526-rslog_soapy2_soa2acc 163
Net-1526-rslog_soapy2_soa2diag 15780
Net-1526-rslog_soapy2_soa2log 68
Net-1526-rslog_soapy2_soa2out 14933
Net-1526-rslog_soapy_adminlog 13
Net-1526-rslog_soapy_bamacc1log 113
Net-1526-rslog_soapy_bamacc2log 113
Net-1526-rslog_soapy_bamdiag1log 9531
Net-1526-rslog_soapy_bamdiag2log 1809
Net-1526-rslog_soapy_bamserver1log 7
Net-1526-rslog_soapy_bamserver2log 17
Net-1526-rslog_soapy_wsmserv1log 9
Net-1531-Ecomm 17641
Net-1531-catalina 3032775
Net-1531-catalina1 2927
Net-1531-catalina2 2927
Net-1531-catalina3 2927
Net-1531-pentaho 50407
Net-1531-rmslog4j 3037252
Net-1531-zcbploader 114
Net-1532-postgres 491268
Net-1533-httpd_access 86234
Net-1533-httpd_error 24
Net-1534 6814
action_283 175
action_308 4333
action_309 139
action_311 14
fwd-udpspoof-1530 175
fwd-udpspoof-1534 6814
fwd-udpspoof-splunk-1514 3232139
- Dave Caplinger
On May 4, 2018, at 4:14 PM, David Lang <[email protected]> wrote:
I'm not going to be able to figure things out withotu the full config, and
stripping lines out that you don't think matter actually makes it much harder
to analyze.
the fact that I see action 308, 309, and 311 in the logs tells me you have a
fairly complex config.
It would probably help if you can name the actions inside the net-1531 ruleset
(or at least one of them so we can see what action numbers are near it)
you should be looking for failures or suspended actions
On Fri, 4 May 2018, Scot Kreienkamp wrote:
Date: Fri, 4 May 2018 15:53:43 +0000
From: Scot Kreienkamp <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Queue stops writing to disk
I put the full impstats output up on https://pastebin.com/Yr2vvLck, hopefully
you can spot something I'm missing. I filtered out the noise (all the lines
where everything equals zero), otherwise it's all there. The problem was first
noticed at 10:30am so I posted the 10am to 11am ipmstats output.
Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | |
Mobile: 7349151444 | Email: [email protected]
-----Original Message-----
From: rsyslog [mailto:[email protected]] On Behalf Of David Lang
Sent: Thursday, May 3, 2018 5:41 PM
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Queue stops writing to disk
it's good that you have impstats running, it will let us track this down
what you need to look for is to find which of the 13 if statements inside the
ruleset are getting blocked and preventing the ruleset from progressing. If
those are action() statements you can name them to make them easy to find in the
pstats output, otherwise they will just be Action number.
David Lang
On Thu, 3 May 2018, Scot
Kreienkamp wrote:
Date: Thu, 3 May 2018 16:52:36 +0000
From: Scot Kreienkamp <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] Queue stops writing to disk
Hi everyone,
I keep running into a situation where a queue will just stop processing until
rsyslog is restarted, and I can't figure out why. Any help would be
appreciated.
I've moved some of my incoming messages to its own queue so it doesn't affect
everything in the main queue, but occasionally this queue will just go into
queueing mode and quit writing to disk until rsyslog is restarted. It happens
at seemingly random times, as much as a month apart or as close as two hours
later. I've ran the pstats through the analyzer and it found nothing. Here's
the pstats of that queue from an occurrence this morning when it quit writing
to disk:
May 3 09:50:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 09:50:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=2067518 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:00:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:00:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=2337424 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:10:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:10:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=2457977 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:20:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:20:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=1333045 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=175505 enqueued=194024 full=0 discarded.full=0
discarded.nf=0 maxqsize=175505
May 3 10:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=176105 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=176105
May 3 10:50:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:50:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=176705 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=176705
May 3 11:00:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:00:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=177305 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=177305
May 3 11:10:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:10:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=177905 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=177905
May 3 11:20:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:20:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=178505 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=178505
May 3 11:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=179105 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=179105
May 3 11:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=179705 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=179705
The truncated config for the queue is:
ruleset(name="Net-1531"
queue.type="LinkedList"
queue.size="250000"
queue.discardmark="200000"
queue.dequeueBatchSize="2048"
queue.workerThreads="1"
queue.workerThreadMinimumMessages="50000"
queue.filename="Net-1531"
){
13 if statements that determine where to log the info to
}
input(type="imudp" port="1531" ruleset="Net-1531")
input(type="imptcp" port="1531" ruleset="Net-1531")
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 | * 734-384-6403 | | * 7349151444 | *
[email protected]<mailto:%7BE-mail%7D>
www<http://www.la-z-boy.com/>.la-z-boy.com<http://www.la-z-boy.com/> |
facebook.<https://www.facebook.com/lazboy>com<https://www.facebook.com/lazboy>/<https://www.facebook.com/lazboy>lazboy<http://facebook.com/lazboy>
| twitter.com/lazboy<https://twitter.com/lazboy> |
youtube.com/<https://www.youtube.com/user/lazboy>lazboy<https://www.youtube.com/user/lazboy>
[cid:lzbVertical_hres.jpg]
This message is intended only for the individual or entity to which it is
addressed. It may contain privileged, confidential information which is exempt
from disclosure under applicable laws. If you are not the intended recipient,
you are strictly prohibited from disseminating or distributing this information
(other than to the intended recipient) or copying this information. If you
have received this communication in error, please notify us immediately by
e-mail or by telephone at the above number. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Confidentiality Notice: The content of this communication, along with any
attachments, is covered by federal and state law governing electronic
communications and may contain confidential and legally privileged information.
If the reader of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution, use or copying of the
information contained herein is strictly prohibited. If you have received this
communication in error, please immediately contact us by telephone at
402.361.3000 or e-mail [email protected].
Copyright 2000-2018 NTT Security (US) Inc., a wholly-owned subsidiary of NTT
Group. All rights reserved. ActiveGuard and Solutionary are registered
trademarks and NTT Security is a trademark of NTT Security GMBH. Solutionary,
the ActiveGuard logo icon, and the Solutionary logo icon are registered service
marks of NTT Security (US) Inc.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
