On 11/19/2013 05:45 AM, Rainer Gerhards wrote:
OK, I am slowly catching up and can most probably look into this problem
now.
Erik, I think I need to create an instrumented version in any case so that
we can get a bit more insight. Are you able to deploy such a version from
git or source tarball? Also, I will need at least a partial debug log, from
yeah, either one of those would work, it's just that last time I
tried it I had hard time finding what the build dependencies are, is
there any list somewhere? I assume there is since you are able to build
ubuntu packages, but didn't find it in git repo.
the next challenge how to install what I've built, if you tell me
which files are relevant (just rsyslogd executable? or relp libs?) I
could copy them over to the right directories...
before things go wrong up until the situation recovers. If the debug log
get's to large, "debug on demand" (turn on via signal) will probably be
helpful.
Could you assist with these things?
I guess, I'll try to get the debug log, but somehow last night my
script that does just short bursts of log messages stopped causing the
problem. Seems there is something else that also has impact on this, not
sure what since nothing else changed... anyway, will try to get the
debug output...
erik
Thx,
Rainer
On Tue, Nov 19, 2013 at 9:08 AM, David Lang <[email protected]> wrote:
On Mon, 18 Nov 2013, Erik Steffl wrote:
On 11/18/2013 08:06 PM, David Lang wrote:
could you run impstats with a rapid reporting cycle while you duplicate
the problem?
run impstats how? I can add it, any suggested configuration? Just log
into a file every minute?
yes, log into a file every minute or so. We may end up shrinking this to a
faster interval
When you log as fast as you can, I expect that the queue will grow, and
then take a little bit of time to drain off. If it gets stuck while
draining, the pstats output will show us.
David Lang
With a rapid trigger like this, a new debug log may be helpful with less
'stray noise' in it.
will try to set that up, at least it will be easier to get close to the
time the problem starts (sometime between 1st and 100th burst message)
Speculation:
My guess is that something is getting 'stuck' processing the last
message of the third batch, and this is stalling the system (probably
something is holding a lock that's preventing RELP from finishing
processing messages, which is causing the senders to pause until RELP
finally acknowledges the sent messages)
knowing if all the messages from the script make it to the log or if
some of them don't make it until things startup again would be useful
data. This may involve creating a custom output template to add extra
time info (received time, now, etc) to the output line and see what
happens.
sounds reasonable, will try to figure out if that's happening,
thanks!
erik
David Lang
On Mon, 18 Nov 2013, Erik Steffl wrote:
figured out more specific scenario that is causing the silence.
to recap this is the current setup:
- 6 machines sending logs to collector machine using RELP
- 1 machine collecting all the logs on disk (20-30 files at a time,
split by origin etc.)
There is fairly steady traffic of 200-300 messages per second, 200 -
300 kB/s.
In this scenario if I run a program that logs 100 messages as fast as
possible the whole system goes silent (every THIRD time this happens),
all six senders stop sending messages (or almost stop, the traffic
goes down to maybe 1kB/s) and doesn't start again until I run another
burst of messages (which is equally weird, why would it start again
after next burst of messages?).
The program that sends 100 messages is running on the log collector
host (not sure if it's relevant, likely not).
This produces the silences virtually 100% (every third time it runs),
as long as I cron the 100-messages-log-sender the problem is there, if
I comment it out it works fine (tested both number of times, sometime
left it running for days).
Previously I thought this might be related to renaming of the files
or reload rsyslog (HUP signal) but that is completely irrelevant.
Regardless of renaming or HUP signal it's the burst of messages that
makes it go silent (and somehow signaling to all senders to go silent,
perhaps by not sending RELP confirmations?).
The program that logs burst of 100 messages is a perl script that
does essentially this (some irrelevant parts left out, think it would
run as is but it might need some fixes):
use strict;
use Log::Log4perl;
my $logConfig = '
log4perl.rootLogger=DEBUG, SYSLOG
log4perl.appender.SYSLOG = Log::Dispatch::Syslog
log4perl.appender.SYSLOG.min_level = debug
log4perl.appender.SYSLOG.ident = myTag
log4perl.appender.SYSLOG.facility = local0
log4perl.appender.SYSLOG.layout = Log::Log4perl::Layout::PatternLayout
log4perl.appender.SYSLOG.layout.ConversionPattern=@cee:%m
';
Log::Log4perl::init(\$logConfig);
my $l = Log::Log4perl::get_logger();
my $now = localtime();
for(my $i = 0; $i < 100; $i++) {
$l->info(JSON::encode_json({now=>$now,i=>$i}));
}
it needs liblog-log4perl-perl and liblog-dispatch-perl ubuntu
packages (or equivalent on other distros).
This does not seem to be related to rate-limiting (as documented)
since there are no dropped messages, didn't find any rsyslog messages
about dropping messages due to rate limiting (grep'd all files in
/var/log/). Could this be related to Queue slowdown? We do not set
dequeueslowdown for any queues.
Any ideas why would a short burst of 100 messages cause this problem?
thanks!
erik
On 11/14/2013 01:53 AM, Rainer Gerhards wrote:
On Thu, Nov 14, 2013 at 6:58 AM, Erik Steffl <[email protected]> wrote:
did you have a chance to look at debug log?
not yet, but I am finalizing the 8.1.0 release. Bear with me another
couple
of days pls.
Rainer
did more testing in the meantime, figured out that:
6 senders, 1 collector version 7.5.4, 300 kB/s always works
6 senders, 1 collector versions 7.5.5, 7.5.6 300 kB/s do not work
6 senders, 2 collectors versions 7.5.5, 7.5.6 300 kB/s mostly
work but
not always (periods of silence are much more predicable if the re is
no
reload/HUP signal)
1 sender, one collector any version always works (this has less
files
open plusthe total traffic is around 100lB/s, not sure if that's
siginificant)
Looked at strace on both sides (sender and receiver) during time
when
traffic went from 300kB/s down to virtually zero but see nothing
suspicious, essentially there is lot of messages sent, then one
message
confirming all of them. Then at some point there is timeout, then few
messages are confirmed one by one then nothing, then it starts again.
As far as I can tell it's the rename of the files that causes the
problem (we rename athen remove them but if I comment out the remove
part
the problem is still there). We never rename the files that rsyslogd
is
actually writing to (even though they are still open).
Given the behaviour in the scenarios above it seems something got
broken
between 7.5.4 and 7.5.5. Did the code that handles renamed files
change in
any way? I looked at changelog (http://www.rsyslog.com/
changelog-for-7-5-5-v7-devel/) before but didn't see anything that
looks
related.
Any ideas how to troubleshoot this further?
thanks!
erik
On 11/05/2013 10:04 AM, Rainer Gerhards wrote:
On Tue, Nov 5, 2013 at 3:41 AM, Erik Steffl <[email protected]> wrote:
See the debug log attached, the first line is the last message
in one
of
the files (maybe not last message overall, but all communication
stops at
that second, there's number of files so it's hard to figure out
which
message is the last one).
Also noticed that number of messages in log with same number
at the
beginning drops right after this message, I assume the beginning
of the
line has something to do with time, these are the counts and
values at
the
beginning of the debug log line:
It's the last 5 digits of a Unix timestamp - so you don't get an
absolute
time, but an idea how things progress. After the dot I think there
are ms
(or so) and after the colon there is a thread ID (same ID = same
thread).
Hope that helps a bit, will try to look at the debug log tomorrow
morning.
Rainer
21264 9547.
22378 9548.
2725 9549.
1877 9550.
...9551 to 9969 left out, low counts around 2000
...9970 to 0519 left out, low counts around 6
6 0520.
6 0521.
39302 0522.
51997 0523.
thanks!
erik
On 11/03/2013 01:05 PM, David Lang wrote:
I would post everything in the log file starting with the last
log that
it successfully sent and continuing a while after that message (a
while
being a couple hundred lines or so to be safe)
once we look at the log we may ask for more.
David Lang
On Sun, 3 Nov 2013, Erik Steffl wrote:
Date: Sun, 03 Nov 2013 11:45:03 -0800
From: Erik Steffl <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Rsyslog with RELP not sending/receiving
messages for
long intervals
I have debug log where I was able to find last message in one
of the
logs files (right before the silence period) but it's 3 GB :)
Any hint
how much to pick the relevant part of the debug log? Is the
timestamp
part of the log message? There is something in the beginning of
each
debug log line but I don't know how to translate it to time
(part of
it is transaction id, the rest not sure about)
thanks! (and of course I unserstand you can't drop everything
and
work on my problem, thanks for answers!)
erik
On 11/02/2013 08:51 AM, Rainer Gerhards wrote:
On Fri, Nov 1, 2013 at 11:31 PM, Erik Steffl <[email protected]>
wrote:
running reload every 43 seconds, behaviour is exactly the
same
(30 min
traffic, 15 min silence) so I think it's not related to reload.
will test the latest version and I guess re-test the older
version and
go from there...
It would probably be a good idea to try to capture a debug
log so
that we
can see what rsyslog itself thinks what happens. Note that you
can
also
just enable it shortly before you expect the problem to happen
(google for
"debug on demand").
Right now, I am totally busy with complex work and cannot look
at it.
If
this is for a business, I would strongly suggest to purchase
one of
the
low-priced support options. That would enable an engineer to
begin
working
on the case within half a day or so. I would *really love* to
drop
this to
someone else, as I really don't like the smell of this bug --
but if
I do,
I am pretty sure other bugs will creep up, and that would again
be the
death of the new work I am doing (and it took me 3 years or so
to get
to
this point!).
So it looks like non-paid work needs to stay focused on that
work,
which
will be useful for all users in the hopefully not so distant
future.
All
non-emergency bugs (usually security related things) or request
thus
need
to be on hold for a couple of days or maybe till the end of
month.
If you get a debug log, I will have time to at least look
quickly at
it,
but I can't say if that will be sufficient to provide a solution.
sorry for that and I hope for your understanding.
Rainer
thanks!
erik
On 11/01/2013 02:29 PM, David Lang wrote:
Ok, this is definantly sounding like a bug in RELP, it
worked on an
earlier version, check if it works on the latest version, and
then I
would suggest fileing a bug on the www.rsyslog.com bugtracker
This sounds like something Rainer is going to need to look
at. He's
in
Germany, so I believe his weekend has started, he may see it
over
the
weekend or it may wait until his work week starts.
David Lang
On Fri, 1 Nov 2013, Erik Steffl wrote:
pretty sure reload is sending HUP (that's upstart job as
inlcluded in
adiscon packages), the rsyslog keeps the same PID, I even
tried to
strace it while running reload, here's syslog message:
Nov 1 10:47:25 ip-10-238-198-103 rsyslogd: [origin
software="rsyslogd" swVersion="7.5.5" x-pid="3865"
x-info="http://www.rsyslog.com";] rsyslogd was HUPed
but I think it must be related to something else cause
when I
removed
the reload (so the log mover was doing only rename, upload,
remove) it
was still exhibiting the same works for 30 min, silent for
15 min
etc.
running reload every 43 seconds now and will see what
happens...
erik
On 11/01/2013 12:57 PM, David Lang wrote:
a HUP of the server should not cause any interruption in
processing
messages.
are you sure upstart is seing HUP not doing a full
stop/start?
could you try sending HUP to rsyslog manually a few times
to see
if you
can duplicate the problem?
David Lang
On Fri, 1 Nov 2013, Erik Steffl wrote:
sorry, it wasn't entirely clear in my previous email
but we
are doing
ubuntu upstart reload which is sending HUP signal, pretty
sure
it's
doing the right thing since logrotate scripts use it, syslog
message
says rsyslog received HUP signal, rsyslogd keeps the same
PID
and all
files are closed (as verified using lsof).
On top of that if, during one of these quiet periods,
I try to
send
messages from another client (on which I just restarted
rsyslog
so it
has no error counters etc.) I think it's also not sending
anything,
when stracing it (the sender) I see it recieved messages
from
logger
but is not sending them anywhere. So I think that the
collector
rsyslog continues to send some kind of signal that it's not
ready but
not sure why or how to figure out whether it actually does
it.
Any ideas how to troubleshoot this? I think I'll
record more
of
strace output and try to reconstruct RELP messages, I guess
there is
some RELP response that has something else than RSP-CODE =
200
(looking at http://www.rsyslog.com/doc/relp.html)
I was just about to publish the S3 upload scripts but
since
they
cause rsyslog to misbehave this way I didn't do it yet,
maybe
I'll
post them as they are and fix later...
erik
On 11/01/2013 01:02 AM, David Lang wrote:
the purpose of RELP is to tell the senders if the
receiver was
able to
accept the message or not, so that's working as designed.
When you do a full restart of rsyslog, it stops
processing new
messages
and disconnects all senders (you are doing a full stop of
rsyslog and
then starting it from scratch, this takes time). My guess
is
that the
senders are detecting 'too many failures' when trying to
send
messages,
so they are backing off and sleeping for a while rather
tthan
performing
a mini DOS attack on the network and server. Every third
restart is
probably triggering a threshold.
Instead of doing a full restart, just send the rsyslog
daemon a
HUP
signal instead. That will tell rsyslog to flush and close
all
files so
that you can rotate them (If you are doing compression on
the
files, you
may need to sleep for a few seconds to let the fluch
complete).
Rsyslog
can continue to receive new messages during this time, so
your
senders
will not see an outage.
By the way, I'm needing a script to upload rsyslog
archives to
S3,
could
you send me a copy of yours? (remove any passwords first
please
:-)
David Lang
On Thu, 31 Oct 2013, Erik Steffl wrote:
Date: Thu, 31 Oct 2013 18:44:23 -0700
From: Erik Steffl <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] Rsyslog with RELP not sending/receiving
messages
for long
intervals
We have a fairly simple setup of 6 hosts sending syslog
messages to
one collector host, all of these run rsyslog
7.5.5-0adiscon2
from
adiscon repo and use RELP to transfer messages. There is
also
load
balancer in front of the collector machine but I dont'
think it
matters in this case.
Rsyslog on collector machine is configured to write to
files,
switching to a new file every 15 minute, using config
like this
(abbreviated a bit):
template(name="jsonFilename" type="list") {
constant(value="/path/")
property(name="$now")
constant(value="/")
property(name="$hour")
constant(value="/")
property(name="$qhour")
constant(value="/")
constant(value="log.json")
}
action(type="omfile" DynaFile="jsonFilename"
Template="jsonFormat")
We run a script at every 2, 17, 32, 47 minute of the
hour
and upload
the just finished file to S3. The uploading works like
this:
- let's say it's 3:02:00, rsyslog is writing to
/path/2013-10-10/03/00/log.json
- get the filename log.json (anything that's not
current,
usually
just one previous file which in the example would be
/path/2013-10-10/02/03/log.json)
- rename /path/2013-10-10/02/03/log.json to
/path/2013-10-10/02/03/log.json.uploading.0
- reload rsyslog (to make sure that even if for some
reason
it was
writing to just renamed file it would close it and open
a new
file)
- upload /path/2013-10-10/02/03/log.json.uploading.0
to S3
- remove /path/2013-10-10/02/03/log.json.uploading.0
Here's what happens every third run (yes, regularly EVERY
THIRD RUN)
of this script:
- rsyslog stops writing to the CURRENT file
(/path/2013-10-10/03/00/log.json, the one that is NOT
being
renamed)
few seconds into the run of the script (e.g. 3:02:04)
- 6 hosts that were sending syslog messages to the log
collector
STOP
sending anything (as verified by stracing rsyslogd,
tcpdump
and in
amazon AWS console metric for network in)
- after this nothing is ever written into
/path/2013-10-10/03/00/log.json
- the 6 clients start sending sysog messages again
when the
next
file
is created (in this example it would be
/path/2013-10-10/03/01/log.json)
I checked and double check the files, dates,
verified that
the
current file is not touched but can't figure out what's
going
on. I
tried the script without reload rsyslog but it didn't
make any
difference. If I don't run this script rsyslog works
flawlessly.
any ideas how to troubleshoot this? What could be
causing
the
rsyslog
to stop writing to the file and for the senders to stop
sending
syslog
messages? I assume the rsyslog on the collector host
somehow
signals
to the 6 hosts that send messages that it's not ready or
something...
thanks!
erik
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow
https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are
ARCHIVED
by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE
and DO
NOT
POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow
https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are
ARCHIVED by
a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO
NOT POST
if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow
https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are
ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and
DO
NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are
ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and
DO NOT
POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
