On Mon, 17 Dec 2012, Rainer Gerhards wrote:
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Rainer Gerhards
Sent: Monday, December 17, 2012 6:27 PM
To: rsyslog-users
Subject: Re: [rsyslog] imfile and omudpspoof
Just a quick follow-up: I currently work with Rick on this issue.
It
looks like there are actually two issues in one:
1) in v5, it can happen that an action in front of an included
get's
reordered, and this seems to have happened here. This is actually the
second time in many years that I see that. I won't fix that in v5, as
it would probably lead to something like the v7 engine ;) Rick has
moved to v7.
2) omudpspoof discards messages that are larger than the network-
supported MTU. I am working to fragment them. Note that I know that
some receivers have problems with fragmented UDP, but hey, what else
shall we do? Messages over 64K can never be transmitted in a single
UDP
packet, and as such 7.2.5 will now truncate them to 64K. I will
probably change v5 so that >1472 byte messages get truncated, but the
message sent instead of being discarded.
Will keep you posted when there is notable news (but maybe you just
see a new release announcement if everything works out and no new
problems pop up).
I would suggest making the default spoof template do the truncation,
so
that
someone who's system can handle larger messages can set it longer.
Side-note: v7 get's a "mtu" action parameter, so that folks with larger mtu's
can avoid fragmentation (right now integrating that one ;)). Just the default is kept at
1500.
This sounds reasonable.
David Lang
Rainer
That's rather complicated, because the template system can limit fields
but not the overall string.
If the consensus is that this is a bad change, I'll revert the change
and keep the old behavior. I just thought I solve this hard-to-find-
and-debug issue, but I don't want to invest any real time in v5 (I
don't even know if I ever will do a new release...).
What do you say in the light of this?
Rainer
David Lang
Rainer
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Rick Brown
Sent: Friday, December 14, 2012 3:24 PM
To: rsyslog-users
Subject: Re: [rsyslog] imfile and omudpspoof
----- Original Message -----
From: "Rainer Gerhards" <[email protected]>
To: "rsyslog-users" <[email protected]>
Sent: Friday, December 14, 2012 9:06:10 AM
Subject: Re: [rsyslog] imfile and omudpspoof
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Rick Brown
Sent: Friday, December 14, 2012 3:04 PM
To: rsyslog-users
Subject: Re: [rsyslog] imfile and omudpspoof
----- Original Message -----
From: "Rainer Gerhards" <[email protected]>
To: "rsyslog-users" <[email protected]>
Sent: Friday, December 14, 2012 2:40:05 AM
Subject: Re: [rsyslog] imfile and omudpspoof
-----Original Message-----
From: Rick Brown [mailto:[email protected]]
Sent: Thursday, December 13, 2012 7:06 PM
To: Rainer Gerhards
Subject: Re: [rsyslog] imfile and omudpspoof
You're right, it's rather large. I gzipp'd the output and
posted
it
at xxxx
Mhhh... That log does not include anything from imfile. Did you
accidently capture the wrong system?
I captured from the syslog server in the middle. Let me try to
be
clearer in stating the problem.
Client A scrapes log files with imfile and sends them to my
syslog
server. The syslog server is configured to forward the messages
to
a
SIEM using imudpspoof. The syslog server successfully writes
the
logs
to disk, however not all messages are being spoofed and sent on.
The
person who runs the SIEM noted the difference and I then
observed
tcpdumps on the syslog server. Watching in two terminals:
tcpdump host A and host syslog
and:
tcpdump host syslog and host SIEM
I see many more messages in the top terminal than the bottom.
On
first
glance, it looked like none of the messages generated by imfile
on
client A were making it to the SIEM, but now that I've stared at
it
long enough, I do occassionally see a couple go by. If I
restarted
rsyslog while watching the tcpdumps, I see a burst of the imfile
messages get spoofed, then they stop for a while, then a few
will
trickle by.
OK - I misunderstood the initial description. So let's continue
with
that debug log file. Can you tell me which logs lines are
imfile-originated (and not delivered) and which one are not?
Rainer
Ahh, good. All of the imfile messages are set to facility news
and
we
no longer actually run news servers, so it's a safe bet that any
news
messages is generated from imfile. I've been particularly
concentrating on a host cas1, who's ip ends in .165.151, as it
generates quite a bit of application logs that I scrape with
imfile.
And the spoof target address ends in .165.95.
From what I have seen, it may also happen that the debug log
info
is
not sufficient. Just treat this as advance warning ;)
We can probably get around a limitation by creating an
additional
log
file, so that I can see all message properties. Please add
*.* /var/log/rsyslog-props.log;RSYSLOG_DebugFormat
To your config (the file name is obviously irrelevant, but the
template is important!).
Make sure that the debug log and the property file are from the
same
run.
Rainer
Can do. I'll go look into this now.
I see that some of the imfile generated messages are making it
through
to the SIEM from time to time, but the vast majority are being
dropped.
I have to wonder if I'm overflowing a buffer in the output
modules.
Thanks for taking a look!
----- Original Message -----
From: "Rainer Gerhards" <[email protected]>
To: "rsyslog-users" <[email protected]>
Sent: Thursday, December 13, 2012 10:05:02 AM
Subject: Re: [rsyslog] imfile and omudpspoof
Can you post a (sufficiently large or complete) debug log
that
exposes the problem? The list will probably reject it, so it
is
best
to put it on something like pastbin.
Rainer
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Rick Brown
Sent: Thursday, December 13, 2012 4:01 PM
To: rsyslog-users
Subject: Re: [rsyslog] imfile and omudpspoof
On Tue, 11 Dec 2012, Rick Brown wrote:
I use imfile to gather application logs such as
apache,
tomcat,
php, etc. and send those on to my syslog server along
with
the
client machines normal syslog traffic. My syslog
server
then
dutifully writes all the messages locally and
additionally
forwards the messages on to a SIEM product via
omudpspoof.
Watching packet captures, I can see some messages are
being
spoofed
and sent on to the SIEM, but some are not. At first
glance,
it
appears that all regular syslog messages that are
generated
on
the
client are being spoofed and sent on to the SIEM, but
most,
if
not
all messages generated via imfile on the client are
not
being
spoofed and sent on to the SIEM at all.
I've tried the standard
*.* :omudspoof:
as well as
$template spooftemplate,"%fromhost-ip% %rawmsg%"
*.* :omudpspoof:;spooftemplate
and
$template spooftemplate,"%rawmsg%"
*.* :omudpspoof:;spooftemplate
All with the same effect. Am I missing something
here?
Is
anyone
else doing similar, or seen similar behavior?
For the record, I'm running a patched version of
5.8.11.
The
patch,
now that I'm reading it again, was to protect against
calling
more
than one instance of libnet code in the omudpspoof
module.
the udpspoof module grabs the first field from the
template
and
uses
that as the
IP address to spoof from. What is the fromhost-ip when
you
get
the
files from
imfile?
Ahh, you may be onto something there.. the messages
coming
into my
syslog server have just the hostname in it, not FQDN, and
not
IP
address. The few that I see leaving for the SIEM have
the
IP
address in that field of the message. Now to figure out
why
they're different.
Hrm. Upon inspection, all logs are coming in with just
the
hostname,
so that's a dead end. All normal syslogs (like cron.info,
kernel.notice, daemon.debug, etc.) I can see are being
spoofed
and
sent
on to the SIEM, but the logs generated by imfile (all
configured as
news.info) from client machines get logged on the syslog
server,
but
aren't being spoofed and sent on to the SIEM.
--
Rick Brown
Office of Information Technology
Georgia Institute of Technology
___________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
--
Rick Brown
Office of Information Technology
Georgia Institute of Technology
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
