is log4j delivering to the first rsyslog server over a network connection or via
/dev/log?
you can log messages with the RSYSLOG_DebugFormat tempate and it will show you
the raw log, what it got from the application, so you can see exactly what it's
working with.
David Lang
On Tue, 10 Dec 2013, Dan Finn wrote:
Date: Tue, 10 Dec 2013 07:41:01 +0000
From: Dan Finn <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] strange change in filtering since upgrading to v7
Unfortunately nothing has changed on that end. I know that when we first
got this up and running we had to put entries in /etc/hosts for all the
servers that were going to log using log4j, I know because I wasn’t
psyched about that. ;)
The /etc/hosts entries are still in place and dns hasn’t changed and is
still resolving fine.
We’ve been looking into this all day and can’t quite track it down.
Unfortunately I had never looked into in depth when it was working, I wish
I had so I could compare. It seems like what is happening is the app is
not reporting a hostname, it’s actually leaving it blank as best as I can
tell. Rsyslog is then adding the IP address in. I confirmed that if I
remove the -x from the local rsyslog, rsyslog does the dns lookup on the
IP and changes it to a hostname and then everything works, it gets passed
to our remote rsyslog servers with a hostname and our filtering works as
expected.
The issue is that we never needed this in the past, we always ran with -x
so we are trying to track down what changed. We would prefer to keep
running with -x so as not to have the overhead of a dns query for every
message logged from this app. I guess if we have to it’s not the end of
the world but somehow this was working fine without it previously but now
is not.
As best as I can tell and from what I’m being told the only thing that
changed around the time that this broke was the upgrade of our remote
rsyslog servers from v4 to v7 but it just doesn’t seem like that would be
related to me. It seems like this info should get set on the local
rsyslog servers and then just passed along to the remotes.
Any log4j experts on here? ;)
On 12/10/13, 12:29 AM, "Edmonds, Alan" <[email protected]> wrote:
i have seen things like this when the client machine's /etc/hosts is
missing an entry for itself (meaning the IP that is being logged), (or
reverse DNS fails to answer).
Alan
________________________________________
From: [email protected]
[[email protected]] On Behalf Of Dan Finn
[[email protected]]
Sent: 09 December 2013 23:38
To: rsyslog-users
Subject: Re: [rsyslog] strange change in filtering since upgrading to v7
To answer my own question I believe I have been able to do this with
tcpdump. It would appear that the app is submitting the log entries to
the client rsyslog on port 514 with the IP. For some reason it used to
use the hostname and now it no longer is. Sorry for the wasted 1’s and
0’s.
Thanks,
Dan
On 12/9/13, 3:52 PM, "Dan Finn" <[email protected]> wrote:
Maybe a dumb question but how would you go about following that chain?
On 12/9/13, 3:22 PM, "David Lang" <[email protected]> wrote:
The hostname field is supposed to be filled out by the system that's
generating
the log, if it sends a log without a hostname field, the first machine
that
receives the log tries to fill it in, with a hostname from reverse DNS
if
configured, or with the sending IP address if the DNS lookup fails (or
is
disabled)
follow the chain from the originating system on and see where it's
picking up
the IP address.
David Lang
On Mon, 9 Dec 2013, Dan Finn wrote:
Date: Mon, 9 Dec 2013 18:43:14 +0000
From: Dan Finn <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] strange change in filtering since upgrading to v7
Remote servers:
CentOS 5.10
rsyslog-7.4.6-1.el5.centos
Client servers:
RHEL 5.4
rsyslog-5.8.6-1.ep
Remote config : http://pastebin.com/xp5wy02d
Client config: http://pastebin.com/17qYD6WX
As far as I know nothing has changed on the client side and the only
change that we have made recently to our logging environment is
upgrading our remote servers from v4 to v7 (huge performance
improvement!). It was noticed recently that the filtering on one of
our
apps is no longer working as expected. Instead of the logs getting
written to /var/log/apps/year/hostname/day/hour/jboss.log they are
ending up in /var/log/apps/year/IP address/day/hour/jboss.log. I
personally don¹t think this is related to the v7 upgrade but this is
causing some pain for us with our log scrapers and I¹ve been asked to
verify with the mailing list.
So far it looks like this is the only application that we have that
this is happening with. This is Jboss and it¹s logging via log4j. All
other apps and OS logs seems to be working just fine.
Here is a current log entry:
2013-12-09T11:00:00-07:00 10.42.30.10 local4: 11:00:00,094
atgprod1-prod_public_8180 atg-log WARN [BrandCategoryLookupD
roplet] 1414419202 dogfunk
7CE0625028E65F3843D63FE249CD0125.atgprod1-prod_public_8180
/Store/catalog/brandLanding.jsp?b
randId=100000630&categoryId=dfCat100434&p=discountPercentUS%3A%5B40+TO+
*
%
5D%7Csize%3Asmall (http-0.0.0.0-8180-187) Brand
category not found for brand: 100000630 category: dfCat100239
And here¹s a log entry from the same server from last week sometime:
2013-12-01T23:00:00-07:00 atgprod1 local4: 23:00:00,103
atgprod1-prod_public_8080 atg-log INFO [ProfileFormHandler] 1387361000
b678482057 bcs
4C185612E4845821163BF0AEDAE745CE.atgprod1-prod_public_8080
/Store/account/login.jsp?locale=en_US&_DARGS=/Store/authModal/includes/
m
o
dalLoginForm.jsp.login-form&_=1385963950864 (http-0.0.0.0-8080-130)
User
Œ[email protected]' attempted to log in, but failed.
We run rsyslog with the x option on the remote servers. This is how
we were also doing it with v4. I tested removing that flag to enable
dns lookups and it didn¹t seem to make a difference for this issue.
As I understand it, that info is coming from the header of the log
message and would be getting set on the client side? Is that also
going
to be something that is set by the application when it submits the log
or would the rsyslog client be inserting that? My app guys are saying
that nothing has changed on their side so we really aren¹t sure what
could be causing this change.
Any insight into what might be causing this would be really
appreciated.
Thanks,
Dan
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
