It's configured to deliver over the network so I am pretty sure I am seeing the raw log with tcpdump.
> On Dec 10, 2013, at 1:00 AM, "David Lang" <[email protected]> wrote: > > is log4j delivering to the first rsyslog server over a network connection or > via /dev/log? > > you can log messages with the RSYSLOG_DebugFormat tempate and it will show > you the raw log, what it got from the application, so you can see exactly > what it's working with. > > David Lang > >> On Tue, 10 Dec 2013, Dan Finn wrote: >> >> Date: Tue, 10 Dec 2013 07:41:01 +0000 >> From: Dan Finn <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] strange change in filtering since upgrading to v7 >> Unfortunately nothing has changed on that end. I know that when we first >> got this up and running we had to put entries in /etc/hosts for all the >> servers that were going to log using log4j, I know because I wasn’t >> psyched about that. ;) >> >> The /etc/hosts entries are still in place and dns hasn’t changed and is >> still resolving fine. >> >> We’ve been looking into this all day and can’t quite track it down. >> Unfortunately I had never looked into in depth when it was working, I wish >> I had so I could compare. It seems like what is happening is the app is >> not reporting a hostname, it’s actually leaving it blank as best as I can >> tell. Rsyslog is then adding the IP address in. I confirmed that if I >> remove the -x from the local rsyslog, rsyslog does the dns lookup on the >> IP and changes it to a hostname and then everything works, it gets passed >> to our remote rsyslog servers with a hostname and our filtering works as >> expected. >> >> The issue is that we never needed this in the past, we always ran with -x >> so we are trying to track down what changed. We would prefer to keep >> running with -x so as not to have the overhead of a dns query for every >> message logged from this app. I guess if we have to it’s not the end of >> the world but somehow this was working fine without it previously but now >> is not. >> >> As best as I can tell and from what I’m being told the only thing that >> changed around the time that this broke was the upgrade of our remote >> rsyslog servers from v4 to v7 but it just doesn’t seem like that would be >> related to me. It seems like this info should get set on the local >> rsyslog servers and then just passed along to the remotes. >> >> Any log4j experts on here? ;) >> >>> On 12/10/13, 12:29 AM, "Edmonds, Alan" <[email protected]> wrote: >>> >>> i have seen things like this when the client machine's /etc/hosts is >>> missing an entry for itself (meaning the IP that is being logged), (or >>> reverse DNS fails to answer). >>> >>> Alan >>> ________________________________________ >>> From: [email protected] >>> [[email protected]] On Behalf Of Dan Finn >>> [[email protected]] >>> Sent: 09 December 2013 23:38 >>> To: rsyslog-users >>> Subject: Re: [rsyslog] strange change in filtering since upgrading to v7 >>> >>> To answer my own question I believe I have been able to do this with >>> tcpdump. It would appear that the app is submitting the log entries to >>> the client rsyslog on port 514 with the IP. For some reason it used to >>> use the hostname and now it no longer is. Sorry for the wasted 1’s and >>> 0’s. >>> >>> Thanks, >>> Dan >>> >>>> On 12/9/13, 3:52 PM, "Dan Finn" <[email protected]> wrote: >>>> >>>> Maybe a dumb question but how would you go about following that chain? >>>> >>>>> On 12/9/13, 3:22 PM, "David Lang" <[email protected]> wrote: >>>>> >>>>> The hostname field is supposed to be filled out by the system that's >>>>> generating >>>>> the log, if it sends a log without a hostname field, the first machine >>>>> that >>>>> receives the log tries to fill it in, with a hostname from reverse DNS >>>>> if >>>>> configured, or with the sending IP address if the DNS lookup fails (or >>>>> is >>>>> disabled) >>>>> >>>>> follow the chain from the originating system on and see where it's >>>>> picking up >>>>> the IP address. >>>>> >>>>> David Lang >>>>> >>>>> >>>>>> On Mon, 9 Dec 2013, Dan Finn wrote: >>>>>> >>>>>> Date: Mon, 9 Dec 2013 18:43:14 +0000 >>>>>> From: Dan Finn <[email protected]> >>>>>> Reply-To: rsyslog-users <[email protected]> >>>>>> To: rsyslog-users <[email protected]> >>>>>> Subject: [rsyslog] strange change in filtering since upgrading to v7 >>>>>> >>>>>> Remote servers: >>>>>> CentOS 5.10 >>>>>> rsyslog-7.4.6-1.el5.centos >>>>>> >>>>>> Client servers: >>>>>> RHEL 5.4 >>>>>> rsyslog-5.8.6-1.ep >>>>>> >>>>>> Remote config : http://pastebin.com/xp5wy02d >>>>>> Client config: http://pastebin.com/17qYD6WX >>>>>> >>>>>> As far as I know nothing has changed on the client side and the only >>>>>> change that we have made recently to our logging environment is >>>>>> upgrading our remote servers from v4 to v7 (huge performance >>>>>> improvement!). It was noticed recently that the filtering on one of >>>>>> our >>>>>> apps is no longer working as expected. Instead of the logs getting >>>>>> written to /var/log/apps/year/hostname/day/hour/jboss.log they are >>>>>> ending up in /var/log/apps/year/IP address/day/hour/jboss.log. I >>>>>> personally don¹t think this is related to the v7 upgrade but this is >>>>>> causing some pain for us with our log scrapers and I¹ve been asked to >>>>>> verify with the mailing list. >>>>>> >>>>>> So far it looks like this is the only application that we have that >>>>>> this is happening with. This is Jboss and it¹s logging via log4j. All >>>>>> other apps and OS logs seems to be working just fine. >>>>>> >>>>>> Here is a current log entry: >>>>>> >>>>>> 2013-12-09T11:00:00-07:00 10.42.30.10 local4: 11:00:00,094 >>>>>> atgprod1-prod_public_8180 atg-log WARN [BrandCategoryLookupD >>>>>> roplet] 1414419202 dogfunk >>>>>> 7CE0625028E65F3843D63FE249CD0125.atgprod1-prod_public_8180 >>>>>> /Store/catalog/brandLanding.jsp?b >>>>>> >>>>>> randId=100000630&categoryId=dfCat100434&p=discountPercentUS%3A%5B40+TO+ >>>>>> * >>>>>> % >>>>>> 5D%7Csize%3Asmall (http-0.0.0.0-8180-187) Brand >>>>>> category not found for brand: 100000630 category: dfCat100239 >>>>>> >>>>>> And here¹s a log entry from the same server from last week sometime: >>>>>> >>>>>> 2013-12-01T23:00:00-07:00 atgprod1 local4: 23:00:00,103 >>>>>> atgprod1-prod_public_8080 atg-log INFO [ProfileFormHandler] 1387361000 >>>>>> b678482057 bcs >>>>>> 4C185612E4845821163BF0AEDAE745CE.atgprod1-prod_public_8080 >>>>>> /Store/account/login.jsp?locale=en_US&_DARGS=/Store/authModal/includes/ >>>>>> m >>>>>> o >>>>>> dalLoginForm.jsp.login-form&_=1385963950864 (http-0.0.0.0-8080-130) >>>>>> User >>>>>> Œ[email protected]' attempted to log in, but failed. >>>>>> >>>>>> We run rsyslog with the x option on the remote servers. This is how >>>>>> we were also doing it with v4. I tested removing that flag to enable >>>>>> dns lookups and it didn¹t seem to make a difference for this issue. >>>>>> >>>>>> As I understand it, that info is coming from the header of the log >>>>>> message and would be getting set on the client side? Is that also >>>>>> going >>>>>> to be something that is set by the application when it submits the log >>>>>> or would the rsyslog client be inserting that? My app guys are saying >>>>>> that nothing has changed on their side so we really aren¹t sure what >>>>>> could be causing this change. >>>>>> >>>>>> Any insight into what might be causing this would be really >>>>>> appreciated. >>>>>> >>>>>> Thanks, >>>>>> Dan >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>>>>> if you DON'T LIKE THAT. >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
