Unfortunately nothing has changed on that end. I know that when we first got this up and running we had to put entries in /etc/hosts for all the servers that were going to log using log4j, I know because I wasn’t psyched about that. ;)
The /etc/hosts entries are still in place and dns hasn’t changed and is still resolving fine. We’ve been looking into this all day and can’t quite track it down. Unfortunately I had never looked into in depth when it was working, I wish I had so I could compare. It seems like what is happening is the app is not reporting a hostname, it’s actually leaving it blank as best as I can tell. Rsyslog is then adding the IP address in. I confirmed that if I remove the -x from the local rsyslog, rsyslog does the dns lookup on the IP and changes it to a hostname and then everything works, it gets passed to our remote rsyslog servers with a hostname and our filtering works as expected. The issue is that we never needed this in the past, we always ran with -x so we are trying to track down what changed. We would prefer to keep running with -x so as not to have the overhead of a dns query for every message logged from this app. I guess if we have to it’s not the end of the world but somehow this was working fine without it previously but now is not. As best as I can tell and from what I’m being told the only thing that changed around the time that this broke was the upgrade of our remote rsyslog servers from v4 to v7 but it just doesn’t seem like that would be related to me. It seems like this info should get set on the local rsyslog servers and then just passed along to the remotes. Any log4j experts on here? ;) On 12/10/13, 12:29 AM, "Edmonds, Alan" <[email protected]> wrote: >i have seen things like this when the client machine's /etc/hosts is >missing an entry for itself (meaning the IP that is being logged), (or >reverse DNS fails to answer). > >Alan >________________________________________ >From: [email protected] >[[email protected]] On Behalf Of Dan Finn >[[email protected]] >Sent: 09 December 2013 23:38 >To: rsyslog-users >Subject: Re: [rsyslog] strange change in filtering since upgrading to v7 > >To answer my own question I believe I have been able to do this with >tcpdump. It would appear that the app is submitting the log entries to >the client rsyslog on port 514 with the IP. For some reason it used to >use the hostname and now it no longer is. Sorry for the wasted 1’s and >0’s. > >Thanks, >Dan > >On 12/9/13, 3:52 PM, "Dan Finn" <[email protected]> wrote: > >>Maybe a dumb question but how would you go about following that chain? >> >>On 12/9/13, 3:22 PM, "David Lang" <[email protected]> wrote: >> >>>The hostname field is supposed to be filled out by the system that's >>>generating >>>the log, if it sends a log without a hostname field, the first machine >>>that >>>receives the log tries to fill it in, with a hostname from reverse DNS >>>if >>>configured, or with the sending IP address if the DNS lookup fails (or >>>is >>>disabled) >>> >>>follow the chain from the originating system on and see where it's >>>picking up >>>the IP address. >>> >>>David Lang >>> >>> >>>On Mon, 9 Dec 2013, Dan Finn wrote: >>> >>>> Date: Mon, 9 Dec 2013 18:43:14 +0000 >>>> From: Dan Finn <[email protected]> >>>> Reply-To: rsyslog-users <[email protected]> >>>> To: rsyslog-users <[email protected]> >>>> Subject: [rsyslog] strange change in filtering since upgrading to v7 >>>> >>>> Remote servers: >>>> CentOS 5.10 >>>> rsyslog-7.4.6-1.el5.centos >>>> >>>> Client servers: >>>> RHEL 5.4 >>>> rsyslog-5.8.6-1.ep >>>> >>>> Remote config : http://pastebin.com/xp5wy02d >>>> Client config: http://pastebin.com/17qYD6WX >>>> >>>> As far as I know nothing has changed on the client side and the only >>>>change that we have made recently to our logging environment is >>>>upgrading our remote servers from v4 to v7 (huge performance >>>>improvement!). It was noticed recently that the filtering on one of >>>>our >>>>apps is no longer working as expected. Instead of the logs getting >>>>written to /var/log/apps/year/hostname/day/hour/jboss.log they are >>>>ending up in /var/log/apps/year/IP address/day/hour/jboss.log. I >>>>personally don¹t think this is related to the v7 upgrade but this is >>>>causing some pain for us with our log scrapers and I¹ve been asked to >>>>verify with the mailing list. >>>> >>>> So far it looks like this is the only application that we have that >>>>this is happening with. This is Jboss and it¹s logging via log4j. All >>>>other apps and OS logs seems to be working just fine. >>>> >>>> Here is a current log entry: >>>> >>>> 2013-12-09T11:00:00-07:00 10.42.30.10 local4: 11:00:00,094 >>>>atgprod1-prod_public_8180 atg-log WARN [BrandCategoryLookupD >>>> roplet] 1414419202 dogfunk >>>>7CE0625028E65F3843D63FE249CD0125.atgprod1-prod_public_8180 >>>>/Store/catalog/brandLanding.jsp?b >>>> >>>>randId=100000630&categoryId=dfCat100434&p=discountPercentUS%3A%5B40+TO+ >>>>* >>>>% >>>>5D%7Csize%3Asmall (http-0.0.0.0-8180-187) Brand >>>> category not found for brand: 100000630 category: dfCat100239 >>>> >>>> And here¹s a log entry from the same server from last week sometime: >>>> >>>> 2013-12-01T23:00:00-07:00 atgprod1 local4: 23:00:00,103 >>>>atgprod1-prod_public_8080 atg-log INFO [ProfileFormHandler] 1387361000 >>>>b678482057 bcs >>>>4C185612E4845821163BF0AEDAE745CE.atgprod1-prod_public_8080 >>>>/Store/account/login.jsp?locale=en_US&_DARGS=/Store/authModal/includes/ >>>>m >>>>o >>>>dalLoginForm.jsp.login-form&_=1385963950864 (http-0.0.0.0-8080-130) >>>>User >>>>Œ[email protected]' attempted to log in, but failed. >>>> >>>> We run rsyslog with the x option on the remote servers. This is how >>>>we were also doing it with v4. I tested removing that flag to enable >>>>dns lookups and it didn¹t seem to make a difference for this issue. >>>> >>>> As I understand it, that info is coming from the header of the log >>>>message and would be getting set on the client side? Is that also >>>>going >>>>to be something that is set by the application when it submits the log >>>>or would the rsyslog client be inserting that? My app guys are saying >>>>that nothing has changed on their side so we really aren¹t sure what >>>>could be causing this change. >>>> >>>> Any insight into what might be causing this would be really >>>>appreciated. >>>> >>>> Thanks, >>>> Dan >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>>>if you DON'T LIKE THAT. >> > >_______________________________________________ >rsyslog mailing list >http://lists.adiscon.net/mailman/listinfo/rsyslog >http://www.rsyslog.com/professional-services/ >What's up with rsyslog? Follow https://twitter.com/rgerhards >NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >DON'T LIKE THAT. >_______________________________________________ >rsyslog mailing list >http://lists.adiscon.net/mailman/listinfo/rsyslog >http://www.rsyslog.com/professional-services/ >What's up with rsyslog? Follow https://twitter.com/rgerhards >NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
