Hello! Just a reminder that a hostname field in a syslog message is just a string sent from sender to collector. So you can craft a custom template with the hostname field defined as you'd like. Though I'd call this a "fallback" way of fixing the issue. The right way is to set the proper hostname on a sender system before rsyslog starts I'd say.
On Wed, 17 Nov 2021 at 08:33, David Lang via rsyslog < [email protected]> wrote: > Rsyslog looks up the hostname as it starts up, so if something after > rsyslog > starts changes the hostname, rsyslog isn't going to notice until you > restart > rsyslog. > > again, fromhost is a receiver side lookup of the name to match > fromhost-ip, so > if hostname is getting set correctly, filter on that instead of on > fromhost. > > David Lang > > On Tue, 16 Nov 2021, Scott Slattery wrote: > > > Date: Tue, 16 Nov 2021 17:28:15 -0700 > > From: Scott Slattery <[email protected]> > > To: David Lang <[email protected]> > > Cc: Scott Slattery via rsyslog <[email protected]> > > Subject: Re: [rsyslog] FROMHOST missing on central log collector > > > > Thanks, David, I think you've done more than enough to try and help me on > > this. I need to do some reading on Amazon (and the link you shared) to > see > > what my options are. I agree with you, it's likely workable. > > > > I've confirmed that the results from the 'hostname' command do match so > > it's a bit of a mystery why rsyslog doesn't detect this but, i think > you're > > on the right track, we need to run a post-deployment script to get these > > instances registered in Route53. > > > > > > *Scott Slattery* > > > > *Sr. Enterprise/Cloud Architect* > > > > *Cloud, Compute, Information & Architecture Team* > > > > motorolasolutions.com > > > > *O: 602.529.8226* > > > > *E*: [email protected] > > > > > > > > > > On Tue, Nov 16, 2021 at 5:20 PM David Lang <[email protected]> wrote: > > > >> if you login to one of the systems, you should find that the name > returned > >> by > >> the hostname command should match what you get in the syslog message > that > >> is > >> delivered to your central collector. (if it doesn't, try restarting > >> rsyslog and > >> see if it changes to match) > >> > >> then the question becomes what mechansims does AMI provide for > customizing > >> the > >> hostname > >> > >> a quick google search shows a new hostnamectl command > >> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_AWSEC2_latest_UserGuide_set-2Dhostname.html&d=DwIBAg&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=1JiTvUcvjB8RxLP9vfPbVSsbAQyitkPK6AzRBhEBUjRBWl-3tAtfR73TCtIFhdHZ&s=WR-Pz8svN0d8vqg4ZKSNj2dbxtcngaMJ4iiRXCPpD6c&e= > >> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cyberciti.biz_faq_set-2Dchange-2Dhostname-2Din-2Damazon-2Dlinux-2Dec2-2Dinstance-2Dserver_&d=DwIBAg&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=1JiTvUcvjB8RxLP9vfPbVSsbAQyitkPK6AzRBhEBUjRBWl-3tAtfR73TCtIFhdHZ&s=2RI1Khq-fBEBJxckXk9nWDESN8pTJxGiGv6xpsbYhzE&e= > >> > >> I know there is a way for you to specify a script to run when an > instance > >> is > >> started, that script can then set things like this. I don't know enough > to > >> point > >> you at specifically how to do that. > >> > >> David Lang > >> > >> > >> On Tue, 16 Nov 2021, Scott Slattery wrote: > >> > >>> Date: Tue, 16 Nov 2021 17:07:47 -0700 > >>> From: Scott Slattery <[email protected]> > >>> To: David Lang <[email protected]> > >>> Cc: Scott Slattery via rsyslog <[email protected]> > >>> Subject: Re: [rsyslog] FROMHOST missing on central log collector > >>> > >>> Thanks David, the hostname is currently set in the AMI (Amazon Master > >>> Image) which is the source image for all instances that are dynamically > >>> created and I can verify that, if you login to one of these dynamic > >>> instances, the hostname is in fact set correctly. > >>> > >>> The issue doesn't seem particularly related to what is set in > >>> /etc/hostname, /etc/hosts, or what was set using 'hostname' command. I > >>> think you can see this is the source of my frustration. It appears the > >>> central log collector relies only on DNS resolution unless there's some > >>> hidden magic inside RSYSLOG to force the sent logs to include a host > >> header > >>> (vs DNS). > >>> > >>> I don't want to continue wasting your time but again, it is much > >>> appreciated. I'll look into some way of dynamically adding these hosts > to > >>> DNS in AWS Route53. It appears rsyslog simply can't do what I'm after. > >>> > >>> > >>> *Scott Slattery* > >>> > >>> *Sr. Enterprise/Cloud Architect* > >>> > >>> *Cloud, Compute, Information & Architecture Team* > >>> > >>> motorolasolutions.com > >>> > >>> *O: 602.529.8226* > >>> > >>> *E*: [email protected] > >>> > >>> > >>> > >>> > >>> On Tue, Nov 16, 2021 at 5:02 PM David Lang <[email protected]> wrote: > >>> > >>>> the hostname command will let you set the hostname (you want to do > that > >>>> before > >>>> you start rsyslog). I would expect that the orcastration tool you use > to > >>>> create > >>>> the systems will have some 'correct for that tool' way to set the > >> hostname > >>>> as it > >>>> starts the instance (sorry I can't provide more specifics, if you can > >>>> mention > >>>> what you are using, possibly someone else can chime in on the best way > >> to > >>>> set > >>>> the hostname with that tool) > >>>> > >>>> David Lang > >>>> > >>>> On Tue, 16 Nov 2021, Scott Slattery wrote: > >>>> > >>>>> Date: Tue, 16 Nov 2021 16:59:17 -0700 > >>>>> From: Scott Slattery <[email protected]> > >>>>> To: David Lang <[email protected]> > >>>>> Cc: Scott Slattery via rsyslog <[email protected]> > >>>>> Subject: Re: [rsyslog] FROMHOST missing on central log collector > >>>>> > >>>>> My follow-on question woudl be how do I set the hostname at the > client > >>>> end? > >>>>> Other than what's in /etc/hosts, /etc/hostname, etc. I don't know how > >>>> else > >>>>> I would affect the log being sent to ensure it's going over. > >>>>> > >>>>> *Scott Slattery* > >>>>> > >>>>> *Sr. Enterprise/Cloud Architect* > >>>>> > >>>>> *Cloud, Compute, Information & Architecture Team* > >>>>> > >>>>> motorolasolutions.com > >>>>> > >>>>> *O: 602.529.8226* > >>>>> > >>>>> *E*: [email protected] > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Tue, Nov 16, 2021 at 4:55 PM David Lang <[email protected]> wrote: > >>>>> > >>>>>> the translation from fromhost-ip to fromhost is done at the > collector, > >>>> but > >>>>>> the > >>>>>> sender sets the hostname field. If you can trust that hostname was > set > >>>>>> correctly, there is no reason to use fromhost > >>>>>> > >>>>>> David Lang > >>>>>> > >>>>>> On Tue, 16 Nov 2021, Scott Slattery wrote: > >>>>>> > >>>>>>> Date: Tue, 16 Nov 2021 16:53:19 -0700 > >>>>>>> From: Scott Slattery <[email protected]> > >>>>>>> To: David Lang <[email protected]> > >>>>>>> Cc: Scott Slattery via rsyslog <[email protected]> > >>>>>>> Subject: Re: [rsyslog] FROMHOST missing on central log collector > >>>>>>> > >>>>>>> Thanks David, I could be wrong but the resolution seems to be > >> happening > >>>>>> at > >>>>>>> the log collection server, not the client end. Given this, I'm not > >> sure > >>>>>>> anything outside of rsyslog on the client would affect what the > >>>> receiving > >>>>>>> collection server is seeing. > >>>>>>> > >>>>>>> My hope was that this could be affected by RSYSLOG on the client > >> device > >>>>>> but > >>>>>>> perhaps not. I'll also look into AWS to see if a dynamically > created > >>>>>>> compute resource can automatically be registered with DNS. > >>>>>>> > >>>>>>> If anything else comes to mind, let me know. As always, I > appreciate > >>>> your > >>>>>>> feedback. > >>>>>>> > >>>>>>> *Scott Slattery* > >>>>>>> > >>>>>>> *Sr. Enterprise/Cloud Architect* > >>>>>>> > >>>>>>> *Cloud, Compute, Information & Architecture Team* > >>>>>>> > >>>>>>> motorolasolutions.com > >>>>>>> > >>>>>>> *O: 602.529.8226* > >>>>>>> > >>>>>>> *E*: [email protected] > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Nov 16, 2021 at 4:37 PM David Lang <[email protected]> wrote: > >>>>>>> > >>>>>>>> Linux has a rather sophisticated mechanism for plugging in > arbitrary > >>>>>> ways > >>>>>>>> of > >>>>>>>> doing name resolution. DNS has 'won' but hitorically there have > been > >>>>>> many > >>>>>>>> other > >>>>>>>> options. Research nsswitch (/etc/nsswitch.conf) and see if there > is > >>>>>>>> something > >>>>>>>> that you can leverage. > >>>>>>>> > >>>>>>>> or, if you can set the hostname of the resources as they are > created > >>>> to > >>>>>> be > >>>>>>>> some > >>>>>>>> predicatable pattern rather than the AWS default of IP based, you > >> can > >>>>>> then > >>>>>>>> make > >>>>>>>> your logic use that. (This is the approach I would look into). > What > >>>>>>>> mechanism > >>>>>>>> this will be will depend on how you are configuring/provisioning > the > >>>>>>>> systems. > >>>>>>>> > >>>>>>>> David Lang > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Tue, 16 Nov 2021, Scott Slattery wrote: > >>>>>>>> > >>>>>>>>> Date: Tue, 16 Nov 2021 15:14:51 -0700 > >>>>>>>>> From: Scott Slattery <[email protected]> > >>>>>>>>> To: David Lang <[email protected]> > >>>>>>>>> Cc: Scott Slattery via rsyslog <[email protected]> > >>>>>>>>> Subject: Re: [rsyslog] FROMHOST missing on central log collector > >>>>>>>>> > >>>>>>>>> Thanks, David, I was hoping this was possible. Since the compute > >>>>>>>> resources > >>>>>>>>> are dynamic, using any sort of local /etc/hosts would be > impossible > >>>>>> since > >>>>>>>>> the IP are unpredictable. Can you point me to how I would do this > >> on > >>>>>> the > >>>>>>>>> client-server? > >>>>>>>>> > >>>>>>>>> Thanks > >>>>>>>>> > >>>>>>>>> *Scott Slattery* > >>>>>>>>> > >>>>>>>>> *Sr. Enterprise/Cloud Architect* > >>>>>>>>> > >>>>>>>>> *Cloud, Compute, Information & Architecture Team* > >>>>>>>>> > >>>>>>>>> motorolasolutions.com > >>>>>>>>> > >>>>>>>>> *O: 602.529.8226* > >>>>>>>>> > >>>>>>>>> *E*: [email protected] > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Tue, Nov 16, 2021 at 2:59 PM David Lang <[email protected]> > wrote: > >>>>>>>>> > >>>>>>>>>> fromhost is the result of a name lookup of fromhost-ip. On the > >>>>>> receiver, > >>>>>>>>>> you can > >>>>>>>>>> control this with your name resolution (DNS, /etc/hosts, other > >>>>>>>> mechanisms) > >>>>>>>>>> > >>>>>>>>>> but a better option would probably be to set the hostname on the > >>>>>> sender. > >>>>>>>>>> The > >>>>>>>>>> hostname field in the message is under the full control of the > >>>> sender. > >>>>>>>>>> > >>>>>>>>>> David Lang > >>>>>>>>>> > >>>>>>>>>> On Tue, 16 Nov 2021, Scott Slattery via rsyslog wrote: > >>>>>>>>>> > >>>>>>>>>>> Date: Tue, 16 Nov 2021 14:56:09 -0700 > >>>>>>>>>>> From: Scott Slattery via rsyslog <[email protected]> > >>>>>>>>>>> To: rsyslog-users <[email protected]> > >>>>>>>>>>> Cc: Scott Slattery <[email protected]> > >>>>>>>>>>> Subject: [rsyslog] FROMHOST missing on central log collector > >>>>>>>>>>> > >>>>>>>>>>> Hello, > >>>>>>>>>>> > >>>>>>>>>>> I have a central log server, many of them, using rsyslog to > >>>> aggregate > >>>>>>>>>> logs > >>>>>>>>>>> from remote servers. Everything works great but I have a new > >>>>>> challenge > >>>>>>>>>> and > >>>>>>>>>>> am hoping for some recommendations. > >>>>>>>>>>> > >>>>>>>>>>> I have a number of AWS auto-scaling groups where compute > >> resources > >>>>>> are > >>>>>>>>>>> dynamically scaled up and down. Each of these will have a > custom > >>>>>>>> rsyslog > >>>>>>>>>>> configuration pulled from the AWS AMI. > >>>>>>>>>>> > >>>>>>>>>>> These dynamic resources are not added to DNS due to their > dynamic > >>>>>>>> nature > >>>>>>>>>> so > >>>>>>>>>>> they will not have DNS assigned FQDNs. > >>>>>>>>>>> > >>>>>>>>>>> Because of the lack of a hostname, my central log server is > >> getting > >>>>>>>> only > >>>>>>>>>>> IP. I aggregate based on FROMHOST-FROMHOST-IP. > >>>>>>>>>>> > >>>>>>>>>>> So what I'm seeing today looks like '10.38.134.77-10.38.134.77' > >>>>>> where I > >>>>>>>>>>> want to see ause1oagbtst03.mydomain.com-10.41.102.168 > >>>>>>>>>>> > >>>>>>>>>>> What I'd want to do is have easy resource send using the same > >>>>>> hostname > >>>>>>>>>> and > >>>>>>>>>>> current IP. This later will allow me to aggregate all resources > >> by > >>>>>>>> name. > >>>>>>>>>>> > >>>>>>>>>>> I did not see any way of affecting the FROMHOST information > >> unless, > >>>>>> on > >>>>>>>>>> the > >>>>>>>>>>> collector, I have rules based on IP address which isn't optimal > >>>> given > >>>>>>>> the > >>>>>>>>>>> dynamic nature of the IPs changing. > >>>>>>>>>>> > >>>>>>>>>>> Any suggestion is appreciated. > >>>>>>>>>>> > >>>>>>>>>>> *Scott Slattery* > >>>>>>>>>>> > >>>>>>>>>>> *Sr. Enterprise/Cloud Architect* > >>>>>>>>>>> > >>>>>>>>>>> *Cloud, Compute, Information & Architecture Team* > >>>>>>>>>>> > >>>>>>>>>>> motorolasolutions.com > >>>>>>>>>>> > >>>>>>>>>>> *O: 602.529.8226* > >>>>>>>>>>> > >>>>>>>>>>> *E*: [email protected] > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>> > >>> > >> > > > > > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- Yury Bushmelev _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
