Thanks for your feedback. There seems to be some understanding that the hostname is not set properly on the client-side. This is not the case, the hostname displays properly on the host itself and is also properly configured from a linux perspective. This is precisely why I'm inquiring about alternatives. The only differentiating factor with respect to these dynamically created hosts is that they do not get registered in DNS since their life is, or can be, quite short based on computing demand.
I was under the impression that the hostname used by the server-side (collector) was the result of a server-side DNS lookup, which will not resolve for these hosts. This is why I was looking for a rsyslog solution that didn't involved DNS. Yuri, if I understand you correctly you're saying a custom template using HOSTNAME vs FROMHOST-IP may be an option. I'll look into this to understand it better. Thanks for this suggestion, it sound like it completely removes the DNS constraint. I'll give it a try. *Scott Slattery* *Sr. Enterprise/Cloud Architect* *Cloud, Compute, Information & Architecture Team* motorolasolutions.com *O: 602.529.8226* *E*: [email protected] On Tue, Nov 16, 2021 at 7:15 PM Yuri Bushmelev <[email protected]> wrote: > Hello! > > Just a reminder that a hostname field in a syslog message is just a string > sent from sender to collector. So you can craft a custom template with the > hostname field defined as you'd like. Though I'd call this a "fallback" way > of fixing the issue. The right way is to set the proper hostname on a > sender system before rsyslog starts I'd say. > > On Wed, 17 Nov 2021 at 08:33, David Lang via rsyslog < > [email protected]> wrote: > >> Rsyslog looks up the hostname as it starts up, so if something after >> rsyslog >> starts changes the hostname, rsyslog isn't going to notice until you >> restart >> rsyslog. >> >> again, fromhost is a receiver side lookup of the name to match >> fromhost-ip, so >> if hostname is getting set correctly, filter on that instead of on >> fromhost. >> >> David Lang >> >> On Tue, 16 Nov 2021, Scott Slattery wrote: >> >> > Date: Tue, 16 Nov 2021 17:28:15 -0700 >> > From: Scott Slattery <[email protected]> >> > To: David Lang <[email protected]> >> > Cc: Scott Slattery via rsyslog <[email protected]> >> > Subject: Re: [rsyslog] FROMHOST missing on central log collector >> > >> > Thanks, David, I think you've done more than enough to try and help me >> on >> > this. I need to do some reading on Amazon (and the link you shared) to >> see >> > what my options are. I agree with you, it's likely workable. >> > >> > I've confirmed that the results from the 'hostname' command do match so >> > it's a bit of a mystery why rsyslog doesn't detect this but, i think >> you're >> > on the right track, we need to run a post-deployment script to get these >> > instances registered in Route53. >> > >> > >> > *Scott Slattery* >> > >> > *Sr. Enterprise/Cloud Architect* >> > >> > *Cloud, Compute, Information & Architecture Team* >> > >> > motorolasolutions.com >> > >> > *O: 602.529.8226* >> > >> > *E*: [email protected] >> > >> > >> > >> > >> > On Tue, Nov 16, 2021 at 5:20 PM David Lang <[email protected]> wrote: >> > >> >> if you login to one of the systems, you should find that the name >> returned >> >> by >> >> the hostname command should match what you get in the syslog message >> that >> >> is >> >> delivered to your central collector. (if it doesn't, try restarting >> >> rsyslog and >> >> see if it changes to match) >> >> >> >> then the question becomes what mechansims does AMI provide for >> customizing >> >> the >> >> hostname >> >> >> >> a quick google search shows a new hostnamectl command >> >> >> >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_AWSEC2_latest_UserGuide_set-2Dhostname.html&d=DwIBAg&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=1JiTvUcvjB8RxLP9vfPbVSsbAQyitkPK6AzRBhEBUjRBWl-3tAtfR73TCtIFhdHZ&s=WR-Pz8svN0d8vqg4ZKSNj2dbxtcngaMJ4iiRXCPpD6c&e= >> >> >> >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cyberciti.biz_faq_set-2Dchange-2Dhostname-2Din-2Damazon-2Dlinux-2Dec2-2Dinstance-2Dserver_&d=DwIBAg&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=1JiTvUcvjB8RxLP9vfPbVSsbAQyitkPK6AzRBhEBUjRBWl-3tAtfR73TCtIFhdHZ&s=2RI1Khq-fBEBJxckXk9nWDESN8pTJxGiGv6xpsbYhzE&e= >> >> >> >> I know there is a way for you to specify a script to run when an >> instance >> >> is >> >> started, that script can then set things like this. I don't know >> enough to >> >> point >> >> you at specifically how to do that. >> >> >> >> David Lang >> >> >> >> >> >> On Tue, 16 Nov 2021, Scott Slattery wrote: >> >> >> >>> Date: Tue, 16 Nov 2021 17:07:47 -0700 >> >>> From: Scott Slattery <[email protected]> >> >>> To: David Lang <[email protected]> >> >>> Cc: Scott Slattery via rsyslog <[email protected]> >> >>> Subject: Re: [rsyslog] FROMHOST missing on central log collector >> >>> >> >>> Thanks David, the hostname is currently set in the AMI (Amazon Master >> >>> Image) which is the source image for all instances that are >> dynamically >> >>> created and I can verify that, if you login to one of these dynamic >> >>> instances, the hostname is in fact set correctly. >> >>> >> >>> The issue doesn't seem particularly related to what is set in >> >>> /etc/hostname, /etc/hosts, or what was set using 'hostname' command. I >> >>> think you can see this is the source of my frustration. It appears the >> >>> central log collector relies only on DNS resolution unless there's >> some >> >>> hidden magic inside RSYSLOG to force the sent logs to include a host >> >> header >> >>> (vs DNS). >> >>> >> >>> I don't want to continue wasting your time but again, it is much >> >>> appreciated. I'll look into some way of dynamically adding these >> hosts to >> >>> DNS in AWS Route53. It appears rsyslog simply can't do what I'm after. >> >>> >> >>> >> >>> *Scott Slattery* >> >>> >> >>> *Sr. Enterprise/Cloud Architect* >> >>> >> >>> *Cloud, Compute, Information & Architecture Team* >> >>> >> >>> motorolasolutions.com >> >>> >> >>> *O: 602.529.8226* >> >>> >> >>> *E*: [email protected] >> >>> >> >>> >> >>> >> >>> >> >>> On Tue, Nov 16, 2021 at 5:02 PM David Lang <[email protected]> wrote: >> >>> >> >>>> the hostname command will let you set the hostname (you want to do >> that >> >>>> before >> >>>> you start rsyslog). I would expect that the orcastration tool you >> use to >> >>>> create >> >>>> the systems will have some 'correct for that tool' way to set the >> >> hostname >> >>>> as it >> >>>> starts the instance (sorry I can't provide more specifics, if you can >> >>>> mention >> >>>> what you are using, possibly someone else can chime in on the best >> way >> >> to >> >>>> set >> >>>> the hostname with that tool) >> >>>> >> >>>> David Lang >> >>>> >> >>>> On Tue, 16 Nov 2021, Scott Slattery wrote: >> >>>> >> >>>>> Date: Tue, 16 Nov 2021 16:59:17 -0700 >> >>>>> From: Scott Slattery <[email protected]> >> >>>>> To: David Lang <[email protected]> >> >>>>> Cc: Scott Slattery via rsyslog <[email protected]> >> >>>>> Subject: Re: [rsyslog] FROMHOST missing on central log collector >> >>>>> >> >>>>> My follow-on question woudl be how do I set the hostname at the >> client >> >>>> end? >> >>>>> Other than what's in /etc/hosts, /etc/hostname, etc. I don't know >> how >> >>>> else >> >>>>> I would affect the log being sent to ensure it's going over. >> >>>>> >> >>>>> *Scott Slattery* >> >>>>> >> >>>>> *Sr. Enterprise/Cloud Architect* >> >>>>> >> >>>>> *Cloud, Compute, Information & Architecture Team* >> >>>>> >> >>>>> motorolasolutions.com >> >>>>> >> >>>>> *O: 602.529.8226* >> >>>>> >> >>>>> *E*: [email protected] >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Tue, Nov 16, 2021 at 4:55 PM David Lang <[email protected]> wrote: >> >>>>> >> >>>>>> the translation from fromhost-ip to fromhost is done at the >> collector, >> >>>> but >> >>>>>> the >> >>>>>> sender sets the hostname field. If you can trust that hostname was >> set >> >>>>>> correctly, there is no reason to use fromhost >> >>>>>> >> >>>>>> David Lang >> >>>>>> >> >>>>>> On Tue, 16 Nov 2021, Scott Slattery wrote: >> >>>>>> >> >>>>>>> Date: Tue, 16 Nov 2021 16:53:19 -0700 >> >>>>>>> From: Scott Slattery <[email protected]> >> >>>>>>> To: David Lang <[email protected]> >> >>>>>>> Cc: Scott Slattery via rsyslog <[email protected]> >> >>>>>>> Subject: Re: [rsyslog] FROMHOST missing on central log collector >> >>>>>>> >> >>>>>>> Thanks David, I could be wrong but the resolution seems to be >> >> happening >> >>>>>> at >> >>>>>>> the log collection server, not the client end. Given this, I'm not >> >> sure >> >>>>>>> anything outside of rsyslog on the client would affect what the >> >>>> receiving >> >>>>>>> collection server is seeing. >> >>>>>>> >> >>>>>>> My hope was that this could be affected by RSYSLOG on the client >> >> device >> >>>>>> but >> >>>>>>> perhaps not. I'll also look into AWS to see if a dynamically >> created >> >>>>>>> compute resource can automatically be registered with DNS. >> >>>>>>> >> >>>>>>> If anything else comes to mind, let me know. As always, I >> appreciate >> >>>> your >> >>>>>>> feedback. >> >>>>>>> >> >>>>>>> *Scott Slattery* >> >>>>>>> >> >>>>>>> *Sr. Enterprise/Cloud Architect* >> >>>>>>> >> >>>>>>> *Cloud, Compute, Information & Architecture Team* >> >>>>>>> >> >>>>>>> motorolasolutions.com >> >>>>>>> >> >>>>>>> *O: 602.529.8226* >> >>>>>>> >> >>>>>>> *E*: [email protected] >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> On Tue, Nov 16, 2021 at 4:37 PM David Lang <[email protected]> wrote: >> >>>>>>> >> >>>>>>>> Linux has a rather sophisticated mechanism for plugging in >> arbitrary >> >>>>>> ways >> >>>>>>>> of >> >>>>>>>> doing name resolution. DNS has 'won' but hitorically there have >> been >> >>>>>> many >> >>>>>>>> other >> >>>>>>>> options. Research nsswitch (/etc/nsswitch.conf) and see if there >> is >> >>>>>>>> something >> >>>>>>>> that you can leverage. >> >>>>>>>> >> >>>>>>>> or, if you can set the hostname of the resources as they are >> created >> >>>> to >> >>>>>> be >> >>>>>>>> some >> >>>>>>>> predicatable pattern rather than the AWS default of IP based, you >> >> can >> >>>>>> then >> >>>>>>>> make >> >>>>>>>> your logic use that. (This is the approach I would look into). >> What >> >>>>>>>> mechanism >> >>>>>>>> this will be will depend on how you are configuring/provisioning >> the >> >>>>>>>> systems. >> >>>>>>>> >> >>>>>>>> David Lang >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Tue, 16 Nov 2021, Scott Slattery wrote: >> >>>>>>>> >> >>>>>>>>> Date: Tue, 16 Nov 2021 15:14:51 -0700 >> >>>>>>>>> From: Scott Slattery <[email protected]> >> >>>>>>>>> To: David Lang <[email protected]> >> >>>>>>>>> Cc: Scott Slattery via rsyslog <[email protected]> >> >>>>>>>>> Subject: Re: [rsyslog] FROMHOST missing on central log collector >> >>>>>>>>> >> >>>>>>>>> Thanks, David, I was hoping this was possible. Since the compute >> >>>>>>>> resources >> >>>>>>>>> are dynamic, using any sort of local /etc/hosts would be >> impossible >> >>>>>> since >> >>>>>>>>> the IP are unpredictable. Can you point me to how I would do >> this >> >> on >> >>>>>> the >> >>>>>>>>> client-server? >> >>>>>>>>> >> >>>>>>>>> Thanks >> >>>>>>>>> >> >>>>>>>>> *Scott Slattery* >> >>>>>>>>> >> >>>>>>>>> *Sr. Enterprise/Cloud Architect* >> >>>>>>>>> >> >>>>>>>>> *Cloud, Compute, Information & Architecture Team* >> >>>>>>>>> >> >>>>>>>>> motorolasolutions.com >> >>>>>>>>> >> >>>>>>>>> *O: 602.529.8226* >> >>>>>>>>> >> >>>>>>>>> *E*: [email protected] >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Tue, Nov 16, 2021 at 2:59 PM David Lang <[email protected]> >> wrote: >> >>>>>>>>> >> >>>>>>>>>> fromhost is the result of a name lookup of fromhost-ip. On the >> >>>>>> receiver, >> >>>>>>>>>> you can >> >>>>>>>>>> control this with your name resolution (DNS, /etc/hosts, other >> >>>>>>>> mechanisms) >> >>>>>>>>>> >> >>>>>>>>>> but a better option would probably be to set the hostname on >> the >> >>>>>> sender. >> >>>>>>>>>> The >> >>>>>>>>>> hostname field in the message is under the full control of the >> >>>> sender. >> >>>>>>>>>> >> >>>>>>>>>> David Lang >> >>>>>>>>>> >> >>>>>>>>>> On Tue, 16 Nov 2021, Scott Slattery via rsyslog wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Date: Tue, 16 Nov 2021 14:56:09 -0700 >> >>>>>>>>>>> From: Scott Slattery via rsyslog <[email protected]> >> >>>>>>>>>>> To: rsyslog-users <[email protected]> >> >>>>>>>>>>> Cc: Scott Slattery <[email protected]> >> >>>>>>>>>>> Subject: [rsyslog] FROMHOST missing on central log collector >> >>>>>>>>>>> >> >>>>>>>>>>> Hello, >> >>>>>>>>>>> >> >>>>>>>>>>> I have a central log server, many of them, using rsyslog to >> >>>> aggregate >> >>>>>>>>>> logs >> >>>>>>>>>>> from remote servers. Everything works great but I have a new >> >>>>>> challenge >> >>>>>>>>>> and >> >>>>>>>>>>> am hoping for some recommendations. >> >>>>>>>>>>> >> >>>>>>>>>>> I have a number of AWS auto-scaling groups where compute >> >> resources >> >>>>>> are >> >>>>>>>>>>> dynamically scaled up and down. Each of these will have a >> custom >> >>>>>>>> rsyslog >> >>>>>>>>>>> configuration pulled from the AWS AMI. >> >>>>>>>>>>> >> >>>>>>>>>>> These dynamic resources are not added to DNS due to their >> dynamic >> >>>>>>>> nature >> >>>>>>>>>> so >> >>>>>>>>>>> they will not have DNS assigned FQDNs. >> >>>>>>>>>>> >> >>>>>>>>>>> Because of the lack of a hostname, my central log server is >> >> getting >> >>>>>>>> only >> >>>>>>>>>>> IP. I aggregate based on FROMHOST-FROMHOST-IP. >> >>>>>>>>>>> >> >>>>>>>>>>> So what I'm seeing today looks like >> '10.38.134.77-10.38.134.77' >> >>>>>> where I >> >>>>>>>>>>> want to see ause1oagbtst03.mydomain.com-10.41.102.168 >> >>>>>>>>>>> >> >>>>>>>>>>> What I'd want to do is have easy resource send using the same >> >>>>>> hostname >> >>>>>>>>>> and >> >>>>>>>>>>> current IP. This later will allow me to aggregate all >> resources >> >> by >> >>>>>>>> name. >> >>>>>>>>>>> >> >>>>>>>>>>> I did not see any way of affecting the FROMHOST information >> >> unless, >> >>>>>> on >> >>>>>>>>>> the >> >>>>>>>>>>> collector, I have rules based on IP address which isn't >> optimal >> >>>> given >> >>>>>>>> the >> >>>>>>>>>>> dynamic nature of the IPs changing. >> >>>>>>>>>>> >> >>>>>>>>>>> Any suggestion is appreciated. >> >>>>>>>>>>> >> >>>>>>>>>>> *Scott Slattery* >> >>>>>>>>>>> >> >>>>>>>>>>> *Sr. Enterprise/Cloud Architect* >> >>>>>>>>>>> >> >>>>>>>>>>> *Cloud, Compute, Information & Architecture Team* >> >>>>>>>>>>> >> >>>>>>>>>>> motorolasolutions.com >> >>>>>>>>>>> >> >>>>>>>>>>> *O: 602.529.8226* >> >>>>>>>>>>> >> >>>>>>>>>>> *E*: [email protected] >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >> >> > >> > >> _______________________________________________ >> rsyslog mailing list >> https://lists.adiscon.net/mailman/listinfo/rsyslog >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=lFiAzSG4O_IwoKCCbEi8i_yQYNFz5X0OWXMx9xGKGjYlvLeLrnYvixFB3egNPybF&s=db-lyqaTcrex58uwzOcY54hh137E9JMAF6vN-1IWnsA&e=> >> http://www.rsyslog.com/professional-services/ >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=lFiAzSG4O_IwoKCCbEi8i_yQYNFz5X0OWXMx9xGKGjYlvLeLrnYvixFB3egNPybF&s=eGWs1Xi6yCyCD3OYNlbvl3fIYBADttEDYjwGyicAZbk&e=> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=lFiAzSG4O_IwoKCCbEi8i_yQYNFz5X0OWXMx9xGKGjYlvLeLrnYvixFB3egNPybF&s=KIBqHKSAQtwhZA0rXY7Uh_or50wek4ABsH6-S4pxX0c&e=> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > > -- > Yury Bushmelev > -- *For more information on how and why we collect your personal information, please visit our Privacy Policy <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.* _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
