Thanks, David, I think you've done more than enough to try and help me on this. I need to do some reading on Amazon (and the link you shared) to see what my options are. I agree with you, it's likely workable.
I've confirmed that the results from the 'hostname' command do match so it's a bit of a mystery why rsyslog doesn't detect this but, i think you're on the right track, we need to run a post-deployment script to get these instances registered in Route53. *Scott Slattery* *Sr. Enterprise/Cloud Architect* *Cloud, Compute, Information & Architecture Team* motorolasolutions.com *O: 602.529.8226* *E*: [email protected] On Tue, Nov 16, 2021 at 5:20 PM David Lang <[email protected]> wrote: > if you login to one of the systems, you should find that the name returned > by > the hostname command should match what you get in the syslog message that > is > delivered to your central collector. (if it doesn't, try restarting > rsyslog and > see if it changes to match) > > then the question becomes what mechansims does AMI provide for customizing > the > hostname > > a quick google search shows a new hostnamectl command > > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_AWSEC2_latest_UserGuide_set-2Dhostname.html&d=DwIBAg&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=1JiTvUcvjB8RxLP9vfPbVSsbAQyitkPK6AzRBhEBUjRBWl-3tAtfR73TCtIFhdHZ&s=WR-Pz8svN0d8vqg4ZKSNj2dbxtcngaMJ4iiRXCPpD6c&e= > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cyberciti.biz_faq_set-2Dchange-2Dhostname-2Din-2Damazon-2Dlinux-2Dec2-2Dinstance-2Dserver_&d=DwIBAg&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=1JiTvUcvjB8RxLP9vfPbVSsbAQyitkPK6AzRBhEBUjRBWl-3tAtfR73TCtIFhdHZ&s=2RI1Khq-fBEBJxckXk9nWDESN8pTJxGiGv6xpsbYhzE&e= > > I know there is a way for you to specify a script to run when an instance > is > started, that script can then set things like this. I don't know enough to > point > you at specifically how to do that. > > David Lang > > > On Tue, 16 Nov 2021, Scott Slattery wrote: > > > Date: Tue, 16 Nov 2021 17:07:47 -0700 > > From: Scott Slattery <[email protected]> > > To: David Lang <[email protected]> > > Cc: Scott Slattery via rsyslog <[email protected]> > > Subject: Re: [rsyslog] FROMHOST missing on central log collector > > > > Thanks David, the hostname is currently set in the AMI (Amazon Master > > Image) which is the source image for all instances that are dynamically > > created and I can verify that, if you login to one of these dynamic > > instances, the hostname is in fact set correctly. > > > > The issue doesn't seem particularly related to what is set in > > /etc/hostname, /etc/hosts, or what was set using 'hostname' command. I > > think you can see this is the source of my frustration. It appears the > > central log collector relies only on DNS resolution unless there's some > > hidden magic inside RSYSLOG to force the sent logs to include a host > header > > (vs DNS). > > > > I don't want to continue wasting your time but again, it is much > > appreciated. I'll look into some way of dynamically adding these hosts to > > DNS in AWS Route53. It appears rsyslog simply can't do what I'm after. > > > > > > *Scott Slattery* > > > > *Sr. Enterprise/Cloud Architect* > > > > *Cloud, Compute, Information & Architecture Team* > > > > motorolasolutions.com > > > > *O: 602.529.8226* > > > > *E*: [email protected] > > > > > > > > > > On Tue, Nov 16, 2021 at 5:02 PM David Lang <[email protected]> wrote: > > > >> the hostname command will let you set the hostname (you want to do that > >> before > >> you start rsyslog). I would expect that the orcastration tool you use to > >> create > >> the systems will have some 'correct for that tool' way to set the > hostname > >> as it > >> starts the instance (sorry I can't provide more specifics, if you can > >> mention > >> what you are using, possibly someone else can chime in on the best way > to > >> set > >> the hostname with that tool) > >> > >> David Lang > >> > >> On Tue, 16 Nov 2021, Scott Slattery wrote: > >> > >>> Date: Tue, 16 Nov 2021 16:59:17 -0700 > >>> From: Scott Slattery <[email protected]> > >>> To: David Lang <[email protected]> > >>> Cc: Scott Slattery via rsyslog <[email protected]> > >>> Subject: Re: [rsyslog] FROMHOST missing on central log collector > >>> > >>> My follow-on question woudl be how do I set the hostname at the client > >> end? > >>> Other than what's in /etc/hosts, /etc/hostname, etc. I don't know how > >> else > >>> I would affect the log being sent to ensure it's going over. > >>> > >>> *Scott Slattery* > >>> > >>> *Sr. Enterprise/Cloud Architect* > >>> > >>> *Cloud, Compute, Information & Architecture Team* > >>> > >>> motorolasolutions.com > >>> > >>> *O: 602.529.8226* > >>> > >>> *E*: [email protected] > >>> > >>> > >>> > >>> > >>> On Tue, Nov 16, 2021 at 4:55 PM David Lang <[email protected]> wrote: > >>> > >>>> the translation from fromhost-ip to fromhost is done at the collector, > >> but > >>>> the > >>>> sender sets the hostname field. If you can trust that hostname was set > >>>> correctly, there is no reason to use fromhost > >>>> > >>>> David Lang > >>>> > >>>> On Tue, 16 Nov 2021, Scott Slattery wrote: > >>>> > >>>>> Date: Tue, 16 Nov 2021 16:53:19 -0700 > >>>>> From: Scott Slattery <[email protected]> > >>>>> To: David Lang <[email protected]> > >>>>> Cc: Scott Slattery via rsyslog <[email protected]> > >>>>> Subject: Re: [rsyslog] FROMHOST missing on central log collector > >>>>> > >>>>> Thanks David, I could be wrong but the resolution seems to be > happening > >>>> at > >>>>> the log collection server, not the client end. Given this, I'm not > sure > >>>>> anything outside of rsyslog on the client would affect what the > >> receiving > >>>>> collection server is seeing. > >>>>> > >>>>> My hope was that this could be affected by RSYSLOG on the client > device > >>>> but > >>>>> perhaps not. I'll also look into AWS to see if a dynamically created > >>>>> compute resource can automatically be registered with DNS. > >>>>> > >>>>> If anything else comes to mind, let me know. As always, I appreciate > >> your > >>>>> feedback. > >>>>> > >>>>> *Scott Slattery* > >>>>> > >>>>> *Sr. Enterprise/Cloud Architect* > >>>>> > >>>>> *Cloud, Compute, Information & Architecture Team* > >>>>> > >>>>> motorolasolutions.com > >>>>> > >>>>> *O: 602.529.8226* > >>>>> > >>>>> *E*: [email protected] > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Tue, Nov 16, 2021 at 4:37 PM David Lang <[email protected]> wrote: > >>>>> > >>>>>> Linux has a rather sophisticated mechanism for plugging in arbitrary > >>>> ways > >>>>>> of > >>>>>> doing name resolution. DNS has 'won' but hitorically there have been > >>>> many > >>>>>> other > >>>>>> options. Research nsswitch (/etc/nsswitch.conf) and see if there is > >>>>>> something > >>>>>> that you can leverage. > >>>>>> > >>>>>> or, if you can set the hostname of the resources as they are created > >> to > >>>> be > >>>>>> some > >>>>>> predicatable pattern rather than the AWS default of IP based, you > can > >>>> then > >>>>>> make > >>>>>> your logic use that. (This is the approach I would look into). What > >>>>>> mechanism > >>>>>> this will be will depend on how you are configuring/provisioning the > >>>>>> systems. > >>>>>> > >>>>>> David Lang > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Tue, 16 Nov 2021, Scott Slattery wrote: > >>>>>> > >>>>>>> Date: Tue, 16 Nov 2021 15:14:51 -0700 > >>>>>>> From: Scott Slattery <[email protected]> > >>>>>>> To: David Lang <[email protected]> > >>>>>>> Cc: Scott Slattery via rsyslog <[email protected]> > >>>>>>> Subject: Re: [rsyslog] FROMHOST missing on central log collector > >>>>>>> > >>>>>>> Thanks, David, I was hoping this was possible. Since the compute > >>>>>> resources > >>>>>>> are dynamic, using any sort of local /etc/hosts would be impossible > >>>> since > >>>>>>> the IP are unpredictable. Can you point me to how I would do this > on > >>>> the > >>>>>>> client-server? > >>>>>>> > >>>>>>> Thanks > >>>>>>> > >>>>>>> *Scott Slattery* > >>>>>>> > >>>>>>> *Sr. Enterprise/Cloud Architect* > >>>>>>> > >>>>>>> *Cloud, Compute, Information & Architecture Team* > >>>>>>> > >>>>>>> motorolasolutions.com > >>>>>>> > >>>>>>> *O: 602.529.8226* > >>>>>>> > >>>>>>> *E*: [email protected] > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Nov 16, 2021 at 2:59 PM David Lang <[email protected]> wrote: > >>>>>>> > >>>>>>>> fromhost is the result of a name lookup of fromhost-ip. On the > >>>> receiver, > >>>>>>>> you can > >>>>>>>> control this with your name resolution (DNS, /etc/hosts, other > >>>>>> mechanisms) > >>>>>>>> > >>>>>>>> but a better option would probably be to set the hostname on the > >>>> sender. > >>>>>>>> The > >>>>>>>> hostname field in the message is under the full control of the > >> sender. > >>>>>>>> > >>>>>>>> David Lang > >>>>>>>> > >>>>>>>> On Tue, 16 Nov 2021, Scott Slattery via rsyslog wrote: > >>>>>>>> > >>>>>>>>> Date: Tue, 16 Nov 2021 14:56:09 -0700 > >>>>>>>>> From: Scott Slattery via rsyslog <[email protected]> > >>>>>>>>> To: rsyslog-users <[email protected]> > >>>>>>>>> Cc: Scott Slattery <[email protected]> > >>>>>>>>> Subject: [rsyslog] FROMHOST missing on central log collector > >>>>>>>>> > >>>>>>>>> Hello, > >>>>>>>>> > >>>>>>>>> I have a central log server, many of them, using rsyslog to > >> aggregate > >>>>>>>> logs > >>>>>>>>> from remote servers. Everything works great but I have a new > >>>> challenge > >>>>>>>> and > >>>>>>>>> am hoping for some recommendations. > >>>>>>>>> > >>>>>>>>> I have a number of AWS auto-scaling groups where compute > resources > >>>> are > >>>>>>>>> dynamically scaled up and down. Each of these will have a custom > >>>>>> rsyslog > >>>>>>>>> configuration pulled from the AWS AMI. > >>>>>>>>> > >>>>>>>>> These dynamic resources are not added to DNS due to their dynamic > >>>>>> nature > >>>>>>>> so > >>>>>>>>> they will not have DNS assigned FQDNs. > >>>>>>>>> > >>>>>>>>> Because of the lack of a hostname, my central log server is > getting > >>>>>> only > >>>>>>>>> IP. I aggregate based on FROMHOST-FROMHOST-IP. > >>>>>>>>> > >>>>>>>>> So what I'm seeing today looks like '10.38.134.77-10.38.134.77' > >>>> where I > >>>>>>>>> want to see ause1oagbtst03.mydomain.com-10.41.102.168 > >>>>>>>>> > >>>>>>>>> What I'd want to do is have easy resource send using the same > >>>> hostname > >>>>>>>> and > >>>>>>>>> current IP. This later will allow me to aggregate all resources > by > >>>>>> name. > >>>>>>>>> > >>>>>>>>> I did not see any way of affecting the FROMHOST information > unless, > >>>> on > >>>>>>>> the > >>>>>>>>> collector, I have rules based on IP address which isn't optimal > >> given > >>>>>> the > >>>>>>>>> dynamic nature of the IPs changing. > >>>>>>>>> > >>>>>>>>> Any suggestion is appreciated. > >>>>>>>>> > >>>>>>>>> *Scott Slattery* > >>>>>>>>> > >>>>>>>>> *Sr. Enterprise/Cloud Architect* > >>>>>>>>> > >>>>>>>>> *Cloud, Compute, Information & Architecture Team* > >>>>>>>>> > >>>>>>>>> motorolasolutions.com > >>>>>>>>> > >>>>>>>>> *O: 602.529.8226* > >>>>>>>>> > >>>>>>>>> *E*: [email protected] > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>> > >>> > >> > > > > > -- *For more information on how and why we collect your personal information, please visit our Privacy Policy <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.* _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
