Think relay: fromhost is the last hop (socket layer sender), hostname is the original sender (syslog layer) - iff the sender works according to RFCs, unfortunately.
Rainer El mié, 17 nov 2021 a las 17:44, Scott Slattery via rsyslog (<[email protected]>) escribió: > > Thanks for your feedback. There seems to be some understanding that the > hostname is not set properly on the client-side. This is not the case, the > hostname displays properly on the host itself and is also properly > configured from a linux perspective. This is precisely why I'm inquiring > about alternatives. The only differentiating factor with respect to these > dynamically created hosts is that they do not get registered in DNS since > their life is, or can be, quite short based on computing demand. > > I was under the impression that the hostname used by the server-side > (collector) was the result of a server-side DNS lookup, which will not > resolve for these hosts. This is why I was looking for a rsyslog solution > that didn't involved DNS. > > Yuri, if I understand you correctly you're saying a custom template using > HOSTNAME vs FROMHOST-IP may be an option. I'll look into this to understand > it better. Thanks for this suggestion, it sound like it completely removes > the DNS constraint. I'll give it a try. > > *Scott Slattery* > > *Sr. Enterprise/Cloud Architect* > > *Cloud, Compute, Information & Architecture Team* > > motorolasolutions.com > > *O: 602.529.8226* > > *E*: [email protected] > > > > > On Tue, Nov 16, 2021 at 7:15 PM Yuri Bushmelev <[email protected]> wrote: > > > Hello! > > > > Just a reminder that a hostname field in a syslog message is just a string > > sent from sender to collector. So you can craft a custom template with the > > hostname field defined as you'd like. Though I'd call this a "fallback" way > > of fixing the issue. The right way is to set the proper hostname on a > > sender system before rsyslog starts I'd say. > > > > On Wed, 17 Nov 2021 at 08:33, David Lang via rsyslog < > > [email protected]> wrote: > > > >> Rsyslog looks up the hostname as it starts up, so if something after > >> rsyslog > >> starts changes the hostname, rsyslog isn't going to notice until you > >> restart > >> rsyslog. > >> > >> again, fromhost is a receiver side lookup of the name to match > >> fromhost-ip, so > >> if hostname is getting set correctly, filter on that instead of on > >> fromhost. > >> > >> David Lang > >> > >> On Tue, 16 Nov 2021, Scott Slattery wrote: > >> > >> > Date: Tue, 16 Nov 2021 17:28:15 -0700 > >> > From: Scott Slattery <[email protected]> > >> > To: David Lang <[email protected]> > >> > Cc: Scott Slattery via rsyslog <[email protected]> > >> > Subject: Re: [rsyslog] FROMHOST missing on central log collector > >> > > >> > Thanks, David, I think you've done more than enough to try and help me > >> on > >> > this. I need to do some reading on Amazon (and the link you shared) to > >> see > >> > what my options are. I agree with you, it's likely workable. > >> > > >> > I've confirmed that the results from the 'hostname' command do match so > >> > it's a bit of a mystery why rsyslog doesn't detect this but, i think > >> you're > >> > on the right track, we need to run a post-deployment script to get these > >> > instances registered in Route53. > >> > > >> > > >> > *Scott Slattery* > >> > > >> > *Sr. Enterprise/Cloud Architect* > >> > > >> > *Cloud, Compute, Information & Architecture Team* > >> > > >> > motorolasolutions.com > >> > > >> > *O: 602.529.8226* > >> > > >> > *E*: [email protected] > >> > > >> > > >> > > >> > > >> > On Tue, Nov 16, 2021 at 5:20 PM David Lang <[email protected]> wrote: > >> > > >> >> if you login to one of the systems, you should find that the name > >> returned > >> >> by > >> >> the hostname command should match what you get in the syslog message > >> that > >> >> is > >> >> delivered to your central collector. (if it doesn't, try restarting > >> >> rsyslog and > >> >> see if it changes to match) > >> >> > >> >> then the question becomes what mechansims does AMI provide for > >> customizing > >> >> the > >> >> hostname > >> >> > >> >> a quick google search shows a new hostnamectl command > >> >> > >> >> > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_AWSEC2_latest_UserGuide_set-2Dhostname.html&d=DwIBAg&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=1JiTvUcvjB8RxLP9vfPbVSsbAQyitkPK6AzRBhEBUjRBWl-3tAtfR73TCtIFhdHZ&s=WR-Pz8svN0d8vqg4ZKSNj2dbxtcngaMJ4iiRXCPpD6c&e= > >> >> > >> >> > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cyberciti.biz_faq_set-2Dchange-2Dhostname-2Din-2Damazon-2Dlinux-2Dec2-2Dinstance-2Dserver_&d=DwIBAg&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=1JiTvUcvjB8RxLP9vfPbVSsbAQyitkPK6AzRBhEBUjRBWl-3tAtfR73TCtIFhdHZ&s=2RI1Khq-fBEBJxckXk9nWDESN8pTJxGiGv6xpsbYhzE&e= > >> >> > >> >> I know there is a way for you to specify a script to run when an > >> instance > >> >> is > >> >> started, that script can then set things like this. I don't know > >> enough to > >> >> point > >> >> you at specifically how to do that. > >> >> > >> >> David Lang > >> >> > >> >> > >> >> On Tue, 16 Nov 2021, Scott Slattery wrote: > >> >> > >> >>> Date: Tue, 16 Nov 2021 17:07:47 -0700 > >> >>> From: Scott Slattery <[email protected]> > >> >>> To: David Lang <[email protected]> > >> >>> Cc: Scott Slattery via rsyslog <[email protected]> > >> >>> Subject: Re: [rsyslog] FROMHOST missing on central log collector > >> >>> > >> >>> Thanks David, the hostname is currently set in the AMI (Amazon Master > >> >>> Image) which is the source image for all instances that are > >> dynamically > >> >>> created and I can verify that, if you login to one of these dynamic > >> >>> instances, the hostname is in fact set correctly. > >> >>> > >> >>> The issue doesn't seem particularly related to what is set in > >> >>> /etc/hostname, /etc/hosts, or what was set using 'hostname' command. I > >> >>> think you can see this is the source of my frustration. It appears the > >> >>> central log collector relies only on DNS resolution unless there's > >> some > >> >>> hidden magic inside RSYSLOG to force the sent logs to include a host > >> >> header > >> >>> (vs DNS). > >> >>> > >> >>> I don't want to continue wasting your time but again, it is much > >> >>> appreciated. I'll look into some way of dynamically adding these > >> hosts to > >> >>> DNS in AWS Route53. It appears rsyslog simply can't do what I'm after. > >> >>> > >> >>> > >> >>> *Scott Slattery* > >> >>> > >> >>> *Sr. Enterprise/Cloud Architect* > >> >>> > >> >>> *Cloud, Compute, Information & Architecture Team* > >> >>> > >> >>> motorolasolutions.com > >> >>> > >> >>> *O: 602.529.8226* > >> >>> > >> >>> *E*: [email protected] > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> On Tue, Nov 16, 2021 at 5:02 PM David Lang <[email protected]> wrote: > >> >>> > >> >>>> the hostname command will let you set the hostname (you want to do > >> that > >> >>>> before > >> >>>> you start rsyslog). I would expect that the orcastration tool you > >> use to > >> >>>> create > >> >>>> the systems will have some 'correct for that tool' way to set the > >> >> hostname > >> >>>> as it > >> >>>> starts the instance (sorry I can't provide more specifics, if you can > >> >>>> mention > >> >>>> what you are using, possibly someone else can chime in on the best > >> way > >> >> to > >> >>>> set > >> >>>> the hostname with that tool) > >> >>>> > >> >>>> David Lang > >> >>>> > >> >>>> On Tue, 16 Nov 2021, Scott Slattery wrote: > >> >>>> > >> >>>>> Date: Tue, 16 Nov 2021 16:59:17 -0700 > >> >>>>> From: Scott Slattery <[email protected]> > >> >>>>> To: David Lang <[email protected]> > >> >>>>> Cc: Scott Slattery via rsyslog <[email protected]> > >> >>>>> Subject: Re: [rsyslog] FROMHOST missing on central log collector > >> >>>>> > >> >>>>> My follow-on question woudl be how do I set the hostname at the > >> client > >> >>>> end? > >> >>>>> Other than what's in /etc/hosts, /etc/hostname, etc. I don't know > >> how > >> >>>> else > >> >>>>> I would affect the log being sent to ensure it's going over. > >> >>>>> > >> >>>>> *Scott Slattery* > >> >>>>> > >> >>>>> *Sr. Enterprise/Cloud Architect* > >> >>>>> > >> >>>>> *Cloud, Compute, Information & Architecture Team* > >> >>>>> > >> >>>>> motorolasolutions.com > >> >>>>> > >> >>>>> *O: 602.529.8226* > >> >>>>> > >> >>>>> *E*: [email protected] > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> On Tue, Nov 16, 2021 at 4:55 PM David Lang <[email protected]> wrote: > >> >>>>> > >> >>>>>> the translation from fromhost-ip to fromhost is done at the > >> collector, > >> >>>> but > >> >>>>>> the > >> >>>>>> sender sets the hostname field. If you can trust that hostname was > >> set > >> >>>>>> correctly, there is no reason to use fromhost > >> >>>>>> > >> >>>>>> David Lang > >> >>>>>> > >> >>>>>> On Tue, 16 Nov 2021, Scott Slattery wrote: > >> >>>>>> > >> >>>>>>> Date: Tue, 16 Nov 2021 16:53:19 -0700 > >> >>>>>>> From: Scott Slattery <[email protected]> > >> >>>>>>> To: David Lang <[email protected]> > >> >>>>>>> Cc: Scott Slattery via rsyslog <[email protected]> > >> >>>>>>> Subject: Re: [rsyslog] FROMHOST missing on central log collector > >> >>>>>>> > >> >>>>>>> Thanks David, I could be wrong but the resolution seems to be > >> >> happening > >> >>>>>> at > >> >>>>>>> the log collection server, not the client end. Given this, I'm not > >> >> sure > >> >>>>>>> anything outside of rsyslog on the client would affect what the > >> >>>> receiving > >> >>>>>>> collection server is seeing. > >> >>>>>>> > >> >>>>>>> My hope was that this could be affected by RSYSLOG on the client > >> >> device > >> >>>>>> but > >> >>>>>>> perhaps not. I'll also look into AWS to see if a dynamically > >> created > >> >>>>>>> compute resource can automatically be registered with DNS. > >> >>>>>>> > >> >>>>>>> If anything else comes to mind, let me know. As always, I > >> appreciate > >> >>>> your > >> >>>>>>> feedback. > >> >>>>>>> > >> >>>>>>> *Scott Slattery* > >> >>>>>>> > >> >>>>>>> *Sr. Enterprise/Cloud Architect* > >> >>>>>>> > >> >>>>>>> *Cloud, Compute, Information & Architecture Team* > >> >>>>>>> > >> >>>>>>> motorolasolutions.com > >> >>>>>>> > >> >>>>>>> *O: 602.529.8226* > >> >>>>>>> > >> >>>>>>> *E*: [email protected] > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> On Tue, Nov 16, 2021 at 4:37 PM David Lang <[email protected]> wrote: > >> >>>>>>> > >> >>>>>>>> Linux has a rather sophisticated mechanism for plugging in > >> arbitrary > >> >>>>>> ways > >> >>>>>>>> of > >> >>>>>>>> doing name resolution. DNS has 'won' but hitorically there have > >> been > >> >>>>>> many > >> >>>>>>>> other > >> >>>>>>>> options. Research nsswitch (/etc/nsswitch.conf) and see if there > >> is > >> >>>>>>>> something > >> >>>>>>>> that you can leverage. > >> >>>>>>>> > >> >>>>>>>> or, if you can set the hostname of the resources as they are > >> created > >> >>>> to > >> >>>>>> be > >> >>>>>>>> some > >> >>>>>>>> predicatable pattern rather than the AWS default of IP based, you > >> >> can > >> >>>>>> then > >> >>>>>>>> make > >> >>>>>>>> your logic use that. (This is the approach I would look into). > >> What > >> >>>>>>>> mechanism > >> >>>>>>>> this will be will depend on how you are configuring/provisioning > >> the > >> >>>>>>>> systems. > >> >>>>>>>> > >> >>>>>>>> David Lang > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> On Tue, 16 Nov 2021, Scott Slattery wrote: > >> >>>>>>>> > >> >>>>>>>>> Date: Tue, 16 Nov 2021 15:14:51 -0700 > >> >>>>>>>>> From: Scott Slattery <[email protected]> > >> >>>>>>>>> To: David Lang <[email protected]> > >> >>>>>>>>> Cc: Scott Slattery via rsyslog <[email protected]> > >> >>>>>>>>> Subject: Re: [rsyslog] FROMHOST missing on central log collector > >> >>>>>>>>> > >> >>>>>>>>> Thanks, David, I was hoping this was possible. Since the compute > >> >>>>>>>> resources > >> >>>>>>>>> are dynamic, using any sort of local /etc/hosts would be > >> impossible > >> >>>>>> since > >> >>>>>>>>> the IP are unpredictable. Can you point me to how I would do > >> this > >> >> on > >> >>>>>> the > >> >>>>>>>>> client-server? > >> >>>>>>>>> > >> >>>>>>>>> Thanks > >> >>>>>>>>> > >> >>>>>>>>> *Scott Slattery* > >> >>>>>>>>> > >> >>>>>>>>> *Sr. Enterprise/Cloud Architect* > >> >>>>>>>>> > >> >>>>>>>>> *Cloud, Compute, Information & Architecture Team* > >> >>>>>>>>> > >> >>>>>>>>> motorolasolutions.com > >> >>>>>>>>> > >> >>>>>>>>> *O: 602.529.8226* > >> >>>>>>>>> > >> >>>>>>>>> *E*: [email protected] > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> On Tue, Nov 16, 2021 at 2:59 PM David Lang <[email protected]> > >> wrote: > >> >>>>>>>>> > >> >>>>>>>>>> fromhost is the result of a name lookup of fromhost-ip. On the > >> >>>>>> receiver, > >> >>>>>>>>>> you can > >> >>>>>>>>>> control this with your name resolution (DNS, /etc/hosts, other > >> >>>>>>>> mechanisms) > >> >>>>>>>>>> > >> >>>>>>>>>> but a better option would probably be to set the hostname on > >> the > >> >>>>>> sender. > >> >>>>>>>>>> The > >> >>>>>>>>>> hostname field in the message is under the full control of the > >> >>>> sender. > >> >>>>>>>>>> > >> >>>>>>>>>> David Lang > >> >>>>>>>>>> > >> >>>>>>>>>> On Tue, 16 Nov 2021, Scott Slattery via rsyslog wrote: > >> >>>>>>>>>> > >> >>>>>>>>>>> Date: Tue, 16 Nov 2021 14:56:09 -0700 > >> >>>>>>>>>>> From: Scott Slattery via rsyslog <[email protected]> > >> >>>>>>>>>>> To: rsyslog-users <[email protected]> > >> >>>>>>>>>>> Cc: Scott Slattery <[email protected]> > >> >>>>>>>>>>> Subject: [rsyslog] FROMHOST missing on central log collector > >> >>>>>>>>>>> > >> >>>>>>>>>>> Hello, > >> >>>>>>>>>>> > >> >>>>>>>>>>> I have a central log server, many of them, using rsyslog to > >> >>>> aggregate > >> >>>>>>>>>> logs > >> >>>>>>>>>>> from remote servers. Everything works great but I have a new > >> >>>>>> challenge > >> >>>>>>>>>> and > >> >>>>>>>>>>> am hoping for some recommendations. > >> >>>>>>>>>>> > >> >>>>>>>>>>> I have a number of AWS auto-scaling groups where compute > >> >> resources > >> >>>>>> are > >> >>>>>>>>>>> dynamically scaled up and down. Each of these will have a > >> custom > >> >>>>>>>> rsyslog > >> >>>>>>>>>>> configuration pulled from the AWS AMI. > >> >>>>>>>>>>> > >> >>>>>>>>>>> These dynamic resources are not added to DNS due to their > >> dynamic > >> >>>>>>>> nature > >> >>>>>>>>>> so > >> >>>>>>>>>>> they will not have DNS assigned FQDNs. > >> >>>>>>>>>>> > >> >>>>>>>>>>> Because of the lack of a hostname, my central log server is > >> >> getting > >> >>>>>>>> only > >> >>>>>>>>>>> IP. I aggregate based on FROMHOST-FROMHOST-IP. > >> >>>>>>>>>>> > >> >>>>>>>>>>> So what I'm seeing today looks like > >> '10.38.134.77-10.38.134.77' > >> >>>>>> where I > >> >>>>>>>>>>> want to see ause1oagbtst03.mydomain.com-10.41.102.168 > >> >>>>>>>>>>> > >> >>>>>>>>>>> What I'd want to do is have easy resource send using the same > >> >>>>>> hostname > >> >>>>>>>>>> and > >> >>>>>>>>>>> current IP. This later will allow me to aggregate all > >> resources > >> >> by > >> >>>>>>>> name. > >> >>>>>>>>>>> > >> >>>>>>>>>>> I did not see any way of affecting the FROMHOST information > >> >> unless, > >> >>>>>> on > >> >>>>>>>>>> the > >> >>>>>>>>>>> collector, I have rules based on IP address which isn't > >> optimal > >> >>>> given > >> >>>>>>>> the > >> >>>>>>>>>>> dynamic nature of the IPs changing. > >> >>>>>>>>>>> > >> >>>>>>>>>>> Any suggestion is appreciated. > >> >>>>>>>>>>> > >> >>>>>>>>>>> *Scott Slattery* > >> >>>>>>>>>>> > >> >>>>>>>>>>> *Sr. Enterprise/Cloud Architect* > >> >>>>>>>>>>> > >> >>>>>>>>>>> *Cloud, Compute, Information & Architecture Team* > >> >>>>>>>>>>> > >> >>>>>>>>>>> motorolasolutions.com > >> >>>>>>>>>>> > >> >>>>>>>>>>> *O: 602.529.8226* > >> >>>>>>>>>>> > >> >>>>>>>>>>> *E*: [email protected] > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>> > >> >>>>> > >> >>>>> > >> >>>> > >> >>> > >> >>> > >> >> > >> > > >> > > >> _______________________________________________ > >> rsyslog mailing list > >> https://lists.adiscon.net/mailman/listinfo/rsyslog > >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=lFiAzSG4O_IwoKCCbEi8i_yQYNFz5X0OWXMx9xGKGjYlvLeLrnYvixFB3egNPybF&s=db-lyqaTcrex58uwzOcY54hh137E9JMAF6vN-1IWnsA&e=> > >> http://www.rsyslog.com/professional-services/ > >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=lFiAzSG4O_IwoKCCbEi8i_yQYNFz5X0OWXMx9xGKGjYlvLeLrnYvixFB3egNPybF&s=eGWs1Xi6yCyCD3OYNlbvl3fIYBADttEDYjwGyicAZbk&e=> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=lFiAzSG4O_IwoKCCbEi8i_yQYNFz5X0OWXMx9xGKGjYlvLeLrnYvixFB3egNPybF&s=KIBqHKSAQtwhZA0rXY7Uh_or50wek4ABsH6-S4pxX0c&e=> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > > > > > -- > > Yury Bushmelev > > > > -- > > > *For more information on how and why we collect your personal > information, please visit our Privacy Policy > <https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.* > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
