on the receiver:
HOSTNAME is what the sender put in the message
FROMHOST-IP is the IP of the last hop before the receiver (in your case the
sender)
FROMHOST is a name lookup of FROMHOST-IP
if everything has the correct name, and that name is in DNS/hosts, then FROMHOST
and HOSTNAME may be the same, or FROMHOST may be FQDN while HOSTNAME is just the
short name (depends on the sending system)
if you can set the HOSTNAME before rsyslog starts so that it's in the message
itself correctly, then you don't need to depend on updating name resolution (and
name lookups cost time and there is always a lag between a system starting up
and when the update will show up to a lookup on the receiver)
David Lang
On Wed, 17 Nov 2021, Scott Slattery wrote:
Date: Wed, 17 Nov 2021 09:43:47 -0700
From: Scott Slattery <[email protected]>
To: Yuri Bushmelev <[email protected]>
Cc: rsyslog-users <[email protected]>, David Lang <[email protected]>
Subject: Re: [rsyslog] FROMHOST missing on central log collector
Thanks for your feedback. There seems to be some understanding that the
hostname is not set properly on the client-side. This is not the case, the
hostname displays properly on the host itself and is also properly
configured from a linux perspective. This is precisely why I'm inquiring
about alternatives. The only differentiating factor with respect to these
dynamically created hosts is that they do not get registered in DNS since
their life is, or can be, quite short based on computing demand.
I was under the impression that the hostname used by the server-side
(collector) was the result of a server-side DNS lookup, which will not
resolve for these hosts. This is why I was looking for a rsyslog solution
that didn't involved DNS.
Yuri, if I understand you correctly you're saying a custom template using
HOSTNAME vs FROMHOST-IP may be an option. I'll look into this to understand
it better. Thanks for this suggestion, it sound like it completely removes
the DNS constraint. I'll give it a try.
*Scott Slattery*
*Sr. Enterprise/Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.8226*
*E*: [email protected]
On Tue, Nov 16, 2021 at 7:15 PM Yuri Bushmelev <[email protected]> wrote:
Hello!
Just a reminder that a hostname field in a syslog message is just a string
sent from sender to collector. So you can craft a custom template with the
hostname field defined as you'd like. Though I'd call this a "fallback" way
of fixing the issue. The right way is to set the proper hostname on a
sender system before rsyslog starts I'd say.
On Wed, 17 Nov 2021 at 08:33, David Lang via rsyslog <
[email protected]> wrote:
Rsyslog looks up the hostname as it starts up, so if something after
rsyslog
starts changes the hostname, rsyslog isn't going to notice until you
restart
rsyslog.
again, fromhost is a receiver side lookup of the name to match
fromhost-ip, so
if hostname is getting set correctly, filter on that instead of on
fromhost.
David Lang
On Tue, 16 Nov 2021, Scott Slattery wrote:
Date: Tue, 16 Nov 2021 17:28:15 -0700
From: Scott Slattery <[email protected]>
To: David Lang <[email protected]>
Cc: Scott Slattery via rsyslog <[email protected]>
Subject: Re: [rsyslog] FROMHOST missing on central log collector
Thanks, David, I think you've done more than enough to try and help me
on
this. I need to do some reading on Amazon (and the link you shared) to
see
what my options are. I agree with you, it's likely workable.
I've confirmed that the results from the 'hostname' command do match so
it's a bit of a mystery why rsyslog doesn't detect this but, i think
you're
on the right track, we need to run a post-deployment script to get these
instances registered in Route53.
*Scott Slattery*
*Sr. Enterprise/Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.8226*
*E*: [email protected]
On Tue, Nov 16, 2021 at 5:20 PM David Lang <[email protected]> wrote:
if you login to one of the systems, you should find that the name
returned
by
the hostname command should match what you get in the syslog message
that
is
delivered to your central collector. (if it doesn't, try restarting
rsyslog and
see if it changes to match)
then the question becomes what mechansims does AMI provide for
customizing
the
hostname
a quick google search shows a new hostnamectl command
https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_AWSEC2_latest_UserGuide_set-2Dhostname.html&d=DwIBAg&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=1JiTvUcvjB8RxLP9vfPbVSsbAQyitkPK6AzRBhEBUjRBWl-3tAtfR73TCtIFhdHZ&s=WR-Pz8svN0d8vqg4ZKSNj2dbxtcngaMJ4iiRXCPpD6c&e=
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cyberciti.biz_faq_set-2Dchange-2Dhostname-2Din-2Damazon-2Dlinux-2Dec2-2Dinstance-2Dserver_&d=DwIBAg&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=1JiTvUcvjB8RxLP9vfPbVSsbAQyitkPK6AzRBhEBUjRBWl-3tAtfR73TCtIFhdHZ&s=2RI1Khq-fBEBJxckXk9nWDESN8pTJxGiGv6xpsbYhzE&e=
I know there is a way for you to specify a script to run when an
instance
is
started, that script can then set things like this. I don't know
enough to
point
you at specifically how to do that.
David Lang
On Tue, 16 Nov 2021, Scott Slattery wrote:
Date: Tue, 16 Nov 2021 17:07:47 -0700
From: Scott Slattery <[email protected]>
To: David Lang <[email protected]>
Cc: Scott Slattery via rsyslog <[email protected]>
Subject: Re: [rsyslog] FROMHOST missing on central log collector
Thanks David, the hostname is currently set in the AMI (Amazon Master
Image) which is the source image for all instances that are
dynamically
created and I can verify that, if you login to one of these dynamic
instances, the hostname is in fact set correctly.
The issue doesn't seem particularly related to what is set in
/etc/hostname, /etc/hosts, or what was set using 'hostname' command. I
think you can see this is the source of my frustration. It appears the
central log collector relies only on DNS resolution unless there's
some
hidden magic inside RSYSLOG to force the sent logs to include a host
header
(vs DNS).
I don't want to continue wasting your time but again, it is much
appreciated. I'll look into some way of dynamically adding these
hosts to
DNS in AWS Route53. It appears rsyslog simply can't do what I'm after.
*Scott Slattery*
*Sr. Enterprise/Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.8226*
*E*: [email protected]
On Tue, Nov 16, 2021 at 5:02 PM David Lang <[email protected]> wrote:
the hostname command will let you set the hostname (you want to do
that
before
you start rsyslog). I would expect that the orcastration tool you
use to
create
the systems will have some 'correct for that tool' way to set the
hostname
as it
starts the instance (sorry I can't provide more specifics, if you can
mention
what you are using, possibly someone else can chime in on the best
way
to
set
the hostname with that tool)
David Lang
On Tue, 16 Nov 2021, Scott Slattery wrote:
Date: Tue, 16 Nov 2021 16:59:17 -0700
From: Scott Slattery <[email protected]>
To: David Lang <[email protected]>
Cc: Scott Slattery via rsyslog <[email protected]>
Subject: Re: [rsyslog] FROMHOST missing on central log collector
My follow-on question woudl be how do I set the hostname at the
client
end?
Other than what's in /etc/hosts, /etc/hostname, etc. I don't know
how
else
I would affect the log being sent to ensure it's going over.
*Scott Slattery*
*Sr. Enterprise/Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.8226*
*E*: [email protected]
On Tue, Nov 16, 2021 at 4:55 PM David Lang <[email protected]> wrote:
the translation from fromhost-ip to fromhost is done at the
collector,
but
the
sender sets the hostname field. If you can trust that hostname was
set
correctly, there is no reason to use fromhost
David Lang
On Tue, 16 Nov 2021, Scott Slattery wrote:
Date: Tue, 16 Nov 2021 16:53:19 -0700
From: Scott Slattery <[email protected]>
To: David Lang <[email protected]>
Cc: Scott Slattery via rsyslog <[email protected]>
Subject: Re: [rsyslog] FROMHOST missing on central log collector
Thanks David, I could be wrong but the resolution seems to be
happening
at
the log collection server, not the client end. Given this, I'm not
sure
anything outside of rsyslog on the client would affect what the
receiving
collection server is seeing.
My hope was that this could be affected by RSYSLOG on the client
device
but
perhaps not. I'll also look into AWS to see if a dynamically
created
compute resource can automatically be registered with DNS.
If anything else comes to mind, let me know. As always, I
appreciate
your
feedback.
*Scott Slattery*
*Sr. Enterprise/Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.8226*
*E*: [email protected]
On Tue, Nov 16, 2021 at 4:37 PM David Lang <[email protected]> wrote:
Linux has a rather sophisticated mechanism for plugging in
arbitrary
ways
of
doing name resolution. DNS has 'won' but hitorically there have
been
many
other
options. Research nsswitch (/etc/nsswitch.conf) and see if there
is
something
that you can leverage.
or, if you can set the hostname of the resources as they are
created
to
be
some
predicatable pattern rather than the AWS default of IP based, you
can
then
make
your logic use that. (This is the approach I would look into).
What
mechanism
this will be will depend on how you are configuring/provisioning
the
systems.
David Lang
On Tue, 16 Nov 2021, Scott Slattery wrote:
Date: Tue, 16 Nov 2021 15:14:51 -0700
From: Scott Slattery <[email protected]>
To: David Lang <[email protected]>
Cc: Scott Slattery via rsyslog <[email protected]>
Subject: Re: [rsyslog] FROMHOST missing on central log collector
Thanks, David, I was hoping this was possible. Since the compute
resources
are dynamic, using any sort of local /etc/hosts would be
impossible
since
the IP are unpredictable. Can you point me to how I would do
this
on
the
client-server?
Thanks
*Scott Slattery*
*Sr. Enterprise/Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.8226*
*E*: [email protected]
On Tue, Nov 16, 2021 at 2:59 PM David Lang <[email protected]>
wrote:
fromhost is the result of a name lookup of fromhost-ip. On the
receiver,
you can
control this with your name resolution (DNS, /etc/hosts, other
mechanisms)
but a better option would probably be to set the hostname on
the
sender.
The
hostname field in the message is under the full control of the
sender.
David Lang
On Tue, 16 Nov 2021, Scott Slattery via rsyslog wrote:
Date: Tue, 16 Nov 2021 14:56:09 -0700
From: Scott Slattery via rsyslog <[email protected]>
To: rsyslog-users <[email protected]>
Cc: Scott Slattery <[email protected]>
Subject: [rsyslog] FROMHOST missing on central log collector
Hello,
I have a central log server, many of them, using rsyslog to
aggregate
logs
from remote servers. Everything works great but I have a new
challenge
and
am hoping for some recommendations.
I have a number of AWS auto-scaling groups where compute
resources
are
dynamically scaled up and down. Each of these will have a
custom
rsyslog
configuration pulled from the AWS AMI.
These dynamic resources are not added to DNS due to their
dynamic
nature
so
they will not have DNS assigned FQDNs.
Because of the lack of a hostname, my central log server is
getting
only
IP. I aggregate based on FROMHOST-FROMHOST-IP.
So what I'm seeing today looks like
'10.38.134.77-10.38.134.77'
where I
want to see ause1oagbtst03.mydomain.com-10.41.102.168
What I'd want to do is have easy resource send using the same
hostname
and
current IP. This later will allow me to aggregate all
resources
by
name.
I did not see any way of affecting the FROMHOST information
unless,
on
the
collector, I have rules based on IP address which isn't
optimal
given
the
dynamic nature of the IPs changing.
Any suggestion is appreciated.
*Scott Slattery*
*Sr. Enterprise/Cloud Architect*
*Cloud, Compute, Information & Architecture Team*
motorolasolutions.com
*O: 602.529.8226*
*E*: [email protected]
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=lFiAzSG4O_IwoKCCbEi8i_yQYNFz5X0OWXMx9xGKGjYlvLeLrnYvixFB3egNPybF&s=db-lyqaTcrex58uwzOcY54hh137E9JMAF6vN-1IWnsA&e=>
http://www.rsyslog.com/professional-services/
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professional-2Dservices_&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=lFiAzSG4O_IwoKCCbEi8i_yQYNFz5X0OWXMx9xGKGjYlvLeLrnYvixFB3egNPybF&s=eGWs1Xi6yCyCD3OYNlbvl3fIYBADttEDYjwGyicAZbk&e=>
What's up with rsyslog? Follow https://twitter.com/rgerhards
<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_C93KOmcBXCBnhee2v6PYlc&r=9VZN8jOeh6Wq3zsBr6Mr_GSxmEpodGbXQ2UxP3oRpciBnWp1cJKyh3iyX6xKS_Zd&m=lFiAzSG4O_IwoKCCbEi8i_yQYNFz5X0OWXMx9xGKGjYlvLeLrnYvixFB3egNPybF&s=KIBqHKSAQtwhZA0rXY7Uh_or50wek4ABsH6-S4pxX0c&e=>
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
--
Yury Bushmelev
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
