Re: [rsyslog] Forward messages from rsyslog server to JSON elasticSeach connector

2018-02-19 Thread David Lang

On Mon, 19 Feb 2018, sophie.loewenthal--- via rsyslog wrote:


Thank you Deoren for your thoughts.

I've seen some junk hostnames already appear in the logging directory. Thanks 
for your explanation. I can create an IP to Hostname table like IP:HOSTNAME 
pairs, but unsure how rsyslog could use this to lookup the incoming IP 
address.  Is there a feature in rsyslog for this?


Yes, there is the table_lookup() function, but it's not available in 8.7

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


Re: [rsyslog] Forward messages from rsyslog server to JSON elasticSeach connector

2018-02-19 Thread deoren

On 2/19/2018 10:17 AM, sophie.loewenthal--- via rsyslog wrote:

Thank you Deoren for your thoughts.


Welcome. Hopefully others will chime in with more details.


I've seen some junk hostnames already appear in the logging directory. Thanks 
for your explanation. I can create an IP to Hostname table  like IP:HOSTNAME 
pairs, but unsure how rsyslog could use this to lookup the incoming IP address. 
 Is there a feature in rsyslog for this?


There is, but I forgot that when I mentioned the functionality that the 
old version you're using may not have support for it (or it may be a 
buggy version if present).


See my recent responses to the list for a link back to the docs. 
Hopefully I'm wrong and it is available for your version, but I suspect 
it isn't.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


Re: [rsyslog] Forward messages from rsyslog server to JSON elasticSeach connector

2018-02-19 Thread sophie.loewenthal--- via rsyslog
Thank you Deoren for your thoughts.  

I've seen some junk hostnames already appear in the logging directory. Thanks 
for your explanation. I can create an IP to Hostname table  like IP:HOSTNAME 
pairs, but unsure how rsyslog could use this to lookup the incoming IP address. 
 Is there a feature in rsyslog for this?

We cannot upgrade until we migrate from Solaris to the latest RHEL. Not on the 
cards,yet.  Planned for later.

The changelog link was very insightful.

Best wishes,
Sophie


-Original Message-
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of deoren
Sent: Monday, February 19, 2018 4:49 PM
To: rsyslog@lists.adiscon.com
Subject: Re: [rsyslog] Forward messages from rsyslog server to JSON 
elasticSeach connector

On 2/19/2018 9:29 AM, sophie.loewenthal--- via rsyslog wrote:
> Hi,
> 
> Does this configuration look ok begore I let this configuration rip in 
> production?
> 
> A server running rsyslog 8.7.4 on Solaris 11 that receives TCP and UDP 
> messages from a mixture of syslog and rsyslog clients .
>   Each client has a %HOST.log created on the server file system.
> The rsyslog server forwards all those incoming messages into an ElasticSearch 
> via a JSON template server listening on a remote server on port 10514.
> 
> The configuration I wrote successfully receives the UDP and TCP messages on 
> the server.
> 
> Can anybody see any configuration there that could cause undue processing, or 
> errors. So far the testing has gone well.
> I've posted the configuration below.

Others can speak to specifics, but one word of warning regarding
expectations: %HOSTNAME% may sometimes have trash values if the remote sender 
doesn't properly format the message (or include reliable information).

We have a vulnerability scanner here that intentionally introduces bogus values 
and to override that behavior I've setup a lookup_table to map its source IP to 
a known value. Depending on your environment you may need to do something 
similar if you need to have reliable values in that field.

Two suggestions:

* If you run into problems it may be worth converting your configuration to use 
the current configuration syntax. That format has a lot of challenges that 
usually result in rsyslog not working as intended (directive order being just 
one)

* Consider upgrading to a current version of rsyslog. I recall you mentioning 
on the forums that you've stuck using that version, but there have been a lot 
of changes since rsyslog 8.7 was released and you may encounter issues that 
have long since been fixed.

I recall you saying that you couldn't reach GitHub, so here is a copy of the 
Changelog from the dev's Git server:

http://git.adiscon.com/?p=rsyslog.git;a=blob_plain;f=ChangeLog;hb=HEAD
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is 
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our 
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential. 
If you receive this message in error,or are not the intended recipient(s), 
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose, 
dissemination or disclosure, either whole or partial, is prohibited. Since the 
internet 
cannot guarantee the integrity of this message which may not be reliable, BNP 
PARIBAS 
(and its subsidiaries) shall not be liable for the message if modified, changed 
or falsified. 
Do not print this message unless it is necessary, consider the environment.

--

Ce message et toutes les pieces jointes (ci-apres le "message") 
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de 
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute 
publication, totale ou partielle, est interdite. L'Internet ne permettant pas 
d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas 
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans 
l'hypothese
ou il aurait ete modifie, deforme ou falsifie. 
N'imprimez ce message que si necessaire, pensez a l'environnement.

___

Re: [rsyslog] Forward messages from rsyslog server to JSON elasticSeach connector

2018-02-19 Thread deoren

On 2/19/2018 9:29 AM, sophie.loewenthal--- via rsyslog wrote:

Hi,

Does this configuration look ok begore I let this configuration rip in 
production?

A server running rsyslog 8.7.4 on Solaris 11 that receives TCP and UDP messages 
from a mixture of syslog and rsyslog clients .
  Each client has a %HOST.log created on the server file system.
The rsyslog server forwards all those incoming messages into an ElasticSearch 
via a JSON template server listening on a remote server on port 10514.

The configuration I wrote successfully receives the UDP and TCP messages on the 
server.

Can anybody see any configuration there that could cause undue processing, or 
errors. So far the testing has gone well.
I've posted the configuration below.


Others can speak to specifics, but one word of warning regarding 
expectations: %HOSTNAME% may sometimes have trash values if the remote 
sender doesn't properly format the message (or include reliable 
information).


We have a vulnerability scanner here that intentionally introduces bogus 
values and to override that behavior I've setup a lookup_table to map 
its source IP to a known value. Depending on your environment you may 
need to do something similar if you need to have reliable values in that 
field.


Two suggestions:

* If you run into problems it may be worth converting your configuration 
to use the current configuration syntax. That format has a lot of 
challenges that usually result in rsyslog not working as intended 
(directive order being just one)


* Consider upgrading to a current version of rsyslog. I recall you 
mentioning on the forums that you've stuck using that version, but there 
have been a lot of changes since rsyslog 8.7 was released and you may 
encounter issues that have long since been fixed.


I recall you saying that you couldn't reach GitHub, so here is a copy of 
the Changelog from the dev's Git server:


http://git.adiscon.com/?p=rsyslog.git;a=blob_plain;f=ChangeLog;hb=HEAD
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.