Re: [Freeipa-users] Logging: IPA to Rsyslog to Logstash
Thanks, I just changed it to follow Mediawiki syntax and renamed it to http://www.freeipa.org/page/Howto/Centralised_Logging_with_Logstash/ElasticSearch/Kibana to keep current Howto structure. Please feel free encouraged to fill up any more details as you go with your adventures that the community may profit from! Thanks, Martin On 01/05/2015 12:49 PM, Innes, Duncan wrote: > Sure - efforts so far at: > > http://www.freeipa.org/page/Centralised_Logging_with_Logstash/ElasticSea > rch/Kibana > > Hope it helps. > > Cheers > Duncan > >> -Original Message- >> From: freeipa-users-boun...@redhat.com > [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Petr Spacek >> Sent: 05 January 2015 08:22 >> To: freeipa-users@redhat.com >> Subject: Re: [Freeipa-users] Logging: IPA to Rsyslog to Logstash >> >> Hello Duncan, >> >> thank you for doing this! >> >> Could you transform this post to >> http://www.freeipa.org/page/HowTos#Working_with_FreeIPA article, > please? I think that other people could use that too. >> >> Thank you very much. >> Petr^2 Spacek > > This message has been checked for viruses and spam by the Virgin Money email > scanning system powered by Messagelabs. > > This e-mail is intended to be confidential to the recipient. If you receive a > copy in error, please inform the sender and then delete this message. > > Virgin Money plc - Registered in England and Wales (Company no. 6952311). > Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL. > Virgin Money plc is authorised by the Prudential Regulation Authority and > regulated by the Financial Conduct Authority and the Prudential Regulation > Authority. > > The following companies also trade as Virgin Money. They are both authorised > and regulated by the Financial Conduct Authority, are registered in England > and Wales and have their registered office at Jubilee House, Gosforth, > Newcastle upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited > (Company no. 3072766) and Virgin Money Unit Trust Managers Limited (Company > no. 3000482). > > For further details of Virgin Money group companies please visit our website > at virginmoney.com > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Logging: IPA to Rsyslog to Logstash
Sure - efforts so far at: http://www.freeipa.org/page/Centralised_Logging_with_Logstash/ElasticSea rch/Kibana Hope it helps. Cheers Duncan > -Original Message- > From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Petr Spacek > Sent: 05 January 2015 08:22 > To: freeipa-users@redhat.com > Subject: Re: [Freeipa-users] Logging: IPA to Rsyslog to Logstash > > Hello Duncan, > > thank you for doing this! > > Could you transform this post to > http://www.freeipa.org/page/HowTos#Working_with_FreeIPA article, please? I think that other people could use that too. > > Thank you very much. > Petr^2 Spacek This message has been checked for viruses and spam by the Virgin Money email scanning system powered by Messagelabs. This e-mail is intended to be confidential to the recipient. If you receive a copy in error, please inform the sender and then delete this message. Virgin Money plc - Registered in England and Wales (Company no. 6952311). Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL. Virgin Money plc is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. The following companies also trade as Virgin Money. They are both authorised and regulated by the Financial Conduct Authority, are registered in England and Wales and have their registered office at Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited (Company no. 3072766) and Virgin Money Unit Trust Managers Limited (Company no. 3000482). For further details of Virgin Money group companies please visit our website at virginmoney.com -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Logging: IPA to Rsyslog to Logstash
-Original Message- > From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Dmitri Pal > Sent: 20 December 2014 03:37 > To: freeipa-users@redhat.com > Subject: Re: [Freeipa-users] Logging: IPA to Rsyslog to Logstash > > On 12/19/2014 11:35 AM, Innes, Duncan wrote: > > > > > Is it feasible to alter the timestamp resolution that dirsrv uses? > > This would help separate log lines properly. > > Please file a 389 RFE. > Done: https://fedorahosted.org/389/ticket/47982 > > -- > Thank you, > Dmitri Pal > > Sr. Engineering Manager IdM portfolio > Red Hat, Inc. > > -- This message has been checked for viruses and spam by the Virgin Money email scanning system powered by Messagelabs. This e-mail is intended to be confidential to the recipient. If you receive a copy in error, please inform the sender and then delete this message. Virgin Money plc - Registered in England and Wales (Company no. 6952311). Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL. Virgin Money plc is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. The following companies also trade as Virgin Money. They are both authorised and regulated by the Financial Conduct Authority, are registered in England and Wales and have their registered office at Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited (Company no. 3072766) and Virgin Money Unit Trust Managers Limited (Company no. 3000482). For further details of Virgin Money group companies please visit our website at virginmoney.com -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Logging: IPA to Rsyslog to Logstash
Hello Duncan,
thank you for doing this!
Could you transform this post to
http://www.freeipa.org/page/HowTos#Working_with_FreeIPA article, please? I
think that other people could use that too.
Thank you very much.
Petr^2 Spacek
On 19.12.2014 17:35, Innes, Duncan wrote:
> Earlier this year I said I'd feed back how my IPA to Rsyslog to Logstash
> experiments went.
>
> They went badly. And I didn't get much time. Today, however, I managed
> to get over my imaginary finishing line:
>
> All systems are RHEL 6.6.
>
> Rsyslog (rsyslog7-7.4.10) is configured to import logs from some dirsrv
> files:
>
> # cat /etc/rsyslog.d/dirsrv.conf
> module(load="imfile" PollingInterval="2")
>
> input(type="imfile"
> File="/var/log/dirsrv/slapd-EXAMPLE-COM/access"
> Tag="dirsrv"
> StateFile="statedirsrv"
> Facility="local0")
>
> input(type="imfile"
> File="/var/log/dirsrv/slapd-EXAMPLE-COM/errors"
> Tag="dirsrv"
> StateFile="statedirsrverr"
> Severity="error"
> Facility="local0")
>
> #
>
> This pulls in those log entries on a regular basis. Rsyslog8 allows you
> to use inotify for file changes, but that's not available to me.
>
> Rsyslog is then also configured to push all logs to my Logstash servers:
>
> # cat /etc/rsyslog.d/logstash.conf
> template(name="ls_json" type="list" option.json="on")
> { constant(value="{")
> constant(value="\"@timestamp\":\"") property(name="timegenerated"
> dateFormat="rfc3339")
> constant(value="\",\"@version\":\"1")
> constant(value="\",\"message\":\"") property(name="msg")
> constant(value="\",\"host\":\"") property(name="hostname")
> constant(value="\",\"my_environment\":\"dev")
> constant(value="\",\"my_project\":\"Infrastructure")
> constant(value="\",\"my_use\":\"IPA")
> constant(value="\",\"logsource\":\"") property(name="fromhost")
> constant(value="\",\"severity_label\":\"")
> property(name="syslogseverity-text")
> constant(value="\",\"severity\":\"") property(name="syslogseverity")
> constant(value="\",\"facility_label\":\"")
> property(name="syslogfacility-text")
> constant(value="\",\"facility\":\"") property(name="syslogfacility")
> constant(value="\",\"program\":\"") property(name="programname")
> constant(value="\",\"pid\":\"") property(name="procid")
> constant(value="\",\"rawmsg\":\"") property(name="rawmsg")
> constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
> constant(value="\"}\n")
> }
>
> *.* @@logstash01.example.com:5500;ls_json
> $ActionExecOnlyWhenPreviousIsSuspended on
> & @@logstash02.example.com:5500;ls_json
> & /var/log/localbuffer
> $ActionExecOnlyWhenPreviousIsSuspended off
>
> [root@lvdlvldap02 ~]#
>
> Which pushes all logs to my logstash servers in JSON format. Failover
> is built in by using 2 logstash servers.
> The client needs to have SELinux managed to allow rsyslog to write to
> port 5500:
>
> # semanage port -a -t syslogd_port_t -p tcp 5500
> # semanage port -l | grep 5500
>
> The Logstash servers are then configured to listen on this port and do
> some simple groking, before sending everything to the ElasticSearch
> cluster:
>
> # cat /etc/logstash/conf.d/syslog.conf
> input {
> tcp {
> type => syslogjson
> port => 5500
> codec => "json"
> }
> }
>
> filter {
> # This replaces the host field (UDP source) with the host that
> generated the message (sysloghost)
> if [sysloghost] {
> mutate {
> replace => [ "host", "%{sysloghost}" ]
> remove_field => "sysloghost" # prune the field after successfully
> replacing "host"
> }
> }
> if [type] == "syslogjson" {
> grok {
> patterns_dir => "/opt/logstash/patterns"
> match => { "message" => "%{VIRGINFW}" }
> match => { "message" => "%{AUDITAVC}" }
> match => { "message" => "%{COMMONAPACHELOG}" }
> tag_on_failure => []
> }
> }
>
> # This filter populates the @timestamp field with the timestamp that's
> in the actual message
> # dirsrv logs are currently pulled in every 2 minutes, so @timestamp
> is wrong
> if [syslogtag] == "dirsrv" {
> mutate {
> remove_field => [ 'rawmsg' ]
> }
> grok {
> match => [ "message", "%{HTTPDATE:log_timestamp}" ]
> }
> date {
> match => [ "log_timestamp", "dd/MMM/YYY:HH:mm:ss Z"]
> locale => "en"
> remove_field => [ "log_timestamp" ]
> }
> }
> }
>
> output {
> elasticsearch {
> protocol => node
> node_name => "Indexer01"
> }
> }
> #
>
> It works well for the most part. I'm not performing any groking of the
> actual message line as yet to pull out various bits of data into their
> own separate fields, but at least I'm managing to log the access and
> errors from multiple IPA servers.
>
> The @timestamp field ends up with the timestamp from the actual message
> line, so it's only down to second accuracy. This means that multiple
> log lines on the same second lose their ordering when viewed in the
> Logstash/Kibana interface. But the impo
Re: [Freeipa-users] Logging: IPA to Rsyslog to Logstash
On 12/19/2014 11:35 AM, Innes, Duncan wrote:
Earlier this year I said I'd feed back how my IPA to Rsyslog to Logstash
experiments went.
They went badly. And I didn't get much time. Today, however, I managed
to get over my imaginary finishing line:
All systems are RHEL 6.6.
Rsyslog (rsyslog7-7.4.10) is configured to import logs from some dirsrv
files:
# cat /etc/rsyslog.d/dirsrv.conf
module(load="imfile" PollingInterval="2")
input(type="imfile"
File="/var/log/dirsrv/slapd-EXAMPLE-COM/access"
Tag="dirsrv"
StateFile="statedirsrv"
Facility="local0")
input(type="imfile"
File="/var/log/dirsrv/slapd-EXAMPLE-COM/errors"
Tag="dirsrv"
StateFile="statedirsrverr"
Severity="error"
Facility="local0")
#
This pulls in those log entries on a regular basis. Rsyslog8 allows you
to use inotify for file changes, but that's not available to me.
Rsyslog is then also configured to push all logs to my Logstash servers:
# cat /etc/rsyslog.d/logstash.conf
template(name="ls_json" type="list" option.json="on")
{ constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timegenerated"
dateFormat="rfc3339")
constant(value="\",\"@version\":\"1")
constant(value="\",\"message\":\"") property(name="msg")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"my_environment\":\"dev")
constant(value="\",\"my_project\":\"Infrastructure")
constant(value="\",\"my_use\":\"IPA")
constant(value="\",\"logsource\":\"") property(name="fromhost")
constant(value="\",\"severity_label\":\"")
property(name="syslogseverity-text")
constant(value="\",\"severity\":\"") property(name="syslogseverity")
constant(value="\",\"facility_label\":\"")
property(name="syslogfacility-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility")
constant(value="\",\"program\":\"") property(name="programname")
constant(value="\",\"pid\":\"") property(name="procid")
constant(value="\",\"rawmsg\":\"") property(name="rawmsg")
constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
constant(value="\"}\n")
}
*.* @@logstash01.example.com:5500;ls_json
$ActionExecOnlyWhenPreviousIsSuspended on
& @@logstash02.example.com:5500;ls_json
& /var/log/localbuffer
$ActionExecOnlyWhenPreviousIsSuspended off
[root@lvdlvldap02 ~]#
Which pushes all logs to my logstash servers in JSON format. Failover
is built in by using 2 logstash servers.
The client needs to have SELinux managed to allow rsyslog to write to
port 5500:
# semanage port -a -t syslogd_port_t -p tcp 5500
# semanage port -l | grep 5500
The Logstash servers are then configured to listen on this port and do
some simple groking, before sending everything to the ElasticSearch
cluster:
# cat /etc/logstash/conf.d/syslog.conf
input {
tcp {
type => syslogjson
port => 5500
codec => "json"
}
}
filter {
# This replaces the host field (UDP source) with the host that
generated the message (sysloghost)
if [sysloghost] {
mutate {
replace => [ "host", "%{sysloghost}" ]
remove_field => "sysloghost" # prune the field after successfully
replacing "host"
}
}
if [type] == "syslogjson" {
grok {
patterns_dir => "/opt/logstash/patterns"
match => { "message" => "%{VIRGINFW}" }
match => { "message" => "%{AUDITAVC}" }
match => { "message" => "%{COMMONAPACHELOG}" }
tag_on_failure => []
}
}
# This filter populates the @timestamp field with the timestamp that's
in the actual message
# dirsrv logs are currently pulled in every 2 minutes, so @timestamp
is wrong
if [syslogtag] == "dirsrv" {
mutate {
remove_field => [ 'rawmsg' ]
}
grok {
match => [ "message", "%{HTTPDATE:log_timestamp}" ]
}
date {
match => [ "log_timestamp", "dd/MMM/YYY:HH:mm:ss Z"]
locale => "en"
remove_field => [ "log_timestamp" ]
}
}
}
output {
elasticsearch {
protocol => node
node_name => "Indexer01"
}
}
#
It works well for the most part. I'm not performing any groking of the
actual message line as yet to pull out various bits of data into their
own separate fields, but at least I'm managing to log the access and
errors from multiple IPA servers.
The @timestamp field ends up with the timestamp from the actual message
line, so it's only down to second accuracy. This means that multiple
log lines on the same second lose their ordering when viewed in the
Logstash/Kibana interface. But the important thing at this point is
that they're now held centrally.
Is it feasible to alter the timestamp resolution that dirsrv uses? This
would help separate log lines properly.
Please file a 389 RFE.
Cheers & Merry Festive Holiday thing
Duncan
This message has been checked for viruses and spam by the Virgin Money email
scanning system powered by Messagelabs.
This e-mail is intended to be confidential to the recipient. If you receive a
copy in error, pleas
[Freeipa-users] Logging: IPA to Rsyslog to Logstash
Earlier this year I said I'd feed back how my IPA to Rsyslog to Logstash
experiments went.
They went badly. And I didn't get much time. Today, however, I managed
to get over my imaginary finishing line:
All systems are RHEL 6.6.
Rsyslog (rsyslog7-7.4.10) is configured to import logs from some dirsrv
files:
# cat /etc/rsyslog.d/dirsrv.conf
module(load="imfile" PollingInterval="2")
input(type="imfile"
File="/var/log/dirsrv/slapd-EXAMPLE-COM/access"
Tag="dirsrv"
StateFile="statedirsrv"
Facility="local0")
input(type="imfile"
File="/var/log/dirsrv/slapd-EXAMPLE-COM/errors"
Tag="dirsrv"
StateFile="statedirsrverr"
Severity="error"
Facility="local0")
#
This pulls in those log entries on a regular basis. Rsyslog8 allows you
to use inotify for file changes, but that's not available to me.
Rsyslog is then also configured to push all logs to my Logstash servers:
# cat /etc/rsyslog.d/logstash.conf
template(name="ls_json" type="list" option.json="on")
{ constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timegenerated"
dateFormat="rfc3339")
constant(value="\",\"@version\":\"1")
constant(value="\",\"message\":\"") property(name="msg")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"my_environment\":\"dev")
constant(value="\",\"my_project\":\"Infrastructure")
constant(value="\",\"my_use\":\"IPA")
constant(value="\",\"logsource\":\"") property(name="fromhost")
constant(value="\",\"severity_label\":\"")
property(name="syslogseverity-text")
constant(value="\",\"severity\":\"") property(name="syslogseverity")
constant(value="\",\"facility_label\":\"")
property(name="syslogfacility-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility")
constant(value="\",\"program\":\"") property(name="programname")
constant(value="\",\"pid\":\"") property(name="procid")
constant(value="\",\"rawmsg\":\"") property(name="rawmsg")
constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
constant(value="\"}\n")
}
*.* @@logstash01.example.com:5500;ls_json
$ActionExecOnlyWhenPreviousIsSuspended on
& @@logstash02.example.com:5500;ls_json
& /var/log/localbuffer
$ActionExecOnlyWhenPreviousIsSuspended off
[root@lvdlvldap02 ~]#
Which pushes all logs to my logstash servers in JSON format. Failover
is built in by using 2 logstash servers.
The client needs to have SELinux managed to allow rsyslog to write to
port 5500:
# semanage port -a -t syslogd_port_t -p tcp 5500
# semanage port -l | grep 5500
The Logstash servers are then configured to listen on this port and do
some simple groking, before sending everything to the ElasticSearch
cluster:
# cat /etc/logstash/conf.d/syslog.conf
input {
tcp {
type => syslogjson
port => 5500
codec => "json"
}
}
filter {
# This replaces the host field (UDP source) with the host that
generated the message (sysloghost)
if [sysloghost] {
mutate {
replace => [ "host", "%{sysloghost}" ]
remove_field => "sysloghost" # prune the field after successfully
replacing "host"
}
}
if [type] == "syslogjson" {
grok {
patterns_dir => "/opt/logstash/patterns"
match => { "message" => "%{VIRGINFW}" }
match => { "message" => "%{AUDITAVC}" }
match => { "message" => "%{COMMONAPACHELOG}" }
tag_on_failure => []
}
}
# This filter populates the @timestamp field with the timestamp that's
in the actual message
# dirsrv logs are currently pulled in every 2 minutes, so @timestamp
is wrong
if [syslogtag] == "dirsrv" {
mutate {
remove_field => [ 'rawmsg' ]
}
grok {
match => [ "message", "%{HTTPDATE:log_timestamp}" ]
}
date {
match => [ "log_timestamp", "dd/MMM/YYY:HH:mm:ss Z"]
locale => "en"
remove_field => [ "log_timestamp" ]
}
}
}
output {
elasticsearch {
protocol => node
node_name => "Indexer01"
}
}
#
It works well for the most part. I'm not performing any groking of the
actual message line as yet to pull out various bits of data into their
own separate fields, but at least I'm managing to log the access and
errors from multiple IPA servers.
The @timestamp field ends up with the timestamp from the actual message
line, so it's only down to second accuracy. This means that multiple
log lines on the same second lose their ordering when viewed in the
Logstash/Kibana interface. But the important thing at this point is
that they're now held centrally.
Is it feasible to alter the timestamp resolution that dirsrv uses? This
would help separate log lines properly.
Cheers & Merry Festive Holiday thing
Duncan
This message has been checked for viruses and spam by the Virgin Money email
scanning system powered by Messagelabs.
This e-mail is intended to be confidential to the recipient. If you receive a
copy in error, please inform the sender and then delete this message.
Virgin Money plc - Registered in England and Wales (Company no. 69523
