Re: [Freeipa-users] Setting up IPA to log remotely
Kibana just renders the data, so I have no specific configuration for
that.
My logstash config (mostly cribbed from logstash.net) is as follows:
/etc/logstash/conf.d/syslog.conf
Containing:
input {
syslog {
type => syslog
port => 5544
}
udp {
type => syslogjson
port => 5500
codec => "json"
}
}
filter {
# This replaces the host field (UDP source) with the host that
generated the message (sysloghost)
if [sysloghost] {
mutate {
replace => [ "host", "%{sysloghost}" ]
remove_field => "sysloghost" # prune the field after successfully
replacing "host"
}
}
}
output {
elasticsearch {
protocol => node
node_name => "Indexer01"
}
}
This is my dev cluster which runs a logstash-1.4.1 RPM install
connecting to an elasticsearch cluster running on 3 workstations and a
laptop. The UDP connection is only used by a single client, so could be
ignored. This is the JSON sending that I referred to previously. Not
entirely successful so far.
On my "prod" system I've also managed to write some grok filters:
/etc/logstash.conf
input {
syslog {
type => syslog
port => 5544
}
}
filter {
if [type] == "syslog" {
grok {
patterns_dir => "/opt/logstash/patterns"
match => { "message" => "%{BESPOKFW}" }
match => { "message" => "%{AUDITAVC}" }
}
}
}
output {
elasticsearch {
embedded => true
template_overwrite => true
manage_template => false
}
}
With
/opt/logstash/patterns/bespokfw containing
NETFILTERMAC
%{COMMONMAC:dst_mac}:%{COMMONMAC:src_mac}:%{ETHTYPE:ethtype}
ETHTYPE (?:(?:[A-Fa-f0-9]{2}):(?:[A-Fa-f0-9]{2}))
IPTABLES1 (?:%{WORD:vmfw}: IN=(%{WORD:in_device})?
OUT=(%{WORD:out_device})? (MAC=%{NETFILTERMAC})?.*SRC=%{IP:src_ip}
DST=%{IP:dst_ip}.*PROTO=%{WORD:proto}?.*SPT=%{INT:src_port}?.*DPT=%{INT:
dst_port}?.*)
IPTABLES2 (?:%{WORD:vmfw}: IN=(%{WORD:in_device})?
OUT=(%{WORD:out_device})? (MAC=%{NETFILTERMAC})?.*SRC=%{IP:src_ip}
DST=%{IP:dst_ip}.*PROTO=%{INT:proto}?.*)
BESPOKFW (?:%{IPTABLES1}|%{IPTABLES2})
And
/opt/logstash/patterns/auditavc containing
AVCDEV (%{NUMBER:devmaj}:%{NUMBER:devmin})
AUDITAVC (?:type=%{WORD:audit_type}
audit\(%{NUMBER:audit_epoch}:%{NUMBER:audit_counter}\):
avc:\s*%{WORD:avc_action}\s*\{ %{WORD:avc_type} \}
for\s*pid=(%{NUMBER:avc_pid})? comm=\"(%{WORD:avc_comm})?\"
%{WORD:avc_class}=\"(%{NOTSPACE:avc_class_value})?\"(
dev=(%{AVCDEV:avc_dev})? ino=(%{NUMBER:avc_ino})?)?
scontext=(%{NOTSPACE:avc_scontext})?
tcontext=(%{NOTSPACE:avc_tcontext})? tclass=(%{WORD:avc_tclass})?)
This is running a tarball version of logstash (1.3.3 I think) with an
embedded elasticsearch instance.
Both work reasonably well. Am looking to bring more log data back at
the moment (i.e. application specific logs).
Cheers
Duncan
> -Original Message-
> From: Josh [mailto:joka...@gmail.com]
> Sent: 03 June 2014 11:54
> To: Innes, Duncan
> Cc: freeipa-users
> Subject: Re: [Freeipa-users] Setting up IPA to log remotely
>
>
> On Jun 3, 2014, at 4:37 AM, Innes, Duncan
> wrote:
>
> > I'm starting to log IPA to a central point too. I'd hoped
> the A part
> > of IPA would have arrived, but other functionality has
> pushed it down
> > the priority list. Would be good to see it arrive as something
> > integrated with systemd/journald with fully separated log fields
> > instead of a simple log text line.
> >
> > For now, rsyslog does a decent job of sending the logs over the
> > network and I'm using logstash to parse logs and pop them into
> > elasticsearch for analysing via Kibana. I've had most trouble with
> > the rsyslog side of things, but that's because I tried to
> get rsyslog
> > to send in JSON format rather than plain text. Once I
> reigned in my
> > ambition, it proved to be somewhat easier -
> >
>
> Any chance you could share your kibana configuration?
> > All I've added to RHEL6 client is a file
> /etc/rsyslog.d/logstash.conf
> > with contents:
> >
> > *.* @logstash.example.com:5544
> >
> > and (firewalls permitting) my logs end up at the logstash
> server for
> > parsing.
> >
> > Duncan
>
>
> -josh
>
> This message has been checked for viruses and spam by the
> Virgin Money email scanning system powered by Messagelabs.
>
This message has been checked for viruses and spam by the Virgin Money email
scanning system powered by Messagelabs.
This e-mail is intended to be confidential to the recipient. If you receive a
copy in error, please inform the sender and then delete this message.
Virgin Money p
Re: [Freeipa-users] Setting up IPA to log remotely
On Jun 3, 2014, at 4:37 AM, Innes, Duncan wrote: > I'm starting to log IPA to a central point too. I'd hoped the A part of > IPA would have arrived, but other functionality has pushed it down the > priority list. Would be good to see it arrive as something integrated > with systemd/journald with fully separated log fields instead of a > simple log text line. > > For now, rsyslog does a decent job of sending the logs over the network > and I'm using logstash to parse logs and pop them into elasticsearch for > analysing via Kibana. I've had most trouble with the rsyslog side of > things, but that's because I tried to get rsyslog to send in JSON format > rather than plain text. Once I reigned in my ambition, it proved to be > somewhat easier - > Any chance you could share your kibana configuration? > All I've added to RHEL6 client is a file /etc/rsyslog.d/logstash.conf > with contents: > > *.* @logstash.example.com:5544 > > and (firewalls permitting) my logs end up at the logstash server for > parsing. > > Duncan -josh ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Setting up IPA to log remotely
I'm starting to log IPA to a central point too. I'd hoped the A part of IPA would have arrived, but other functionality has pushed it down the priority list. Would be good to see it arrive as something integrated with systemd/journald with fully separated log fields instead of a simple log text line. For now, rsyslog does a decent job of sending the logs over the network and I'm using logstash to parse logs and pop them into elasticsearch for analysing via Kibana. I've had most trouble with the rsyslog side of things, but that's because I tried to get rsyslog to send in JSON format rather than plain text. Once I reigned in my ambition, it proved to be somewhat easier - All I've added to RHEL6 client is a file /etc/rsyslog.d/logstash.conf with contents: *.* @logstash.example.com:5544 and (firewalls permitting) my logs end up at the logstash server for parsing. Duncan > -Original Message- > From: freeipa-users-boun...@redhat.com > [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Brendan Kearney > Sent: 03 June 2014 03:26 > To: Steven Jones > Cc: freeipa-users@redhat.com > Subject: Re: [Freeipa-users] Setting up IPA to log remotely > > On Tue, 2014-06-03 at 00:42 +, Steven Jones wrote: > > Hi, > > > > I'll raise a request for this to be added then. > > > > Its a bit of an enterprise requirement feature that is of > use for us. > > > > Not having much luck with rsyslog and application logs at > the moment, good and accurate docs seem lacking for RHEL6. > > > > regards > > > > Steven > > ________ > > From: Rob Crittenden > > Sent: Tuesday, 3 June 2014 9:27 a.m. > > To: Steven Jones > > Cc: freeipa-users@redhat.com > > Subject: Re: [Freeipa-users] Setting up IPA to log remotely > > > > Steven Jones wrote: > > > Is there a way to get IPA to send its logs remotely? > > > > We intend to do something like this with audit, most likely > using the > > systemd journal, but it's a ways off. > > > > For now you'd need to do it manually on a per-service basis. I'd > > suggest looking at rsyslogd. You should be able to at least get the > > Apache and 389-ds logs using that. > > > > rob > > > > > > > > ___ > > Freeipa-users mailing list > > Freeipa-users@redhat.com > > https://www.redhat.com/mailman/listinfo/freeipa-users > > check out http://www.rsyslog.com/doc/master/index.html for > good and accurate docs. i am using fedora 16 and 20 with > RELP, fowarding syslog from everywhere to a central location, > and then dumping the logs into mysql. phplogcon bolts on top > of it for a web view of all the logs. > > on a sending source: > $ModLoad imuxsock # provides support for local system logging > (e.g. via logger command) $SystemLogRateLimitInterval 0 > $IMUXSockRateLimitInterval 0 > > $ModLoad imklog # provides kernel logging support > (previously done by > rklogd) > #$ModLoad immark # provides --MARK-- message capability > > # Provides UDP syslog reception > $ModLoad imudp > $UDPServerRun 514 > > # Provides TCP syslog reception > $ModLoad imtcp > $InputTCPServerRun 514 > > # Provides RELP transmission > $ModLoad omrelp > *.* :omrelp:192.168.25.1:20514;RSYSLOG_ForwardFormat > &~ > > on a receiving destination: > $ModLoad imuxsock # provides support for local system logging > (e.g. via logger command) $SystemLogRateLimitInterval 0 > $IMUXSockRateLimitInterval 0 > > $ModLoad imklog # provides kernel logging support > (previously done by > rklogd) > #$ModLoad immark # provides --MARK-- message capability > > # Provides UDP syslog reception > $ModLoad imudp > $UDPServerRun 514 > > # Provides TCP syslog reception > $ModLoad imtcp > $InputTCPServerRun 514 > > # Provides RELP reception > $ModLoad imrelp > $InputRELPServerRun 20514 > > # Provides MySQL connectivity > $ModLoad ommysql > # MASSIVE INSERT RATE FOR DB / SCALED DB LOGGING > $WorkDirectory /var/spool/rsyslog # default location for work > (spool) files $ActionQueueType LinkedList # use asynchronous > processing > $ActionQueueFileName dbq# set file name, also enables disk mode > $ActionResumeRetryCount -1 # infinite retries on insert > failure # for PostgreSQL replace :ommysql: by :ompgsql: below: > *.* :ommysql:server.domain.tld,Syslog,user,password > > > ___ > Freeipa-users mailing list > Freeipa-users@redhat.com > https://www.redhat.com/mailman/listinfo/f
Re: [Freeipa-users] Setting up IPA to log remotely
On Tue, 2014-06-03 at 00:42 +, Steven Jones wrote: > Hi, > > I'll raise a request for this to be added then. > > Its a bit of an enterprise requirement feature that is of use for us. > > Not having much luck with rsyslog and application logs at the moment, good > and accurate docs seem lacking for RHEL6. > > regards > > Steven > > From: Rob Crittenden > Sent: Tuesday, 3 June 2014 9:27 a.m. > To: Steven Jones > Cc: freeipa-users@redhat.com > Subject: Re: [Freeipa-users] Setting up IPA to log remotely > > Steven Jones wrote: > > Is there a way to get IPA to send its logs remotely? > > We intend to do something like this with audit, most likely using the > systemd journal, but it's a ways off. > > For now you'd need to do it manually on a per-service basis. I'd suggest > looking at rsyslogd. You should be able to at least get the Apache and > 389-ds logs using that. > > rob > > > > ___ > Freeipa-users mailing list > Freeipa-users@redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users check out http://www.rsyslog.com/doc/master/index.html for good and accurate docs. i am using fedora 16 and 20 with RELP, fowarding syslog from everywhere to a central location, and then dumping the logs into mysql. phplogcon bolts on top of it for a web view of all the logs. on a sending source: $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $SystemLogRateLimitInterval 0 $IMUXSockRateLimitInterval 0 $ModLoad imklog # provides kernel logging support (previously done by rklogd) #$ModLoad immark # provides --MARK-- message capability # Provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 # Provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 514 # Provides RELP transmission $ModLoad omrelp *.* :omrelp:192.168.25.1:20514;RSYSLOG_ForwardFormat &~ on a receiving destination: $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $SystemLogRateLimitInterval 0 $IMUXSockRateLimitInterval 0 $ModLoad imklog # provides kernel logging support (previously done by rklogd) #$ModLoad immark # provides --MARK-- message capability # Provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 # Provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 514 # Provides RELP reception $ModLoad imrelp $InputRELPServerRun 20514 # Provides MySQL connectivity $ModLoad ommysql # MASSIVE INSERT RATE FOR DB / SCALED DB LOGGING $WorkDirectory /var/spool/rsyslog # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq# set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure # for PostgreSQL replace :ommysql: by :ompgsql: below: *.* :ommysql:server.domain.tld,Syslog,user,password ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Setting up IPA to log remotely
Hi, I'll raise a request for this to be added then. Its a bit of an enterprise requirement feature that is of use for us. Not having much luck with rsyslog and application logs at the moment, good and accurate docs seem lacking for RHEL6. regards Steven From: Rob Crittenden Sent: Tuesday, 3 June 2014 9:27 a.m. To: Steven Jones Cc: freeipa-users@redhat.com Subject: Re: [Freeipa-users] Setting up IPA to log remotely Steven Jones wrote: > Is there a way to get IPA to send its logs remotely? We intend to do something like this with audit, most likely using the systemd journal, but it's a ways off. For now you'd need to do it manually on a per-service basis. I'd suggest looking at rsyslogd. You should be able to at least get the Apache and 389-ds logs using that. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Setting up IPA to log remotely
Steven Jones wrote: > Is there a way to get IPA to send its logs remotely? We intend to do something like this with audit, most likely using the systemd journal, but it's a ways off. For now you'd need to do it manually on a per-service basis. I'd suggest looking at rsyslogd. You should be able to at least get the Apache and 389-ds logs using that. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
