From: Martin Kosek [mailto:mko...@redhat.com]
Sent: 29 April 2015 07:42
To: Innes, Duncan; firstname.lastname@example.org
Subject: Re: [Freeipa-devel] Suggestion for the A part of IPA
> On 04/28/2015 05:58 PM, Innes, Duncan wrote:
> > Folks,
> > The A part of IPA has always been of great interest to me. Our
> > current IPA infrastructure works well at the I & P parts, giving us
> > great failover abilities and connectivity through hardware firewalls
> > without punching too many holes.
> Good to hear :-) We recently also started investigating the Audit
> capabilities for (notice I write "for" and not "in") IPA. You can
> my initial nudge to the freeipa-users list, which was unfortunately
> with no reply:
> For the beginning, I would be interested for your use cases, if you
> only looking for a centralized log store, or you are also looking for
> more analytics in the logs (like what API commands were run, user
> logins, etc.) or utilization of the server core services
I'm actually interested in both pieces. It started off as generic
centralised logging, but when I moved from plain rsyslog data to sending
JSON formatted data I started getting richer and richer data. I got the
IPA servers sending their dirsrv access and error logs back by using
rsyslog's imfile module.
This isn't too good, however, as imfile polls on an interval with my
version of rsyslog7 and IPA logs are written only with HH:MM:SS level
(I'm sure there was a ticket to increase timestamp resolution, but I
find it right now)
But now I've got a load of the IPA logs back centrally, I'm beyond my
abilities in parsing much good stuff out of it. I haven't even parsed
out into any key:value pairs yet - which would be a big boost to ELK
Searching on it.
> > Whilst the A part may not be solely about centralised logging, it's
> > the thing I've been looking into recently. To do this I've built a
> > setup around the ELK stack using a pair of Logstash servers and an
> > ElasticSearch cluster of 5 servers (overkill on the ES side perhaps,
> > but this is proof of concept still). To expand on this, I've been
> > looking at running the Logstash serviceon each of our IPA servers as
> > that gives us a failover pair in each part of our network. The
> > Logstash servers then connect to the ES cluster as non-data nodes.
> > Each client has an
> > rsyslog7 (still using RHEL6 at the moment) config that writes sends
> > the logs in JSON format with some extra bespoke fields added (such
> > Project, Environment, and Use to help us search better). The
> > is done in rsyslog's rather clunky failover method to the local pair
> > of Logstash servers (with a third failover being to /dev/null).
> Ah, so you are running Logstash service on each IPA service? Isn't
> too heavyweight? In our tests, we mostly simply wanted just configure
> rsyslog and get the logs out of the server, to the centralized
> ELK/REK/EFK servers which did the heavy lifting. After all, the IPA
> servers may be of different environments (Fedora, RHEL, CentOS, ...)
> and with different versions of the log processing software.
The Logstash servers really don't produce much load on our systems. I
expected them to, but it didn't materialise. The load may be a lighter
due to me using rsyslog templates to send the logs pre-formatted as
> On the REK server (yes, we did not use logstash at the POC), we are
> to process the logs (make them structured), store and display them.
> allows us do searches like "list of admins which added users in the
> This if course required adding parsing rules to rsyslog to get the
> structure out of the API logs. Are you using logstash for the parsing
> did you not start the parsing part yet?
No parsing of the dirsrv logs yet. I've been concentrating on pushing
Whole centralised logging issue from PoC to Production so far. The IPA
logs were a side-issue for us, but it's definitely something that would
help if could get them properly structured.
> > It struck me that this kind of setup might not be too far removed
> > some of the A part of IPA.
> The centralized log processing itself is a too big task for IPA
> it is specialized on other things. But some integration should be
> in time, I agree. It may be minimal, from top of my head for example:
> * Support of (rsyslog) configuration in ipa-client-install or
> * Providing the secure, GSSAPI-based log transfer to the IPA clients
> ELK/REK/EFK server
> * Providing parsing templates for rsyslog or base queries for Kibana
That sounds the right way to start for sure. I've also been tracking
journalctl's ability to send logs to an ELK setup. This is already
to be a much simpler setup once a few issues are cleared up.
> > I'm not good at ASCII flowchart diagrams, so will leave it there for
> > now. The main point of this - does any of this idea sound
> > to add in to FreeIPA? To me it sounds like a good fit for getting
> > (some) logging data back to a central point.
> > The Logstash indexers currently have a very low load (perhaps due to
> > the incoming data already being JSON) and small memory footprint.
> > They run without issue on our IPA servers. The ES nodes are
> > and I won't pretent to be any sort of expert in what they do. They
> > load up a bit when I shut 1 of them down, but that's the rebalancing
> > Apologies if this is off topic, or wide of the mark.
> > Cheers
> > Duncan Innes
This message has been checked for viruses and spam by the Virgin Money email
scanning system powered by Messagelabs.
This e-mail is intended to be confidential to the recipient. If you receive a
copy in error, please inform the sender and then delete this message.
Virgin Money plc - Registered in England and Wales (Company no. 6952311).
Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL.
Virgin Money plc is authorised by the Prudential Regulation Authority and
regulated by the Financial Conduct Authority and the Prudential Regulation
The following companies also trade as Virgin Money. They are both authorised
and regulated by the Financial Conduct Authority, are registered in England and
Wales and have their registered office at Jubilee House, Gosforth, Newcastle
upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited (Company no.
3072766) and Virgin Money Unit Trust Managers Limited (Company no. 3000482).
For further details of Virgin Money group companies please visit our website at
Manage your subscription for the Freeipa-devel mailing list:
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code