Re: [Freeipa-devel] Suggestion for the A part of IPA
-Original Message-
From: Martin Kosek [mailto:mko...@redhat.com]
Sent: 05 June 2015 13:31
To: Innes, Duncan; freeipa-devel@redhat.com; Jakub Hrozek
Subject: Re: [Freeipa-devel] Suggestion for the A part of IPA
On 06/02/2015 10:29 AM, Innes, Duncan wrote:
Just a bit of a head's up and a refresh of this with perhaps some
new
data.
We have more now! I created
https://www.freeipa.org/page/Centralized_Logging
where you can read more about this POC project and even see demo
showing
what the current POC ELK server can do (with link to Docker server
container and sources of course).
Excellent stuff - I fumbled together the page linked in your Other
Resources section at the bottom.
Will be upgrading my configs to replicate what you've done with respect
to
pulling in extra log files.
Thoughts? I'm not saying they should always be paired, but that if
a
user designs a system with enough horse power, this piggy-backing
could work well.
Ah, interesting idea and measurement. We have not thought about this
kind
of architecture yet. What we did in our POC is to configure FreeIPA
clients and servers to send the logs directly to the logging server
which
was on completely different machine (container) than rest of the
infrastructure.
It may be an alternative scheme, to have FreeIPA server containing the
log processing and then forwarding further to other REK/ELK server and
clients simply forwarding the logs to the same server as where they
are
authenticating.
If all the log configuration is baked in
ipa-{server,replica,client}-install, it would be extremely easy to
integrate.
I was also thinking that this kind of setup would work well in a heavily
firewalled environment. We have both hardware and host-based iptables
firewalls across the estate. In our case, pairing the firewall rules
for logging to the IPA servers is much easier than creating new servers
and requesting separate rules for them. Every client need to talk to
the
IPA servers via the IPA ports after all. Adding in an extra port to the
firewall rule group for IPA isn't hard to maintain.
I am not sure if the authentication+logging binding would be that
easy,
nor that it belong together that much. SSSD would need to dynamically
export the address of the FreeIPA server it communicates with, maybe
similarly as it does with Kerberos
(http://linux.die.net/man/8/sssd_krb5_locator_plugin) - but that does
not seem as a good fit either.
No - perhaps not. Again I'm thinking more from my current situation.
We
were not given access to create _SRV_ records by the AD team, so we have
had to hard-code our IPA servers into the config files. i.e.
[domain/unix.example.com]
cache_credentials = True
krb5_realm = UNIX.EXAMPLE.COM
ipa_domain = unix.example.com
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ipa_hostname = client01.unix.example.com
chpass_provider = ipa
ipa_server = ipa01.unix.example.com, ipa02.unix.example.com
ldap_tls_cacert = /etc/ipa/ca.crt
We try and figure out which IPA servers we can see, then randomise the
order of available servers and put it in the ipa_server setting.
So relatively simple for our setup to copy this config into an rsyslog.d
config file. Less easy if you just use _srv_ in there.
Not sure how you'd do that to be fair. But if it's possible to parse
the data coming back from the DNS _SRV_ query, couldn't all the
potential
IPA servers be included for rsyslog failover?
If all my remote servers are down, my failover reverts to /dev/null, so
no data will be written to disk if I'm isolated.
That's for rsyslog of course. I'm also working on getting
systemd-journal-upload to send direct to logstash (hopefully with the
http input plugin).
In any case, CCing Jakub for reference.
Thanks,
Martin
This message has been checked for viruses and spam by the Virgin Money email
scanning system powered by Messagelabs.
This e-mail is intended to be confidential to the recipient. If you receive a
copy in error, please inform the sender and then delete this message.
Virgin Money plc - Registered in England and Wales (Company no. 6952311).
Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL.
Virgin Money plc is authorised by the Prudential Regulation Authority and
regulated by the Financial Conduct Authority and the Prudential Regulation
Authority.
The following companies also trade as Virgin Money. They are both authorised
and regulated by the Financial Conduct Authority, are registered in England and
Wales and have their registered office at Jubilee House, Gosforth, Newcastle
upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited (Company no.
3072766) and Virgin Money Unit Trust Managers Limited (Company no. 3000482).
For further details of Virgin Money group companies please visit our website at
virginmoney.com
--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman
Re: [Freeipa-devel] Suggestion for the A part of IPA
Just a bit of a head's up and a refresh of this with perhaps some new data. Good to hear :-) We recently also started investigating the Audit capabilities for (notice I write for and not in) IPA. You can check my initial nudge to the freeipa-users list, which was unfortunately with no reply: https://www.redhat.com/archives/freeipa-users/2015-March/msg00940.html First up, just got round to reading this Martin. Not sure how I missed it when it first came out as it's a strong area of interest for me. The main part of what this message is about is a big change I made to our logging recently. I added in 4 of our main production IPA servers (there are 8 in total, but 4 sit beyond firewalls that take more scrutiny for changes than I wanted for now). The 4 I've added, though, serve more clients I figure. The amount of log traffic to the pair of Logstash servers has now jumped from around 50k records/hour to around 250k. Doubtless this still doesn't push any of the parts to their limits, but there has been a barely noticeably increase in CPU usage on the 2 Logstash servers. We've gone from around 2% CPU usage to 4%. Since the CPU usage on our 'loudest' IPA server rarely peaks above 10%, this doesn't present nearly as much load as I had anticipated. I have run Logstash parsers on my DEV IPA boxes, but will now investigate running them on my Prod servers too. What I'm getting at is that perhaps clients sending logs back to the IPA servers for parsing, then being sent on to a central DB for storage, isn't going to break the bank performance-wise. All of the systems in question here are 2vCPU with 4Gb vRAM running on ESXi hosts, so nothing special in the performance arena. It strikes me as a reasonably elegant solution to pair the authentication and log parsing services on the same set of servers. This would allow each client to use the same servers/failover etc for SSSD as for rsyslog. There may, of course, be other considerations, but I'm suggesting that system load isn't necessarily one of them. Much as projects such as Katello can run with everything on the same server, or split out Postgres and the like onto separate servers when there are performance considerations. Thoughts? I'm not saying they should always be paired, but that if a user designs a system with enough horse power, this piggy-backing could work well. Cheers Duncan This message has been checked for viruses and spam by the Virgin Money email scanning system powered by Messagelabs. This e-mail is intended to be confidential to the recipient. If you receive a copy in error, please inform the sender and then delete this message. Virgin Money plc - Registered in England and Wales (Company no. 6952311). Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL. Virgin Money plc is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. The following companies also trade as Virgin Money. They are both authorised and regulated by the Financial Conduct Authority, are registered in England and Wales and have their registered office at Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited (Company no. 3072766) and Virgin Money Unit Trust Managers Limited (Company no. 3000482). For further details of Virgin Money group companies please visit our website at virginmoney.com -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Suggestion for the A part of IPA
On 28/04/2015 19:08 PM, Young, Adam wrote: I think I am in alignment with what you are saying. I like rsyslogd as the basic ship the log off the server tool. Let's use what the platform support first natively and formost; We want something native, not Ruby, not even Python if we can avoid it, for the normal case. Bumping up to logstash for more complex host-side rules might be fine. Remember, the Hosts side of integration with FreeIPA is sssd. Logstash can be the server side of the audit collection as well, and then it puts fewer demands on the server. We need to ensure that the audit data can be sent over a GSSAPI protected pathway. Absolutely - this is something I need to get round to. Concentrated on getting the data back and in a good state for a start. Figured I'd get round to securing stuff at a later point. On the IPA side, I would think we would register the audit server as a host, and have specific service entires for the protocols supported. Would you see IPA owning the audit server, or just integrating in with an existing one? I've built mine completely separately and then brought them closer together by running Logstash on the IPA server. Not sure if there should be an exclusive ownership going on. IPA could create an initial setup, but there's huge room for creating more complex systems that IPA might want to leave well alone. I don't think the IPA server itself should be the ELK server for obvious reasons. I would love to see the ELK server supported along the lines of how we do a replica setup. ELK is trivial to set up as a simple cluster. It gets more complicated to do it properly, but what I've got going at the moment fits along the trivial lines, but provides an extremely robust database. This message has been checked for viruses and spam by the Virgin Money email scanning system powered by Messagelabs. This e-mail is intended to be confidential to the recipient. If you receive a copy in error, please inform the sender and then delete this message. Virgin Money plc - Registered in England and Wales (Company no. 6952311). Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL. Virgin Money plc is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. The following companies also trade as Virgin Money. They are both authorised and regulated by the Financial Conduct Authority, are registered in England and Wales and have their registered office at Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited (Company no. 3072766) and Virgin Money Unit Trust Managers Limited (Company no. 3000482). For further details of Virgin Money group companies please visit our website at virginmoney.com -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] Suggestion for the A part of IPA
-Original Message- From: Martin Kosek [mailto:mko...@redhat.com] Sent: 29 April 2015 07:42 To: Innes, Duncan; freeipa-devel@redhat.com Subject: Re: [Freeipa-devel] Suggestion for the A part of IPA On 04/28/2015 05:58 PM, Innes, Duncan wrote: Folks, The A part of IPA has always been of great interest to me. Our current IPA infrastructure works well at the I P parts, giving us great failover abilities and connectivity through hardware firewalls without punching too many holes. Good to hear :-) We recently also started investigating the Audit capabilities for (notice I write for and not in) IPA. You can check my initial nudge to the freeipa-users list, which was unfortunately with no reply: https://www.redhat.com/archives/freeipa-users/2015-March/msg00940.html For the beginning, I would be interested for your use cases, if you are only looking for a centralized log store, or you are also looking for more analytics in the logs (like what API commands were run, user logins, etc.) or utilization of the server core services (LDAP/Kerberos/DNS/...) I'm actually interested in both pieces. It started off as generic centralised logging, but when I moved from plain rsyslog data to sending JSON formatted data I started getting richer and richer data. I got the IPA servers sending their dirsrv access and error logs back by using rsyslog's imfile module. http://www.freeipa.org/page/Howto/Centralised_Logging_with_Logstash/Elas ticSearch/Kibana This isn't too good, however, as imfile polls on an interval with my version of rsyslog7 and IPA logs are written only with HH:MM:SS level accuracy. (I'm sure there was a ticket to increase timestamp resolution, but I can't find it right now) But now I've got a load of the IPA logs back centrally, I'm beyond my own abilities in parsing much good stuff out of it. I haven't even parsed it out into any key:value pairs yet - which would be a big boost to ELK Searching on it. Whilst the A part may not be solely about centralised logging, it's the thing I've been looking into recently. To do this I've built a setup around the ELK stack using a pair of Logstash servers and an ElasticSearch cluster of 5 servers (overkill on the ES side perhaps, but this is proof of concept still). To expand on this, I've been looking at running the Logstash serviceon each of our IPA servers as that gives us a failover pair in each part of our network. The Logstash servers then connect to the ES cluster as non-data nodes. Each client has an rsyslog7 (still using RHEL6 at the moment) config that writes sends the logs in JSON format with some extra bespoke fields added (such as Project, Environment, and Use to help us search better). The sending is done in rsyslog's rather clunky failover method to the local pair of Logstash servers (with a third failover being to /dev/null). Ah, so you are running Logstash service on each IPA service? Isn't that too heavyweight? In our tests, we mostly simply wanted just configure rsyslog and get the logs out of the server, to the centralized ELK/REK/EFK servers which did the heavy lifting. After all, the IPA servers may be of different environments (Fedora, RHEL, CentOS, ...) and with different versions of the log processing software. The Logstash servers really don't produce much load on our systems. I expected them to, but it didn't materialise. The load may be a lighter due to me using rsyslog templates to send the logs pre-formatted as JSON? On the REK server (yes, we did not use logstash at the POC), we are able to process the logs (make them structured), store and display them. This allows us do searches like list of admins which added users in the last month. This if course required adding parsing rules to rsyslog to get the structure out of the API logs. Are you using logstash for the parsing or did you not start the parsing part yet? No parsing of the dirsrv logs yet. I've been concentrating on pushing the Whole centralised logging issue from PoC to Production so far. The IPA logs were a side-issue for us, but it's definitely something that would help if could get them properly structured. It struck me that this kind of setup might not be too far removed from some of the A part of IPA. The centralized log processing itself is a too big task for IPA itself, it is specialized on other things. But some integration should be added in time, I agree. It may be minimal, from top of my head for example: * Support of (rsyslog) configuration in ipa-client-install or ipa-server-install * Providing the secure, GSSAPI-based log transfer to the IPA clients and ELK/REK/EFK server * Providing parsing templates for rsyslog or base queries for Kibana That sounds the right way to start for sure. I've also been tracking journalctl's ability to send logs to an ELK setup. This is already looking to be a much simpler setup once a few
[Freeipa-devel] Suggestion for the A part of IPA
Folks, The A part of IPA has always been of great interest to me. Our current IPA infrastructure works well at the I P parts, giving us great failover abilities and connectivity through hardware firewalls without punching too many holes. Whilst the A part may not be solely about centralised logging, it's the thing I've been looking into recently. To do this I've built a setup around the ELK stack using a pair of Logstash servers and an ElasticSearch cluster of 5 servers (overkill on the ES side perhaps, but this is proof of concept still). To expand on this, I've been looking at running the Logstash serviceon each of our IPA servers as that gives us a failover pair in each part of our network. The Logstash servers then connect to the ES cluster as non-data nodes. Each client has an rsyslog7 (still using RHEL6 at the moment) config that writes sends the logs in JSON format with some extra bespoke fields added (such as Project, Environment, and Use to help us search better). The sending is done in rsyslog's rather clunky failover method to the local pair of Logstash servers (with a third failover being to /dev/null). It struck me that this kind of setup might not be too far removed from some of the A part of IPA. I'm not good at ASCII flowchart diagrams, so will leave it there for now. The main point of this - does any of this idea sound reasonable to add in to FreeIPA? To me it sounds like a good fit for getting (some) logging data back to a central point. The Logstash indexers currently have a very low load (perhaps due to the incoming data already being JSON) and small memory footprint. They run without issue on our IPA servers. The ES nodes are different and I won't pretent to be any sort of expert in what they do. They load up a bit when I shut 1 of them down, but that's the rebalancing happening. Apologies if this is off topic, or wide of the mark. Cheers Duncan Innes This message has been checked for viruses and spam by the Virgin Money email scanning system powered by Messagelabs. This e-mail is intended to be confidential to the recipient. If you receive a copy in error, please inform the sender and then delete this message. Virgin Money plc - Registered in England and Wales (Company no. 6952311). Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL. Virgin Money plc is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. The following companies also trade as Virgin Money. They are both authorised and regulated by the Financial Conduct Authority, are registered in England and Wales and have their registered office at Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited (Company no. 3072766) and Virgin Money Unit Trust Managers Limited (Company no. 3000482). For further details of Virgin Money group companies please visit our website at virginmoney.com-- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
