On 12/04/2014 11:01 AM, Janelle wrote:
To help understand the environment a bit - perhaps this will help.
1. Approx 7500 clients across 3 datacenters- all manor of *nix,
ranging from AIX, Linux, HP-UX and Solaris - hence the reason why
they all can't use ipa-client configs. Although that is in the
plan at least for Linux systems.
2. Servers/masters/replicas = 12 (4 each in 3 datacenters). Each DC
has a primary CA so the replication agreements are between the 3
servers in each DC plus 1 server is a hot-spare and used strictly
for replication to the other datacenters.
3. running out of FDs because there are 100s of jobs running across
all the servers that do a lot of sudo and group lookups and more
have to happen. Also, approx 1100 users accessing servers in vary
random ways - but just using ssh/pssh/other-tools.
This is what nscd/sssd/others are used for - to cache those client
connections/loads to keep the load off of the server.
Note that openldap uses epoll instead of poll() which is why it is
better able to scale to 65k connections. 389 works well in environments
where client caching is used. The use of 65k connections in 389 could
explain the high CPU load, but we would need stacktraces to determine
that (as in the aforementioned Debugging_Hangs).
This still doesn't explain the crashing.
Not sure if this helps - but perhaps?
On 12/4/14 8:41 AM, Ludwig Krispenz wrote:
On 12/04/2014 04:56 PM, Janelle wrote:
just (pam)auth and nslcd
It was ported from a running OpenLDAP environment to IPA. Just
trying to do conversions in stages so as not to change too much all
at once. Thought I could go from OpenLDAP to IPA and just use the
backend of 389ds. Functionally it does work, but the load kills it.
Seems like FDs are a huge problem. But all the settings documented
don't see to resolve the magic:
/ Netscape Portable Runtime error -5971 (Process open FD table is
Shouldn't this increase file descriptors in conjunction with
/etc/sysconfig/dirsrv.systemd change? FS-limits across the OS are
set to 65535 - /etc/security/limits.conf, /proc, sysctl.conf --
everything but 389-ds itself. But I still can't get this to work,
although it does not give an error.
ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=config,cn=ldbm database,cn=plugins,cn=config
there are two problems
- how to increase the connetction table size in DS
I would suggest to replace nsslapd-conntablesize and restart the
server, the change is not dynamic
- why you are running out of file descriptors
you should query "cn=monitor" to check the effective tablesize and
the number of established connections
it could be a problem of clients not well behaving and not closing
connections. you could set a low value of nsslapd-idletimeout to get
rid of these connections
On 12/4/14 7:40 AM, Rob Crittenden wrote:
Dmitri Pal wrote:
On 12/04/2014 09:41 AM, Rich Megginson wrote:
On 12/04/2014 08:39 AM, Rich Megginson wrote:
On 12/04/2014 01:45 AM, Petr Spacek wrote:
On 4.12.2014 05:02, Janelle wrote:
Thanks -- still a bit strange that it did not show up on some
servers - vary
random and intermittent.
BTW - a bit of information others might find useful. If you try to
"LDAP" portion of IPA for authentication - rather than fulling
IPA client and using Kerberos - the servers running ds-389 do not
do well in
handling the load. In other words - a few hundred hosts trying to
via LDAP only will send CPU through the roof and crashes the slapd
That should not happen.
For crashes, we would need to look at some stack traces:
For situations when the CPU is through the roof, that is very similar
to debugging hangs:
Sorry, forgot to mention that since this is IPA you'll also need to
install the ipa-debuginfo and slapi-nis-debuginfo packages.
I would also add a question about your client configuration.
For example if you use SSSD with enumerate=true for your clients then
yes you will get your environment to the knees pretty quickly.
I assumed SSSD wasn't being used at all which begs the question: what
is? nss_ldap? Is nslcd being used?
What is hitting LDAP, only auth or something else (e.g. sudo, automount).
Since IPA is supposed to handle all options, I guess I am
On 12/3/14 2:56 PM, Dmitri Pal wrote:
On 12/03/2014 04:40 PM, Janelle wrote:
Here is a bit of baffling one on 4.0.5:
Replica install p11-kit???
This is a part of the DNSSEC set of packages.
Connection from master to replica is OK.
Connection check OK
p11-kit: ipa.p11-kit: x-public-key-info: invalid or unsupported
Configuring NTP daemon (ntpd)
[1/4]: stopping ntpd
[2/4]: writing configuration
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.
LDAP error: UNWILLING_TO_PERFORM
database is read-only
We need more information about your problem.
As always, please start with information requested on
/var/log/ipa*.log from affected replica will be invaluable (along
package version numbers [including p11-kit] and repo configuration).
Manage your subscription for the Freeipa-users mailing list:
Go To http://freeipa.org for more info on the project