On 08/30/2013 01:31 PM, John Moyer wrote:
Rob or anyone else,

So while struggling along on this server I just grabbed the logs off it and ran that log program with the options you suggested. There are a lot of unindexed requests. These are the top issues I've removed the one username that showed up.

So just to double check what I'm thinking. I need to create three indexes
1. objectclass pres
No, do not create this one
2. objectclass eq
This should already be indexed
3. uid pres
I suppose the UI might be doing this search?

Please let me know if I'm reading this correctly or if I'm way off?


7337            (objectclass=inetorgperson)
4597            (objectclass=*)
4560  (&(objectclass=inetorgperson)(uid=senior.developer.login))
307             (objectclass=krbticketpolicyaux)
292             (uid=*)



Thanks,
_____________________________________________________
John Moyer
Director, IT Operations
*Digital Reasoning Systems, Inc.*
john.mo...@digitalreasoning.com <mailto:john.mo...@digitalreasoning.com>
Office:703.678.2311
Mobile:240.460.0023
Fax:703.678.2312
www.digitalreasoning.com <http://www.digitalreasoning.com/>

On Aug 28, 2013, at 11:40 AM, Rob Crittenden <rcrit...@redhat.com <mailto:rcrit...@redhat.com>> wrote:

John Moyer wrote:
So this method of search logs is great, and it shows some indexes that would likely highly increase efficiency with my usage. So, are there instructions how to do that? or do you know off hand how to do that?

I'd start with https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes

Note that you'll want to create the same index on all hosts. This configuration is not replicated.

You can see the ones we create in /usr/share/ipa/indices.ldif and /usr/share/ipa/updates/20-indices.update

rob



Thanks,
_____________________________________________________
John Moyer
Director, IT Operations
Digital Reasoning Systems, Inc.
john.mo...@digitalreasoning.com <mailto:john.mo...@digitalreasoning.com>
Office:703.678.2311
Mobile:240.460.0023
Fax:703.678.2312
www.digitalreasoning.com

On Aug 27, 2013, at 4:45 PM, Rob Crittenden <rcrit...@redhat.com> wrote:

John Moyer wrote:
Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output?

There are a slew of options you can provide to logconv.pl. I typically use logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis.

rob




Start of Log:    27/Aug/2013:02:36:08
End of Log:      27/Aug/2013:12:17:15

Processed Log Time:  9 Hours, 41 Minutes, 7 Seconds

Restarts:                     2
Total Connections:            45224
SSL Connections:              44735
Peak Concurrent Connections:  76
Total Operations:             132568
Total Results:                132737
Overall Performance:          100.0%

Searches:                     61318      (1.76/sec)  (105.52/min)
Modifications:                277        (0.01/sec)  (0.48/min)
Adds:                         10         (0.00/sec)  (0.02/min)
Deletes:                      12         (0.00/sec)  (0.02/min)
Mod RDNs:                     0          (0.00/sec)  (0.00/min)
Compares:                     0          (0.00/sec)  (0.00/min)
Binds:                        62143      (1.78/sec)  (106.94/min)

Proxied Auth Operations:      0
Persistent Searches:          3
Internal Operations:          0
Entry Operations:             0
Extended Operations:          8808
Abandoned Requests:           0
Smart Referrals Received:     0

VLV Operations:               0
VLV Unindexed Searches:       0
SORT Operations:              353

Entire Search Base Queries:   106
Unindexed Searches:           319

FDs Taken:                    45262
FDs Returned:                 45210
Highest FD Taken:             139

Broken Pipes:                 0
Connections Reset By Peer:    0
Resource Unavailable:         0

Binds:                        62143
Unbinds:                      44539

 LDAP v2 Binds:               2
 LDAP v3 Binds:               62141
 SSL Client Binds:            0
 Failed SSL Client Binds:     0
 SASL Binds:                  1466
  1458  GSSAPI
  8     EXTERNAL

 Directory Manager Binds:     10
 Anonymous Binds:             1476
 Other Binds:                 60657





Thanks,
_____________________________________________________
John Moyer
Director, IT Operations
On Aug 27, 2013, at 1:13 PM, Rob Crittenden <rcrit...@redhat.com> wrote:

John Moyer wrote:
Is there any way to see what fields are index'ed?

$ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config'

Your best bet is to use the logconv.pl tool to examine your logs.

rob


Thanks,
_____________________________________________________
John Moyer
Director, IT Operations
Digital Reasoning Systems, Inc.
john.mo...@digitalreasoning.com
Office:703.678.2311
Mobile:240.460.0023
Fax:703.678.2312
www.digitalreasoning.com

On Aug 27, 2013, at 10:36 AM, John Moyer <john.mo...@digitalreasoning.com> wrote:

That looks like the output I just got shown below:


dn: cn=mapping tree,cn=config

dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config

dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config

dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\
2Cdc\3Dcom,cn=mapping tree,cn=config
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
uccessfulauth krblastfailedauth krbloginfailedcount


Thanks,
_____________________________________________________
John Moyer
Director, IT Operations


On Aug 27, 2013, at 10:14 AM, Rob Crittenden <rcrit...@redhat.com> wrote:

John Moyer wrote:
Ok, so we tried to implement this again, and as soon as we put on a server that authenticates heavily the IPA came to it's knees again. This time I was able to watch it closely and try to troubleshoot a lot more, and also know exactly what server caused it (Mercurial with help of bamboo). This runs fine on a normal old openldap servers. The user is logging in very quickly and each time it logs in I can see in the logs that the krbLastsuccessfullogin parameter (or whatever it is
called) is updated over and over and over in the changelog
(/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY
quickly and then disappear fairly quickly as well.

Issue 1: This is causing severe disk latency which obviously slows
everything down wait times were around 25%+
Issue 2: These changes need to be replicated to my slave server thus
adding to the mess


My question is, why does the IPA server fail to keep up with the load
when the openLDAP server didn't have an issue.   Indexes?


I'm running the following:

CentOS release 6.4 (Final)
389-ds-base-1.2.11.15-20.el6_4.x86_64
389-ds-base-libs-1.2.11.15-20.el6_4.x86_64
ipa-python-3.0.0-26.el6_4.4.x86_64
ipa-admintools-3.0.0-26.el6_4.4.x86_64
ipa-pki-common-theme-9.0.3-7.el6.noarch
python-iniparse-0.3.1-2.1.el6.noarch
ipa-server-3.0.0-26.el6_4.4.x86_64
ipa-pki-ca-theme-9.0.3-7.el6.noarch
ipa-server-selinux-3.0.0-26.el6_4.4.x86_64
libipa_hbac-1.9.2-82.7.el6_4.x86_64
ipa-client-3.0.0-26.el6_4.4.x86_64
libipa_hbac-python-1.9.2-82.7.el6_4.x86_64


So I've implemented this server anyway (against my better judgement with these issues and just made the user that logs into mercurial a local
user instead of IPA).

Also note before I did that for fun I implemented a RAM disk to put the change logs on, and that dropped the wait time to 0 (except bursts where it would raise to 30 to write the access log) but the CPU drove to 100% trying to keep up with the load. I have also killed the replication as
well.

Any help would be appreciated.


krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog.

You can confirm that they are excluded by searching the agreements:

$ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal

They should look like:

nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount

nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount

rob









_______________________________________________
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users

Reply via email to