On 09/04/2013 08:53 AM, John Moyer wrote: > That summary is correct. The only thing I would add is that other > applications could easily bring the IPA server to it's knees as well.
Yes this is what I meant. It is not only JIRA. Any client that creates a lot of connections can cause problems. > Our artifact server also did many connections per sec when used, and > one person doing a build could bring IPA to it's knees as well. Also, > not only would IPA be maxed at 100%, but users would complain that > their builds were taking longer than normal (with or without the JIRA > sync going, however, it was obviously worse when JIRA was running). > > Also, my IPA server was a larger/faster server than my LDAP server. > So my LDAP server would run circles around IPA even though it was on a > smaller machine. LDAP would run at about 10% maybe 15% CPU when the > JIRA sync ran. > > IF you need any other information let me know. No this seems to be enough. Thank you. Would you be willing to test a fix if one is provided? Thanks Dmitri > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > > > On Sep 4, 2013, at 8:32 AM, Dmitri Pal <d...@redhat.com > <mailto:d...@redhat.com>> wrote: > >> On 09/04/2013 08:01 AM, John Moyer wrote: >>> Martin, >>> >>> I apologize there was a large offline conversation between Rich and >>> myself. Rich was kind enough to help me through some of my issues. >>> We did a lot more tests and poking and prodding. We discovered >>> that IPA is not as efficient when dealing with large number of >>> connections. Most of my load inefficiently reconnect to IPA over >>> and over and over and though LDAP can deal with this fairly >>> efficiently, IPA apparently drops to it's knees. >>> >>> A ticket was opened to addressed this issue. >>> >>> https://fedorahosted.org/freeipa/ticket/3892 >>> >>> >> >> Thank you for reporting this ticket. >> Martin is investigating it and trying to see what is the cause. The >> information mentioned above is missing from the the ticket, thus the >> question. >> >> So to summarize: you identified that the cause of the performance >> issue is that JIRA makes a lot of parallel connections to LDAP server >> and IPA is slow processing bind operations thus clients that do a lot >> of connections can experience a low performance. >> >> Martin, I wonder if we can have a test that would just do a lot of binds. >> There are a lot of plugins and one of the recent ones is the OTP one. >> I wonder if we do too much during bind when OTP is not enabled (by >> default). >> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> *Digital Reasoning Systems, Inc.* >>> john.mo...@digitalreasoning.com <mailto:john.mo...@digitalreasoning.com> >>> Office:703.678.2311 >>> Mobile:240.460.0023 >>> Fax:703.678.2312 >>> www.digitalreasoning.com <http://www.digitalreasoning.com/> >>> >>> On Sep 4, 2013, at 3:44 AM, Martin Kosek <mko...@redhat.com >>> <mailto:mko...@redhat.com>> wrote: >>> >>>> On 08/30/2013 11:08 PM, John Moyer wrote: >>>>> Well IPA has machine entries on some test clusters that I'm >>>>> rolling IPA >>>>> out on (20 machines maybe) but the user base is the same (about 80 >>>>> ~ 100) >>>>> accounts with maybe 40 to 50 groups? >>>>> >>>>> I've stood up a clone of the jira server along with IPA. I >>>>> cleared my >>>>> logs and then did the sync and ran the log analyzer on it. These >>>>> stats >>>>> are pretty much ONLY for that jira sync I don't have any other >>>>> connections >>>>> pointed to it. >>>>> >>>>> >>>>> Start of Log: 30/Aug/2013:15:57:13 End of Log: >>>>> 30/Aug/2013:16:01:14 >>>>> >>>>> Processed Log Time: Hours, 4 Minutes, 1 Seconds >>>>> >>>>> Restarts: 1 Total Connections: 824 SSL >>>>> Connections: 824 Peak Concurrent Connections: 6 Total >>>>> Operations: 1806 Total Results: 1805 >>>>> Overall >>>>> Performance: 99.9% >>>>> >>>>> Searches: 968 (4.02/sec) (241.00/min) >>>>> Modifications: 5 (0.02/sec) (1.24/min) Adds: >>>>> 0 (0.00/sec) (0.00/min) Deletes: 0 >>>>> (0.00/sec) (0.00/min) Mod RDNs: 0 >>>>> (0.00/sec) >>>>> (0.00/min) Compares: 0 (0.00/sec) >>>>> (0.00/min) Binds: 833 (3.46/sec) >>>>> (207.39/min) >>>>> >>>>> Proxied Auth Operations: 0 Persistent Searches: 1 >>>>> Internal >>>>> Operations: 0 Entry Operations: 0 Extended >>>>> Operations: 0 Abandoned Requests: 0 Smart Referrals >>>>> Received: 0 >>>>> >>>>> VLV Operations: 0 VLV Unindexed Searches: 0 SORT >>>>> Operations: 0 >>>>> >>>>> Entire Search Base Queries: 0 Unindexed Searches: 1 >>>>> >>>> >>>> This looks like a promising way to find out the reason, thanks >>>> John. However, >>>> I see just one unindexed search. Is the access log complete? >>>> Previously I see >>>> that the sync takes 900 seconds/15 minutes, but there is only 4 >>>> minutes the >>>> access log. Note that it it may take some time until the log is dumped. >>>> >>>> I think it would be also useful to run the analyzer with "-ula" >>>> flags as Rob >>>> suggested earlier to find out the unindexed searches (if any). >>>> >>>> What I find interesting is that JIRA does a lot of LDAP BINDs. Can the >>>> problem be in longer BINDs than with than expected (compared to for >>>> example >>>> plain LDAP servers)? Performance-wise, it would be I think better >>>> if JIRA >>>> does just one BIND and run all the LDAP searches the established >>>> connection. >>>> But I do not know if it can be configured this way. >>>> >>>> Rich, Rob, I am wondering if the slow up is not really caused by >>>> the binds, >>>> we have several DS plugins tied to the BIND operation, it may be >>>> useful to >>>> analyze if they do not take too long. >>>> >>>> Martin >>> >>> >>> >>> _______________________________________________ >>> Freeipa-users mailing list >>> Freeipa-users@redhat.com >>> https://www.redhat.com/mailman/listinfo/freeipa-users >> >> >> -- >> Thank you, >> Dmitri Pal >> >> Sr. Engineering Manager for IdM portfolio >> Red Hat Inc. >> >> >> ------------------------------- >> Looking to carve out IT costs? >> www.redhat.com/carveoutcosts/ >> >> >> _______________________________________________ >> Freeipa-users mailing list >> Freeipa-users@redhat.com <mailto:Freeipa-users@redhat.com> >> https://www.redhat.com/mailman/listinfo/freeipa-users > -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. ------------------------------- Looking to carve out IT costs? www.redhat.com/carveoutcosts/
_______________________________________________ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
