Sure, just let me know what needs to be run/applied.  I've already rolled back 
to LDAP, so if the fix looks like it works I can then roll it out again.

Thanks, 
_____________________________________________________
John Moyer
Director, IT Operations

On Sep 4, 2013, at 9:12 AM, Dmitri Pal <d...@redhat.com> wrote:

> On 09/04/2013 08:53 AM, John Moyer wrote:
>> 
>> That summary is correct.   The only thing I would add is that other 
>> applications could easily bring the IPA server to it's knees as well.  
> 
> Yes this is what I meant. It is not only JIRA. Any client that creates a lot 
> of connections can cause problems.
> 
>> Our artifact server also did many connections per sec when used, and one 
>> person doing a build could bring IPA to it's knees as well.  Also, not only 
>> would IPA be maxed at 100%, but users would complain that their builds were 
>> taking longer than normal (with or without the JIRA sync going, however, it 
>> was obviously worse when JIRA was running).   
>> 
>>  Also, my IPA server was a larger/faster server than my LDAP server.   So my 
>> LDAP server would run circles around IPA even though it was on a smaller 
>> machine. LDAP would run at about 10% maybe 15% CPU when the JIRA sync ran. 
>> 
>>  IF you need any other information let me know.
> 
> No this seems to be enough.
> Thank you.
> 
> Would you be willing to test a fix if one is provided?
> 
> Thanks
> Dmitri
> 
>> 
>> Thanks, 
>> _____________________________________________________
>> John Moyer
>> Director, IT Operations
>> 
>> 
>> On Sep 4, 2013, at 8:32 AM, Dmitri Pal <d...@redhat.com> wrote:
>> 
>>> On 09/04/2013 08:01 AM, John Moyer wrote:
>>>> 
>>>> Martin, 
>>>> 
>>>>  I apologize there was a large offline conversation between Rich and 
>>>> myself.   Rich was kind enough to help me through some of my issues.  We 
>>>> did a lot more tests and poking and prodding.   We discovered that IPA is 
>>>> not as efficient when dealing with large number of connections.  Most of 
>>>> my load inefficiently reconnect to IPA over and over and over and though 
>>>> LDAP can deal with this fairly efficiently, IPA apparently drops to it's 
>>>> knees.   
>>>> 
>>>>  A ticket was opened to addressed this issue.
>>>>  
>>>>  https://fedorahosted.org/freeipa/ticket/3892
>>>> 
>>>> 
>>> 
>>> Thank you for reporting this ticket.
>>> Martin is investigating it and trying to see what is the cause. The 
>>> information mentioned above is missing from the the ticket, thus the 
>>> question.
>>> 
>>> So to summarize: you identified that the cause of the performance issue is 
>>> that JIRA makes a lot of parallel connections to LDAP server and IPA is 
>>> slow processing bind operations thus clients that do a lot of connections 
>>> can experience a low performance.
>>> 
>>> Martin, I wonder if we can have a test that would just do a lot of binds.
>>> There are a lot of plugins and one of the recent ones is the OTP one. I 
>>> wonder if we do too much during bind when OTP is not enabled (by default).
>>> 
>>>> Thanks, 
>>>> _____________________________________________________
>>>> John Moyer
>>>> Director, IT Operations
>>>> Digital Reasoning Systems, Inc.
>>>> john.mo...@digitalreasoning.com
>>>> Office: 703.678.2311
>>>> Mobile: 240.460.0023
>>>> Fax: 703.678.2312
>>>> www.digitalreasoning.com
>>>> 
>>>> On Sep 4, 2013, at 3:44 AM, Martin Kosek <mko...@redhat.com> wrote:
>>>> 
>>>>> On 08/30/2013 11:08 PM, John Moyer wrote:
>>>>>> Well IPA has machine entries on some test clusters that I'm rolling IPA
>>>>>> out on (20 machines maybe) but the user base is the same (about 80 ~ 100)
>>>>>> accounts with maybe 40 to 50 groups?
>>>>>> 
>>>>>> I've stood up a clone of the jira server along with IPA.   I cleared my
>>>>>> logs and then did the sync and ran the log analyzer on it.   These stats
>>>>>> are pretty much ONLY for that jira sync I don't have any other 
>>>>>> connections
>>>>>> pointed to it.
>>>>>> 
>>>>>> 
>>>>>> Start of Log:    30/Aug/2013:15:57:13 End of Log:
>>>>>> 30/Aug/2013:16:01:14
>>>>>> 
>>>>>> Processed Log Time:   Hours, 4 Minutes, 1 Seconds
>>>>>> 
>>>>>> Restarts:                     1 Total Connections:            824 SSL
>>>>>> Connections:              824 Peak Concurrent Connections:  6 Total
>>>>>> Operations:             1806 Total Results:                1805 Overall
>>>>>> Performance:          99.9%
>>>>>> 
>>>>>> Searches:                     968        (4.02/sec)  (241.00/min) 
>>>>>> Modifications:                5          (0.02/sec)  (1.24/min) Adds:
>>>>>> 0          (0.00/sec)  (0.00/min) Deletes:                      0
>>>>>> (0.00/sec)  (0.00/min) Mod RDNs:                     0          
>>>>>> (0.00/sec)
>>>>>> (0.00/min) Compares:                     0          (0.00/sec)
>>>>>> (0.00/min) Binds:                        833        (3.46/sec)
>>>>>> (207.39/min)
>>>>>> 
>>>>>> Proxied Auth Operations:      0 Persistent Searches:          1 Internal
>>>>>> Operations:          0 Entry Operations:             0 Extended
>>>>>> Operations:          0 Abandoned Requests:           0 Smart Referrals
>>>>>> Received:     0
>>>>>> 
>>>>>> VLV Operations:               0 VLV Unindexed Searches:       0 SORT
>>>>>> Operations:              0
>>>>>> 
>>>>>> Entire Search Base Queries:   0 Unindexed Searches:           1
>>>>>> 
>>>>> 
>>>>> This looks like a promising way to find out the reason, thanks John. 
>>>>> However,
>>>>> I see just one unindexed search. Is the access log complete? Previously I 
>>>>> see
>>>>> that the sync takes 900 seconds/15 minutes, but there is only 4 minutes 
>>>>> the
>>>>> access log. Note that it it may take some time until the log is dumped.
>>>>> 
>>>>> I think it would be also useful to run the analyzer with "-ula" flags as 
>>>>> Rob
>>>>> suggested earlier to find out the unindexed searches (if any).
>>>>> 
>>>>> What I find interesting is that JIRA does a lot of LDAP BINDs. Can the
>>>>> problem be in longer BINDs than with than expected (compared to for 
>>>>> example
>>>>> plain LDAP servers)? Performance-wise, it would be I think better if JIRA
>>>>> does just one BIND and run all the LDAP searches the established 
>>>>> connection.
>>>>> But I do not know if it can be configured this way.
>>>>> 
>>>>> Rich, Rob, I am wondering if the slow up is not really caused by the 
>>>>> binds,
>>>>> we have several DS plugins tied to the BIND operation, it may be useful to
>>>>> analyze if they do not take too long.
>>>>> 
>>>>> Martin
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Freeipa-users mailing list
>>>> Freeipa-users@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> 
>>> 
>>> -- 
>>> Thank you,
>>> Dmitri Pal
>>> 
>>> Sr. Engineering Manager for IdM portfolio
>>> Red Hat Inc.
>>> 
>>> 
>>> -------------------------------
>>> Looking to carve out IT costs?
>>> www.redhat.com/carveoutcosts/
>>> 
>>> 
>>> _______________________________________________
>>> Freeipa-users mailing list
>>> Freeipa-users@redhat.com
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> 
> 
> 
> -- 
> Thank you,
> Dmitri Pal
> 
> Sr. Engineering Manager for IdM portfolio
> Red Hat Inc.
> 
> 
> -------------------------------
> Looking to carve out IT costs?
> www.redhat.com/carveoutcosts/
> 
> 

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail

_______________________________________________
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users

Reply via email to