unsubscribe
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of
[email protected]
Sent: Tuesday, August 09, 2011 12:40 PM
To: [email protected]
Subject: EXT :Kerberos Digest, Vol 104, Issue 11
Send Kerberos mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://mailman.mit.edu/mailman/listinfo/kerberos
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Kerberos digest..."
Today's Topics:
1. Re: Performance issues with krb5-1.9.1 (Jonathan Reams)
2. Re: Performance issues with krb5-1.9.1 (Jonathan Reams)
3. Problem after adding new realm in krb5.conf (Sanket Sangodkar)
4. Error while adding new realm to krb5.conf (Sangodkar, Sanket)
5. Re: Error while adding new realm to krb5.conf (Vipin Rathor)
6. Re: Performance issues with krb5-1.9.1 (Chris Hecker)
----------------------------------------------------------------------
Message: 1
Date: Mon, 8 Aug 2011 15:49:26 -0400
From: Jonathan Reams <[email protected]>
Subject: Re: Performance issues with krb5-1.9.1
To: Greg Hudson <[email protected]>
Cc: "[email protected]" <[email protected]>
Message-ID: <[email protected]>
Content-Type: text/plain; charset=us-ascii
Hi Greg,
I applied this patch and saw a great improvement on our test KDC. I should
follow up and say that the performance degradation appeared to be compounded by
clients resubmitting their requests after they timed out, so the KDC wasn't
only handling new requests, it was trying to fulfill old requests that were
being re-submitted. Below is the results of my little performance test with the
patch applied to krb5-1.9.1.
[root@naiad ~]# for i in `seq 1 4`; do time for j in `seq 1 1000`; do kinit -k;
kvno iprop_monitor | grep -v kvno; done; echo finished round $i; sleep 5; done
real 0m20.312s
user 0m4.513s
sys 0m13.690s
finished round 1
real 0m20.842s
user 0m4.512s
sys 0m13.896s
finished round 2
real 0m21.123s
user 0m4.605s
sys 0m13.787s
finished round 3
real 0m21.425s
user 0m4.508s
sys 0m13.642s
finished round 4
This is much better than the 56 seconds we saw without the patch. We'll roll
this out to our secondary KDC and see how it goes this week. Thanks for
resolving this so quickly.
Jonathan Reams
CUIT Systems Engineering
Columbia University
On Aug 8, 2011, at 2:21 PM, Greg Hudson wrote:
>
> I found a regression which would affect these tests, but I'm not sure it
> accounts for your global performance issues.
>
> The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache,
> but due to a bug, it is actually opening and reading a replay cache for
> every TGS request, which is significantly less efficient than the 1.8
> behavior (using a replay cache which stays open for the lifetime of the
> KDC).
>
> In a test which runs in under five minutes, this regression produces
> visible O(n^2) performance characteristics. This would not necessarily
> account for performance degradation over hours, as the performance drag
> of the replay cache should become stable after five minutes. It's
> possible that the constant drag was enough to cause the KDC to fall
> behind on the request load, but it's also possible that there's a second
> problem which isn't so easily reproduced.
>
> I've attached a patch. Note that there is a second, in-memory
> "lookaside" cache with O(n^2) performance characteristics in the short
> term, which holds queries for up to two minutes. You may see a slight
> degradation in performance in test cases due to this. You can
> temporarily rebuild the kdc directory with "make clean;
> CPPFLAGS=-DNOCACHE" if you want to remove this variable from your
> performance tests.
------------------------------
Message: 2
Date: Tue, 9 Aug 2011 10:13:02 -0400
From: Jonathan Reams <[email protected]>
Subject: Re: Performance issues with krb5-1.9.1
To: [email protected]
Message-ID: <[email protected]>
Content-Type: text/plain; charset=us-ascii
Chris,
We didn't actually see any problems either until the KDC was under heavy load.
The unpatched version of 1.9.1 was and still is running on our secondary KDC
without issue, and we had been using 1.9.1 in testing and development for
months without issue as well. During the period where we saw the performance
degradation, the primary KDC handled 467000 distinct AS/TGS requests. Which
means the KDC was handling roughly 43 requests per second (not counting lots of
retransmits). That is typical of our primary production KDC's workload
throughout the day, but we don't have any other KDC that gets that amount of
traffic; by contrast, our secondary KDC gets a request once or twice a minute.
So it would seem the performance problem only really comes into play when the
KDC is under heavy load.
Jonathan
On Aug 9, 2011, at 4:23 AM, Chris Hecker wrote:
>
> Just another data point: I'm not seeing this on my locally built (but
> not with the attached patch) 1.9.1:
>
> real 0m41.409s
> user 0m3.358s
> sys 0m3.683s
> finished round 1
>
> real 0m35.036s
> user 0m3.441s
> sys 0m3.658s
> finished round 2
>
> real 0m44.344s
> user 0m3.363s
> sys 0m3.728s
> finished round 3
>
> real 0m40.930s
> user 0m3.465s
> sys 0m3.973s
> finished round 4
>
> I had to reduce the number of inner iterations to 300 because my machine
> is slow. The variance in the above numbers is because there's a bunch
> of stuff running on this machine.
>
> Chris
>
> On 2011/08/08 11:21, Greg Hudson wrote:
>> On Mon, 2011-08-08 at 11:22 -0400, Jonathan Reams wrote:
>>> I did some performance testing on our test KDC and was able to
>>> reproduce the performance issue with 1.9.1.
>>
>> I found a regression which would affect these tests, but I'm not sure it
>> accounts for your global performance issues.
>>
>> The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache,
>> but due to a bug, it is actually opening and reading a replay cache for
>> every TGS request, which is significantly less efficient than the 1.8
>> behavior (using a replay cache which stays open for the lifetime of the
>> KDC).
>>
>> In a test which runs in under five minutes, this regression produces
>> visible O(n^2) performance characteristics. This would not necessarily
>> account for performance degradation over hours, as the performance drag
>> of the replay cache should become stable after five minutes. It's
>> possible that the constant drag was enough to cause the KDC to fall
>> behind on the request load, but it's also possible that there's a second
>> problem which isn't so easily reproduced.
>>
>> I've attached a patch. Note that there is a second, in-memory
>> "lookaside" cache with O(n^2) performance characteristics in the short
>> term, which holds queries for up to two minutes. You may see a slight
>> degradation in performance in test cases due to this. You can
>> temporarily rebuild the kdc directory with "make clean;
>> CPPFLAGS=-DNOCACHE" if you want to remove this variable from your
>> performance tests.
>>
>>
>>
>>
>> ________________________________________________
>> Kerberos mailing list [email protected]
>> https://mailman.mit.edu/mailman/listinfo/kerberos
> ________________________________________________
> Kerberos mailing list [email protected]
> https://mailman.mit.edu/mailman/listinfo/kerberos
>
------------------------------
Message: 3
Date: Tue, 9 Aug 2011 12:21:37 +0000 (UTC)
From: Sanket Sangodkar <[email protected]>
Subject: Problem after adding new realm in krb5.conf
To: [email protected]
Message-ID: <[email protected]>
Content-Type: text/plain; charset=us-ascii
Hi,
We have configured SSO using Kerberos with Apache Http Server.
The realms name are defined in krb5.conf.
We needed to add one more domain to krb5.conf file -
We have inserted new realm as -
xxx.xxx.COM = {
kdc = <dns_name>
admin_server = <dns_name>
}
But after executing kinit command for respective domain xxx.xxx.COM it
specifies follwing error message -
kinit: Client not found in Kerberos database while getting initial credentials
We wanted to know whether only specifying new name under [realm] content will
add new domain/realm to Kerberos configuration.
Or are we missing any more configuration to add new realm ?
Regards,
Sanket
------------------------------
Message: 4
Date: Tue, 9 Aug 2011 17:57:25 +0530
From: "Sangodkar, Sanket" <[email protected]>
Subject: Error while adding new realm to krb5.conf
To: <[email protected]>
Message-ID:
<dc18c77d955e0447bccbfbd690781b99f87...@inicx002.in.int.atosorigin.com>
Content-Type: text/plain; charset="us-ascii"
Hi,
We have configured SSO using Kerberos with Apache Http Server.
The realms name are defined in krb5.conf.
We needed to add one more domain to krb5.conf file -
We have inserted new realm as -
xxx.xxx.COM = {
kdc = <dns_name>
admin_server = <dns_name>
}
But after executing kinit command for respective domain xxx.xxx.COM it
specifies following error message -
kinit: Client not found in Kerberos database while getting initial
credentials
We wanted to know whether only specifying new name under [realm] content
will
add new domain/realm to Kerberos configuration.
Or are we missing any more configuration to add new realm ?
Regards,
Sanket
This e-mail and the documents attached are confidential and intended solely for
the addressee; it may also be privileged. If you receive this e-mail in error,
please notify the sender immediately and destroy it. As its integrity cannot be
secured on the Internet, the Atos group liability cannot be triggered for the
message content. Although the sender endeavors to maintain a computer
virus-free network, the sender does not warrant that this transmission is
virus-free and will not be liable for any damages resulting from any virus
transmitted.
------------------------------
Message: 5
Date: Wed, 10 Aug 2011 00:20:32 +0530
From: Vipin Rathor <[email protected]>
Subject: Re: Error while adding new realm to krb5.conf
To: "Sangodkar, Sanket" <[email protected]>
Cc: [email protected]
Message-ID:
<CAN-7Vpkoi+yoeyKWTcVGOcbO6W93rXewrtvLzYPqTsvB=gy...@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
> kinit: Client not found in Kerberos database while getting initial
> credentials
Are you sure that you are using correct username (for kinit) which
exist in the new realm?
Please make sure that:
1. you are specifying correct username + realm name for getting
ticket. (e.g. kinit [email protected])
2. The new KDC & kadmin servers are reachable.
> Or are we missing any more configuration to add new realm ?
Can't think of any more configuration that is required.
On Tue, Aug 9, 2011 at 5:57 PM, Sangodkar, Sanket
<[email protected]> wrote:
>
>
> Hi,
>
>
>
> We have configured SSO using Kerberos with Apache Http Server.
>
> The realms name are defined in krb5.conf.
>
> We needed to add one more domain to krb5.conf file -
>
>
>
> We have inserted new realm as -
>
> xxx.xxx.COM = {
>
> ? ? ? ? ? ? ? ?kdc = <dns_name>
>
> ? ? ? ? ? ? ? ?admin_server = <dns_name>
>
> ? ? ? ?}
>
>
>
> But after executing kinit command for respective domain xxx.xxx.COM it
>
> specifies following error message -
>
> kinit: Client not found in Kerberos database while getting initial
> credentials
>
>
>
> We wanted to know whether only specifying new name under [realm] content
> will
>
> add new domain/realm to Kerberos configuration.
>
> Or are we missing any more configuration to add new realm ?
>
>
>
> Regards,
>
> Sanket
>
>
>
>
>
>
>
> This e-mail and the documents attached are confidential and intended solely
> for the addressee; it may also be privileged. If you receive this e-mail in
> error, please notify the sender immediately and destroy it. As its integrity
> cannot be secured on the Internet, the Atos group liability cannot be
> triggered for the message content. Although the sender endeavors to maintain
> a computer virus-free network, the sender does not warrant that this
> transmission is virus-free and will not be liable for any damages resulting
> from any virus transmitted.
> ________________________________________________
> Kerberos mailing list ? ? ? ? ? [email protected]
> https://mailman.mit.edu/mailman/listinfo/kerberos
>
--
-Rathor
------------------------------
Message: 6
Date: Tue, 09 Aug 2011 12:39:46 -0700
From: Chris Hecker <[email protected]>
Subject: Re: Performance issues with krb5-1.9.1
To: [email protected]
Message-ID: <[email protected]>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Ah, yeah, my tests had krb5kdc at about 50% of one core (slapd was an
additional 15%), but it wasn't completely saturating the machine.
Glad the patch fixed it!
Chris
On 2011/08/09 07:13, Jonathan Reams wrote:
> Chris,
>
> We didn't actually see any problems either until the KDC was under heavy
> load. The unpatched version of 1.9.1 was and still is running on our
> secondary KDC without issue, and we had been using 1.9.1 in testing and
> development for months without issue as well. During the period where we saw
> the performance degradation, the primary KDC handled 467000 distinct AS/TGS
> requests. Which means the KDC was handling roughly 43 requests per second
> (not counting lots of retransmits). That is typical of our primary production
> KDC's workload throughout the day, but we don't have any other KDC that gets
> that amount of traffic; by contrast, our secondary KDC gets a request once or
> twice a minute. So it would seem the performance problem only really comes
> into play when the KDC is under heavy load.
>
> Jonathan
>
> On Aug 9, 2011, at 4:23 AM, Chris Hecker wrote:
>
>>
>> Just another data point: I'm not seeing this on my locally built (but
>> not with the attached patch) 1.9.1:
>>
>> real 0m41.409s
>> user 0m3.358s
>> sys 0m3.683s
>> finished round 1
>>
>> real 0m35.036s
>> user 0m3.441s
>> sys 0m3.658s
>> finished round 2
>>
>> real 0m44.344s
>> user 0m3.363s
>> sys 0m3.728s
>> finished round 3
>>
>> real 0m40.930s
>> user 0m3.465s
>> sys 0m3.973s
>> finished round 4
>>
>> I had to reduce the number of inner iterations to 300 because my machine
>> is slow. The variance in the above numbers is because there's a bunch
>> of stuff running on this machine.
>>
>> Chris
>>
>> On 2011/08/08 11:21, Greg Hudson wrote:
>>> On Mon, 2011-08-08 at 11:22 -0400, Jonathan Reams wrote:
>>>> I did some performance testing on our test KDC and was able to
>>>> reproduce the performance issue with 1.9.1.
>>>
>>> I found a regression which would affect these tests, but I'm not sure it
>>> accounts for your global performance issues.
>>>
>>> The KDC in krb5 1.9 isn't supposed to be using an on-disk replay cache,
>>> but due to a bug, it is actually opening and reading a replay cache for
>>> every TGS request, which is significantly less efficient than the 1.8
>>> behavior (using a replay cache which stays open for the lifetime of the
>>> KDC).
>>>
>>> In a test which runs in under five minutes, this regression produces
>>> visible O(n^2) performance characteristics. This would not necessarily
>>> account for performance degradation over hours, as the performance drag
>>> of the replay cache should become stable after five minutes. It's
>>> possible that the constant drag was enough to cause the KDC to fall
>>> behind on the request load, but it's also possible that there's a second
>>> problem which isn't so easily reproduced.
>>>
>>> I've attached a patch. Note that there is a second, in-memory
>>> "lookaside" cache with O(n^2) performance characteristics in the short
>>> term, which holds queries for up to two minutes. You may see a slight
>>> degradation in performance in test cases due to this. You can
>>> temporarily rebuild the kdc directory with "make clean;
>>> CPPFLAGS=-DNOCACHE" if you want to remove this variable from your
>>> performance tests.
>>>
>>>
>>>
>>>
>>> ________________________________________________
>>> Kerberos mailing list [email protected]
>>> https://mailman.mit.edu/mailman/listinfo/kerberos
>> ________________________________________________
>> Kerberos mailing list [email protected]
>> https://mailman.mit.edu/mailman/listinfo/kerberos
>>
>
>
> ________________________________________________
> Kerberos mailing list [email protected]
> https://mailman.mit.edu/mailman/listinfo/kerberos
>
------------------------------
_______________________________________________
Kerberos mailing list
[email protected]
https://mailman.mit.edu/mailman/listinfo/kerberos
End of Kerberos Digest, Vol 104, Issue 11
*****************************************
________________________________________________
Kerberos mailing list [email protected]
https://mailman.mit.edu/mailman/listinfo/kerberos
