Re: [Freeipa-users] Server error with multiple clients joining domain simultaneously

[Petr Spacek]

Interesting, we have to investigate it!

Here is a ticket:
https://fedorahosted.org/freeipa/ticket/5653

You can Cc yourself to it and watch the progress.

Petr^2 Spacek

On 28.1.2016 20:17, David Zabner wrote:
> I was guessing that it was a problem with mod_auth_gssapi and so I tried 
> switching the auth method back to mod_auth_kerb which did not work. (although 
> it is entirely possible that I did not switch it correctly)
> 
> I did it by changing the gssapi settings in /etc/httpd/conf.d/ipa.conf to:
> 
>   AuthType Kerberos
>   AuthName "Kerberos Login"
>   KrbMethodNegotiate on
>   KrbMethodK5Passwd off
>   KrbServiceName HTTP
>   KrbAuthRealms $realm
>   Krb5KeyTab /etc/httpd/conf/ipa.keytab
>   KrbSaveCredentials on
>   KrbConstrainedDelegation on
>   Require valid-user
>   ErrorDocument 401 /ipa/errors/unauthorized.html
> 
> It just seemed to cause other problems...
> 
> On Jan 28, 2016, at 1:44 PM, Izzo, Anthony 
> <aizz...@harris.com<mailto:aizz...@harris.com>> wrote:
> 
> I should add that some of my team members have tried serializing their 
> instance launches, and this problem does not seem to occur under those 
> circumstances.  (That’s not a solution, just a data point for those 
> interested in this behavior).  Thanks.
> 
> 
> From: Izzo, Anthony (U.S. Person)
> Sent: Thursday, January 28, 2016 1:35 PM
> To: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com>
> Cc: 'David Zabner' <da...@cazena.com<mailto:da...@cazena.com>>
> Subject: RE: [Freeipa-users] Server error with multiple clients joining 
> domain simultaneously
> 
> Yes, that’s it!
> 
> From: David Zabner [mailto:da...@cazena.com]
> Sent: Thursday, January 28, 2016 1:31 PM
> To: Izzo, Anthony (U.S. Person) 
> <aizz...@harris.com<mailto:aizz...@harris.com>>
> Cc: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com>
> Subject: Re: [Freeipa-users] Server error with multiple clients joining 
> domain simultaneously
> 
> This sounds exactly like the problem I am having. I will attach my error log. 
> Is this what yours looks like?
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
> 
> 
> 


-- 
Petr^2 Spacek

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Server error with multiple clients joining domain simultaneously

[Rob Crittenden]

David Zabner wrote:
> Any guesses as to why I couldn’t revert to using the mod_auth_kerb library? 
> It seems like this is the only place where the library is referenced one way 
> or the other…
> 

You need to set this globally:

KrbConstrainedDelegationLock ipa

And I assume you replaced $realm with your actual realm, right?

It would also be useful to know how it doesn't work.

rob

> Thanks for all your help.
> 
>> On Jan 29, 2016, at 6:35 AM, Petr Spacek <pspa...@redhat.com> wrote:
>>
>> Interesting, we have to investigate it!
>>
>> Here is a ticket:
>> https://fedorahosted.org/freeipa/ticket/5653
>>
>> You can Cc yourself to it and watch the progress.
>>
>> Petr^2 Spacek
>>
>> On 28.1.2016 20:17, David Zabner wrote:
>>> I was guessing that it was a problem with mod_auth_gssapi and so I tried 
>>> switching the auth method back to mod_auth_kerb which did not work. 
>>> (although it is entirely possible that I did not switch it correctly)
>>>
>>> I did it by changing the gssapi settings in /etc/httpd/conf.d/ipa.conf to:
>>> 
>>>  AuthType Kerberos
>>>  AuthName "Kerberos Login"
>>>  KrbMethodNegotiate on
>>>  KrbMethodK5Passwd off
>>>  KrbServiceName HTTP
>>>  KrbAuthRealms $realm
>>>  Krb5KeyTab /etc/httpd/conf/ipa.keytab
>>>  KrbSaveCredentials on
>>>  KrbConstrainedDelegation on
>>>  Require valid-user
>>>  ErrorDocument 401 /ipa/errors/unauthorized.html
>>> 
>>> It just seemed to cause other problems...
>>>
>>> On Jan 28, 2016, at 1:44 PM, Izzo, Anthony 
>>> <aizz...@harris.com<mailto:aizz...@harris.com>> wrote:
>>>
>>> I should add that some of my team members have tried serializing their 
>>> instance launches, and this problem does not seem to occur under those 
>>> circumstances.  (That’s not a solution, just a data point for those 
>>> interested in this behavior).  Thanks.
>>>
>>>
>>> From: Izzo, Anthony (U.S. Person)
>>> Sent: Thursday, January 28, 2016 1:35 PM
>>> To: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com>
>>> Cc: 'David Zabner' <da...@cazena.com<mailto:da...@cazena.com>>
>>> Subject: RE: [Freeipa-users] Server error with multiple clients joining 
>>> domain simultaneously
>>>
>>> Yes, that’s it!
>>>
>>> From: David Zabner [mailto:da...@cazena.com]
>>> Sent: Thursday, January 28, 2016 1:31 PM
>>> To: Izzo, Anthony (U.S. Person) 
>>> <aizz...@harris.com<mailto:aizz...@harris.com>>
>>> Cc: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com>
>>> Subject: Re: [Freeipa-users] Server error with multiple clients joining 
>>> domain simultaneously
>>>
>>> This sounds exactly like the problem I am having. I will attach my error 
>>> log. Is this what yours looks like?
>>> --
>>> Manage your subscription for the Freeipa-users mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> Go to http://freeipa.org for more info on the project
>>>
>>>
>>>
>>
>>
>> -- 
>> Petr^2 Spacek
>>
>> -- 
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go to http://freeipa.org for more info on the project
> 
> 

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Server error with multiple clients joining domain simultaneously

[David Zabner]

Any guesses as to why I couldn’t revert to using the mod_auth_kerb library? It 
seems like this is the only place where the library is referenced one way or 
the other…

Thanks for all your help.

> On Jan 29, 2016, at 6:35 AM, Petr Spacek <pspa...@redhat.com> wrote:
> 
> Interesting, we have to investigate it!
> 
> Here is a ticket:
> https://fedorahosted.org/freeipa/ticket/5653
> 
> You can Cc yourself to it and watch the progress.
> 
> Petr^2 Spacek
> 
> On 28.1.2016 20:17, David Zabner wrote:
>> I was guessing that it was a problem with mod_auth_gssapi and so I tried 
>> switching the auth method back to mod_auth_kerb which did not work. 
>> (although it is entirely possible that I did not switch it correctly)
>> 
>> I did it by changing the gssapi settings in /etc/httpd/conf.d/ipa.conf to:
>> 
>>  AuthType Kerberos
>>  AuthName "Kerberos Login"
>>  KrbMethodNegotiate on
>>  KrbMethodK5Passwd off
>>  KrbServiceName HTTP
>>  KrbAuthRealms $realm
>>  Krb5KeyTab /etc/httpd/conf/ipa.keytab
>>  KrbSaveCredentials on
>>  KrbConstrainedDelegation on
>>  Require valid-user
>>  ErrorDocument 401 /ipa/errors/unauthorized.html
>> 
>> It just seemed to cause other problems...
>> 
>> On Jan 28, 2016, at 1:44 PM, Izzo, Anthony 
>> <aizz...@harris.com<mailto:aizz...@harris.com>> wrote:
>> 
>> I should add that some of my team members have tried serializing their 
>> instance launches, and this problem does not seem to occur under those 
>> circumstances.  (That’s not a solution, just a data point for those 
>> interested in this behavior).  Thanks.
>> 
>> 
>> From: Izzo, Anthony (U.S. Person)
>> Sent: Thursday, January 28, 2016 1:35 PM
>> To: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com>
>> Cc: 'David Zabner' <da...@cazena.com<mailto:da...@cazena.com>>
>> Subject: RE: [Freeipa-users] Server error with multiple clients joining 
>> domain simultaneously
>> 
>> Yes, that’s it!
>> 
>> From: David Zabner [mailto:da...@cazena.com]
>> Sent: Thursday, January 28, 2016 1:31 PM
>> To: Izzo, Anthony (U.S. Person) 
>> <aizz...@harris.com<mailto:aizz...@harris.com>>
>> Cc: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com>
>> Subject: Re: [Freeipa-users] Server error with multiple clients joining 
>> domain simultaneously
>> 
>> This sounds exactly like the problem I am having. I will attach my error 
>> log. Is this what yours looks like?
>> --
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go to http://freeipa.org for more info on the project
>> 
>> 
>> 
> 
> 
> -- 
> Petr^2 Spacek
> 
> -- 
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project


-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Server error with multiple clients joining domain simultaneously

[David Zabner]

d.org/freeipa/ticket/5653
>>> 
>>> You can Cc yourself to it and watch the progress.
>>> 
>>> Petr^2 Spacek
>>> 
>>> On 28.1.2016 20:17, David Zabner wrote:
>>>> I was guessing that it was a problem with mod_auth_gssapi and so I tried 
>>>> switching the auth method back to mod_auth_kerb which did not work. 
>>>> (although it is entirely possible that I did not switch it correctly)
>>>> 
>>>> I did it by changing the gssapi settings in /etc/httpd/conf.d/ipa.conf to:
>>>> 
>>>> AuthType Kerberos
>>>> AuthName "Kerberos Login"
>>>> KrbMethodNegotiate on
>>>> KrbMethodK5Passwd off
>>>> KrbServiceName HTTP
>>>> KrbAuthRealms $realm
>>>> Krb5KeyTab /etc/httpd/conf/ipa.keytab
>>>> KrbSaveCredentials on
>>>> KrbConstrainedDelegation on
>>>> Require valid-user
>>>> ErrorDocument 401 /ipa/errors/unauthorized.html
>>>> 
>>>> It just seemed to cause other problems...
>>>> 
>>>> On Jan 28, 2016, at 1:44 PM, Izzo, Anthony 
>>>> <aizz...@harris.com<mailto:aizz...@harris.com>> wrote:
>>>> 
>>>> I should add that some of my team members have tried serializing their 
>>>> instance launches, and this problem does not seem to occur under those 
>>>> circumstances.  (That’s not a solution, just a data point for those 
>>>> interested in this behavior).  Thanks.
>>>> 
>>>> 
>>>> From: Izzo, Anthony (U.S. Person)
>>>> Sent: Thursday, January 28, 2016 1:35 PM
>>>> To: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com>
>>>> Cc: 'David Zabner' <da...@cazena.com<mailto:da...@cazena.com>>
>>>> Subject: RE: [Freeipa-users] Server error with multiple clients joining 
>>>> domain simultaneously
>>>> 
>>>> Yes, that’s it!
>>>> 
>>>> From: David Zabner [mailto:da...@cazena.com]
>>>> Sent: Thursday, January 28, 2016 1:31 PM
>>>> To: Izzo, Anthony (U.S. Person) 
>>>> <aizz...@harris.com<mailto:aizz...@harris.com>>
>>>> Cc: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com>
>>>> Subject: Re: [Freeipa-users] Server error with multiple clients joining 
>>>> domain simultaneously
>>>> 
>>>> This sounds exactly like the problem I am having. I will attach my error 
>>>> log. Is this what yours looks like?
>>>> --
>>>> Manage your subscription for the Freeipa-users mailing list:
>>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>>> Go to http://freeipa.org for more info on the project
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> -- 
>>> Petr^2 Spacek
>>> 
>>> -- 
>>> Manage your subscription for the Freeipa-users mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> Go to http://freeipa.org for more info on the project
>> 
>> 
> 

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Server error with multiple clients joining domain simultaneously

[Rob Crittenden]

David Zabner wrote:
> Ok so I added the line "KrbConstrainedDelegationLock ipa” to ipa.conf (httpd 
> configuration)
> 
> 
> My error log is now full of network errors:
> 

config looks right to me. Does this mean that some requests are
successful and others are not?

I'd set LogLevel debug in nss.conf and restart and you should get more
verbose info out of mod_auth_kerb.

rob

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Server error with multiple clients joining domain simultaneously

[Izzo, Anthony]

I should add that some of my team members have tried serializing their instance 
launches, and this problem does not seem to occur under those circumstances.  
(That's not a solution, just a data point for those interested in this 
behavior).  Thanks.


From: Izzo, Anthony (U.S. Person)
Sent: Thursday, January 28, 2016 1:35 PM
To: freeipa-users@redhat.com
Cc: 'David Zabner' <da...@cazena.com>
Subject: RE: [Freeipa-users] Server error with multiple clients joining domain 
simultaneously

Yes, that's it!

From: David Zabner [mailto:da...@cazena.com]
Sent: Thursday, January 28, 2016 1:31 PM
To: Izzo, Anthony (U.S. Person) <aizz...@harris.com<mailto:aizz...@harris.com>>
Cc: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com>
Subject: Re: [Freeipa-users] Server error with multiple clients joining domain 
simultaneously

This sounds exactly like the problem I am having. I will attach my error log. 
Is this what yours looks like?
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Server error with multiple clients joining domain simultaneously

[David Zabner]

This sounds exactly like the problem I am having. I will attach my error log. 
Is this what yours looks like?


error_log
Description: error_log
On Jan 28, 2016, at 1:10 PM, Izzo, Anthony  wrote:I’m seeing what feels like a concurrency error.  I’m in a cloud environment and launching a group of instances which are all trying to join a domain at about the same time via ipa-client-install.  Some of these operations succeed, and others fail. The error message on those that fail is that they failed to join the domain, and the HTTP response was 500 instead of 200. The Apache error_log file on the server, shows a python stack trace (which unfortunately I can’t reproduce in its entirety here), which culminates in the complaint that a file (/var/run/httpd/ipa/clientcaches/@) was not found.  What it seems like is that multiple attempts to join the domain from different hosts are stepping on one another. I’m wondering if I am trying to do something that is not supported, or if I have something misconfigured.  I’m tempted to catch the error and retry after a random interval (the output of the failing command indicates that it is rolling back to the initial state) – that would be the easiest thing.  But if this is pointing to an underlying error on my part I’d rather fix it if possible. Additional info in case it helps – I’m running RHEL7/FreeIPA4.2 on the servers (two in a replication agreement).  I’m running RHEL6/FreeIPA3.0 on the clients (most recent attempt I tried to launch 7 instances, three of which failed).  Thanks. Tony  -- Manage your subscription for the Freeipa-users mailing list:https://www.redhat.com/mailman/listinfo/freeipa-usersGo to http://freeipa.org for more info on the project-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Freeipa-users] Server error with multiple clients joining domain simultaneously

[Izzo, Anthony]

I'm seeing what feels like a concurrency error.  I'm in a cloud environment and 
launching a group of instances which are all trying to join a domain at about 
the same time via ipa-client-install.  Some of these operations succeed, and 
others fail.

The error message on those that fail is that they failed to join the domain, 
and the HTTP response was 500 instead of 200.

The Apache error_log file on the server, shows a python stack trace (which 
unfortunately I can't reproduce in its entirety here), which culminates in the 
complaint that a file (/var/run/httpd/ipa/clientcaches/@) 
was not found.  What it seems like is that multiple attempts to join the domain 
from different hosts are stepping on one another.

I'm wondering if I am trying to do something that is not supported, or if I 
have something misconfigured.  I'm tempted to catch the error and retry after a 
random interval (the output of the failing command indicates that it is rolling 
back to the initial state) - that would be the easiest thing.  But if this is 
pointing to an underlying error on my part I'd rather fix it if possible.

Additional info in case it helps - I'm running RHEL7/FreeIPA4.2 on the servers 
(two in a replication agreement).  I'm running RHEL6/FreeIPA3.0 on the clients 
(most recent attempt I tried to launch 7 instances, three of which failed).  
Thanks.

Tony


-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project