Re: [ovirt-users] slow kerberos authentication

[Fabrice Bacchella]

It works much better now. Goes from 6s to less than 500ms. Not blazing fast but 
much more usable, thanks a lot.

> Le 12 mai 2017 à 15:58, Ondra Machacek  a écrit :
> 
> This is new feature in aaa-ldap tracked here[1].
> By default for AD profiles we use this feature, and it should
> increase performance in most cases.
> 
> But if this is not the case for you, can you just try to change the profile
> from:
> 
>  include = 
> 
> to
> 
>  include = 
> 
> And see if it will be better?
> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1393407 
> 
> 
> On Fri, May 12, 2017 at 2:54 PM, Fabrice Bacchella 
> > wrote:
> I found that:
> 
> http://dunnry.com/blog/TransitiveLinkValueFilterEvaluation.aspx 
> 
> 
> 
>> Le 12 mai 2017 à 14:44, Fabrice Bacchella > > a écrit :
>> 
>> Ok, I found where it's slow, it's a ldapsearch on our AD:
>> 
>> time ldapsearch -a never -E pr=100/noprompt -H ldap://ad1 <> -b DC=... -s 
>> sub '(&(groupType:1.2.840.113556.1.4.803:=2147483648 
>> )(&(objectCategory=group)(member:1.2.840.113556.1.4.1941:=userdn)))'
>>  objectGUID name description
>> 
>> # numResponses: 70
>> # numEntries: 66
>> # numReferences: 3
>> 
>> real 0m10.801s
>> user 0m0.007s
>> sys  0m0.012s
>> 
>> That matches the log line:
>> 2017-05-12 14:22:17,413+02 DEBUG 
>> [org.ovirt.engineextensions.aaa.ldap.Framework] (pool-25-thread-2) [] 
>> Performing SearchRequest 'SearchRequest(baseDN='...', scope=SUB, 
>> deref=NEVER, sizeLimit=0, timeLimit=0, 
>> filter='&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=2147483648)(member:1.2.840.113556.1.4.1941:=...)',
>>  attrs={objectGUID, name, description}, 
>> controls={SimplePagedResultsControl(pageSize=100, isCritical=false)})' 
>> request on server '...'
>> 2017-05-12 14:22:24,456+02 DEBUG 
>> [org.ovirt.engineextensions.aaa.ldap.Framework] (pool-25-thread-1) [] 
>> SearchResult: SearchResult(resultCode=0 (success), messageID=3, 
>> entriesReturned=66, referencesReturned=0, 
>> responseControls={SimplePagedResultsControl(pageSize=0, isCritical=false)})
>> 
>> 
>> And without 1.2.840.113556.1.4.1941
>> 
>> # numResponses: 54
>> # numEntries: 50
>> # numReferences: 3
>> 
>> real 0m0.051s
>> user 0m0.008s
>> sys  0m0.007s
>> 
>> So it's an AD problem. 1.2.840.113556.1.4.1941 make it slow, but without it, 
>> the result is not the same. But I don't know if it's an AD or ovirt problem. 
>> I'll keep investigating.
>> 
>> Thank's for your help.
>> ___
>> Users mailing list
>> Users@ovirt.org 
>> http://lists.ovirt.org/mailman/listinfo/users 
>> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] slow kerberos authentication

[Ondra Machacek]

This is new feature in aaa-ldap tracked here[1].
By default for AD profiles we use this feature, and it should
increase performance in most cases.

But if this is not the case for you, can you just try to change the profile
from:

 include = 

to

 include = 

And see if it will be better?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1393407

On Fri, May 12, 2017 at 2:54 PM, Fabrice Bacchella <
fabrice.bacche...@orange.fr> wrote:

> I found that:
>
> http://dunnry.com/blog/TransitiveLinkValueFilterEvaluation.aspx
>
>
> Le 12 mai 2017 à 14:44, Fabrice Bacchella  a
> écrit :
>
> Ok, I found where it's slow, it's a ldapsearch on our AD:
>
> time ldapsearch -a never -E pr=100/noprompt -H ldap://ad1 -b DC=... -s
> sub '(&(groupType:1.2.840.113556.1.4.803:=2147483648 <(214)%20748-3648>
> )(&(objectCategory=group)(member:1.2.840.113556.1.4.1941:=userdn)))'
> objectGUID name description
>
> # numResponses: 70
> # numEntries: 66
> # numReferences: 3
>
> real 0m10.801s
> user 0m0.007s
> sys 0m0.012s
>
> That matches the log line:
> 2017-05-12 14:22:17,413+02 DEBUG 
> [org.ovirt.engineextensions.aaa.ldap.Framework]
> (pool-25-thread-2) [] Performing SearchRequest 'SearchRequest(baseDN='...',
> scope=SUB, deref=NEVER, sizeLimit=0, timeLimit=0, filter='&(objectCategory=
> group)(groupType:1.2.840.113556.1.4.803:=2147483648)(
> member:1.2.840.113556.1.4.1941:=...)', attrs={objectGUID, name,
> description}, controls={SimplePagedResultsControl(pageSize=100,
> isCritical=false)})' request on server '...'
> 2017-05-12 14:22:24,456+02 DEBUG 
> [org.ovirt.engineextensions.aaa.ldap.Framework]
> (pool-25-thread-1) [] SearchResult: SearchResult(resultCode=0 (success),
> messageID=3, entriesReturned=66, referencesReturned=0, responseControls={
> SimplePagedResultsControl(pageSize=0, isCritical=false)})
>
>
> And without 1.2.840.113556.1.4.1941
>
> # numResponses: 54
> # numEntries: 50
> # numReferences: 3
>
> real 0m0.051s
> user 0m0.008s
> sys 0m0.007s
>
> So it's an AD problem. 1.2.840.113556.1.4.1941 make it slow, but without
> it, the result is not the same. But I don't know if it's an AD or ovirt
> problem. I'll keep investigating.
>
> Thank's for your help.
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] slow kerberos authentication

[Fabrice Bacchella]

I found that:

http://dunnry.com/blog/TransitiveLinkValueFilterEvaluation.aspx


> Le 12 mai 2017 à 14:44, Fabrice Bacchella  a 
> écrit :
> 
> Ok, I found where it's slow, it's a ldapsearch on our AD:
> 
> time ldapsearch -a never -E pr=100/noprompt -H ldap://ad1  -b 
> DC=... -s sub 
> '(&(groupType:1.2.840.113556.1.4.803:=2147483648)(&(objectCategory=group)(member:1.2.840.113556.1.4.1941:=userdn)))'
>  objectGUID name description
> 
> # numResponses: 70
> # numEntries: 66
> # numReferences: 3
> 
> real  0m10.801s
> user  0m0.007s
> sys   0m0.012s
> 
> That matches the log line:
> 2017-05-12 14:22:17,413+02 DEBUG 
> [org.ovirt.engineextensions.aaa.ldap.Framework] (pool-25-thread-2) [] 
> Performing SearchRequest 'SearchRequest(baseDN='...', scope=SUB, deref=NEVER, 
> sizeLimit=0, timeLimit=0, 
> filter='&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=2147483648)(member:1.2.840.113556.1.4.1941:=...)',
>  attrs={objectGUID, name, description}, 
> controls={SimplePagedResultsControl(pageSize=100, isCritical=false)})' 
> request on server '...'
> 2017-05-12 14:22:24,456+02 DEBUG 
> [org.ovirt.engineextensions.aaa.ldap.Framework] (pool-25-thread-1) [] 
> SearchResult: SearchResult(resultCode=0 (success), messageID=3, 
> entriesReturned=66, referencesReturned=0, 
> responseControls={SimplePagedResultsControl(pageSize=0, isCritical=false)})
> 
> 
> And without 1.2.840.113556.1.4.1941
> 
> # numResponses: 54
> # numEntries: 50
> # numReferences: 3
> 
> real  0m0.051s
> user  0m0.008s
> sys   0m0.007s
> 
> So it's an AD problem. 1.2.840.113556.1.4.1941 make it slow, but without it, 
> the result is not the same. But I don't know if it's an AD or ovirt problem. 
> I'll keep investigating.
> 
> Thank's for your help.
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] slow kerberos authentication

[Fabrice Bacchella]

Ok, I found where it's slow, it's a ldapsearch on our AD:

time ldapsearch -a never -E pr=100/noprompt -H ldap://ad1 -b DC=... -s sub 
'(&(groupType:1.2.840.113556.1.4.803:=2147483648)(&(objectCategory=group)(member:1.2.840.113556.1.4.1941:=userdn)))'
 objectGUID name description

# numResponses: 70
# numEntries: 66
# numReferences: 3

real0m10.801s
user0m0.007s
sys 0m0.012s

That matches the log line:
2017-05-12 14:22:17,413+02 DEBUG 
[org.ovirt.engineextensions.aaa.ldap.Framework] (pool-25-thread-2) [] 
Performing SearchRequest 'SearchRequest(baseDN='...', scope=SUB, deref=NEVER, 
sizeLimit=0, timeLimit=0, 
filter='&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=2147483648)(member:1.2.840.113556.1.4.1941:=...)',
 attrs={objectGUID, name, description}, 
controls={SimplePagedResultsControl(pageSize=100, isCritical=false)})' request 
on server '...'
2017-05-12 14:22:24,456+02 DEBUG 
[org.ovirt.engineextensions.aaa.ldap.Framework] (pool-25-thread-1) [] 
SearchResult: SearchResult(resultCode=0 (success), messageID=3, 
entriesReturned=66, referencesReturned=0, 
responseControls={SimplePagedResultsControl(pageSize=0, isCritical=false)})


And without 1.2.840.113556.1.4.1941

# numResponses: 54
# numEntries: 50
# numReferences: 3

real0m0.051s
user0m0.008s
sys 0m0.007s

So it's an AD problem. 1.2.840.113556.1.4.1941 make it slow, but without it, 
the result is not the same. But I don't know if it's an AD or ovirt problem. 
I'll keep investigating.

Thank's for your help.___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] slow kerberos authentication

[Fabrice Bacchella]

> Le 12 mai 2017 à 13:35, Ondra Machacek  a écrit :
> 
> 
> 
> On Fri, May 12, 2017 at 1:18 PM, Fabrice Bacchella 
> > wrote:
> The request is indeed quite slow within ovirt, using the setup given by Juan:
> 
> /ovirt-engine/sso/oauth/token-http-auth 7001ms
> 
> I was not able to authenticate jboss-cli.sh, I don't know why: 
> 'admin@internal-authz': No valid profile found in credentials.
> 
> It should be admin@internal.
>  

Indeed, but an export don't show that:


  
  
39323336363566612D37622D346532612D396530632D316630396536643634636432
  
  
  admin
  *
  admin
  admin@internal-authz
  
internal-authz
  
  
  
  
  



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] slow kerberos authentication

[Ondra Machacek]

On Fri, May 12, 2017 at 1:18 PM, Fabrice Bacchella <
fabrice.bacche...@orange.fr> wrote:

> The request is indeed quite slow within ovirt, using the setup given by
> Juan:
>
> /ovirt-engine/sso/oauth/token-http-auth 7001ms
>
> I was not able to authenticate jboss-cli.sh, I don't know why:
> 'admin@internal-authz': No valid profile found in credentials.
>

It should be admin@internal.


>
> So I tried to modifie usr/share/ovirt-engine/services/ovirt-engine/ovirt-
> engine-logging.properties.in, adding:
> org.ovirt.engineextensions.aaa=ALL
> org.ovirt.engine.core.bll.aaa=ALL
> and then restart ovirt-engine. But that changed nothing. That's not the
> good syntax ?
>

You must change the file in ovirt-engine.xml.in same file as Juan send
above.
See here:
https://github.com/oVirt/ovirt-engine-extension-aaa-ldap/blob/ovirt-engine-extension-aaa-ldap-1.0/README#L377

But I think better to use JBoss CLI, you don't have to restart oVirt engine
then.


>
>
>
>
> > Le 12 mai 2017 à 09:25, Ondra Machacek  a écrit :
> >
> > I am not aware of anything, but debug log of all aaa stuff would help,
> > to understand what takes the most time.
> >
> >  - org.ovirt.engineextensions.aaa.ldap
> >  - org.ovirt.engineextensions.aaa.misc
> >  - org.ovirt.engine.core.aaa
> >  - org.ovirt.engine.core.sso
> >
> > To enable it in runtime, please follow:
> >
> >  https://github.com/oVirt/ovirt-engine-extension-aaa-
> ldap/blob/master/README#L469
> >
> > On Thu, May 11, 2017 at 7:24 PM, Fabrice Bacchella <
> fabrice.bacche...@orange.fr> wrote:
> > I'm using kerberos authentication in ovirt for the URL
> /sso/oauth/token-http-auth, but kerberos is done in Apache using
> auth_gssapi_module and it's quite slow, about 6s for a request.
> >
> > I'm trying to understand if it's apache or ovirt-engine that are slow.
> Is there a way to get response time metered for http requests inside ovirt
> instead of seen from apache ?
> >
> > ___
> > Users mailing list
> > Users@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] slow kerberos authentication

[Fabrice Bacchella]

The request is indeed quite slow within ovirt, using the setup given by Juan:

/ovirt-engine/sso/oauth/token-http-auth 7001ms

I was not able to authenticate jboss-cli.sh, I don't know why: 
'admin@internal-authz': No valid profile found in credentials.

So I tried to modifie 
usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine-logging.properties.in,
 adding:
org.ovirt.engineextensions.aaa=ALL
org.ovirt.engine.core.bll.aaa=ALL
and then restart ovirt-engine. But that changed nothing. That's not the good 
syntax ?




> Le 12 mai 2017 à 09:25, Ondra Machacek  a écrit :
> 
> I am not aware of anything, but debug log of all aaa stuff would help,
> to understand what takes the most time.
> 
>  - org.ovirt.engineextensions.aaa.ldap
>  - org.ovirt.engineextensions.aaa.misc
>  - org.ovirt.engine.core.aaa
>  - org.ovirt.engine.core.sso
> 
> To enable it in runtime, please follow:
> 
>  
> https://github.com/oVirt/ovirt-engine-extension-aaa-ldap/blob/master/README#L469
> 
> On Thu, May 11, 2017 at 7:24 PM, Fabrice Bacchella 
>  wrote:
> I'm using kerberos authentication in ovirt for the URL 
> /sso/oauth/token-http-auth, but kerberos is done in Apache using 
> auth_gssapi_module and it's quite slow, about 6s for a request.
> 
> I'm trying to understand if it's apache or ovirt-engine that are slow. Is 
> there a way to get response time metered for http requests inside ovirt 
> instead of seen from apache ?
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> 

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] slow kerberos authentication

[Juan Hernández]

On 05/12/2017 11:45 AM, Juan Hernández wrote:
> On 05/12/2017 10:04 AM, Yaniv Kaul wrote:
>>
>>
>> On May 11, 2017 8:25 PM, "Fabrice Bacchella"
>> > wrote:
>>
>> I'm using kerberos authentication in ovirt for the URL
>> /sso/oauth/token-http-auth, but kerberos is done in Apache using
>> auth_gssapi_module and it's quite slow, about 6s for a request.
>>
>> I'm trying to understand if it's apache or ovirt-engine that are
>> slow. Is there a way to get response time metered for http requests
>> inside ovirt instead of seen from apache ?
>>
>>
>> In 4.1, look under /var/log/httpd, there should be an ovirt specific log
>> file for exactly this - end to end latency of requests. 
>> Y. 
>>
> 
> The name of that file is 'ovirt-requests-log', and it contains messages
> like this:
> 
>   [12/May/2017:11:09:30 +0200] 192.168.122.1 "Correlation-Id:
> 9e259b75-ee9e-4501-9737-b38d2c318123" "Duration: 393514us" "GET
> /ovirt-engine/api/vms HTTP/1.1" 2322
> 
> Note however that it is generated by the web server, so the reported
> time will include all the web server activities required to complete the
> request.
> 
> If you need to get the same measurement from the point of view of the
> application server you can edit the
> /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.xml.in file
> and do the following modifications:
> 
> 1. Modify the AJP connector (the one that Apache talks to) so that it
> records the start time of the request:
> 
>name="ajp"
> socket-binding="ajp"
> redirect-socket="redirect"
> record-request-start-time="true"  <-- This is new
>   />
> 
> 2. Enable the access log:
> 
>   
> 
>  prefix="my" suffix=".log"/>  <-- This is new
>   
> 
> Then restart the engine. It will start to write to
> /var/log/ovirt-engine/my.log lines like this:
> 
>   /ovirt-engine/api/vms 801ms
> 
> The format of the pattern is described here:
> 
>   http://undertow.io/javadoc/1.4.x/index.html
> 

Actually here:


http://undertow.io/javadoc/1.4.x/io/undertow/server/handlers/accesslog/AccessLogHandler.html

> Remember that the ovirt-engine.xml.in file isn't considered a
> configuration file, so your changes will be lost next time you update
> the engine RPMs.
> 

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] slow kerberos authentication

[Juan Hernández]

On 05/12/2017 10:04 AM, Yaniv Kaul wrote:
> 
> 
> On May 11, 2017 8:25 PM, "Fabrice Bacchella"
> > wrote:
> 
> I'm using kerberos authentication in ovirt for the URL
> /sso/oauth/token-http-auth, but kerberos is done in Apache using
> auth_gssapi_module and it's quite slow, about 6s for a request.
> 
> I'm trying to understand if it's apache or ovirt-engine that are
> slow. Is there a way to get response time metered for http requests
> inside ovirt instead of seen from apache ?
> 
> 
> In 4.1, look under /var/log/httpd, there should be an ovirt specific log
> file for exactly this - end to end latency of requests. 
> Y. 
> 

The name of that file is 'ovirt-requests-log', and it contains messages
like this:

  [12/May/2017:11:09:30 +0200] 192.168.122.1 "Correlation-Id:
9e259b75-ee9e-4501-9737-b38d2c318123" "Duration: 393514us" "GET
/ovirt-engine/api/vms HTTP/1.1" 2322

Note however that it is generated by the web server, so the reported
time will include all the web server activities required to complete the
request.

If you need to get the same measurement from the point of view of the
application server you can edit the
/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.xml.in file
and do the following modifications:

1. Modify the AJP connector (the one that Apache talks to) so that it
records the start time of the request:

  

2. Enable the access log:

  

  <-- This is new
  

Then restart the engine. It will start to write to
/var/log/ovirt-engine/my.log lines like this:

  /ovirt-engine/api/vms 801ms

The format of the pattern is described here:

  http://undertow.io/javadoc/1.4.x/index.html

Remember that the ovirt-engine.xml.in file isn't considered a
configuration file, so your changes will be lost next time you update
the engine RPMs.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] slow kerberos authentication

[Yaniv Kaul]

On May 11, 2017 8:25 PM, "Fabrice Bacchella" 
wrote:

I'm using kerberos authentication in ovirt for the URL
/sso/oauth/token-http-auth, but kerberos is done in Apache using
auth_gssapi_module and it's quite slow, about 6s for a request.

I'm trying to understand if it's apache or ovirt-engine that are slow. Is
there a way to get response time metered for http requests inside ovirt
instead of seen from apache ?


In 4.1, look under /var/log/httpd, there should be an ovirt specific log
file for exactly this - end to end latency of requests.
Y.


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] slow kerberos authentication

[Ondra Machacek]

I am not aware of anything, but debug log of all aaa stuff would help,
to understand what takes the most time.

 - org.ovirt.engineextensions.aaa.ldap
 - org.ovirt.engineextensions.aaa.misc
 - org.ovirt.engine.core.aaa
 - org.ovirt.engine.core.sso

To enable it in runtime, please follow:


https://github.com/oVirt/ovirt-engine-extension-aaa-ldap/blob/master/README#L469

On Thu, May 11, 2017 at 7:24 PM, Fabrice Bacchella <
fabrice.bacche...@orange.fr> wrote:

> I'm using kerberos authentication in ovirt for the URL
> /sso/oauth/token-http-auth, but kerberos is done in Apache using
> auth_gssapi_module and it's quite slow, about 6s for a request.
>
> I'm trying to understand if it's apache or ovirt-engine that are slow. Is
> there a way to get response time metered for http requests inside ovirt
> instead of seen from apache ?
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] slow kerberos authentication

[Fabrice Bacchella]

I'm using kerberos authentication in ovirt for the URL 
/sso/oauth/token-http-auth, but kerberos is done in Apache using 
auth_gssapi_module and it's quite slow, about 6s for a request.

I'm trying to understand if it's apache or ovirt-engine that are slow. Is there 
a way to get response time metered for http requests inside ovirt instead of 
seen from apache ?

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users