Re: [ovirt-users] [ovirt-devel] ovirt-engine-sdk-python too slow

[John Hunter]

So is this a bug or something?

Can I do something to fix the bug?

btw, adding Michael Pasternak

On Thu, Nov 26, 2015 at 7:52 PM, Nir Soffer  wrote:

> Thanks John, very interesting results.
>
> On Thu, Nov 26, 2015 at 3:17 AM, John Hunter  wrote:
>
>> Hi Juan,
>>
>> On Thu, Nov 26, 2015 at 2:15 AM, Juan Hernández 
>> wrote:
>>
>>> On 11/25/2015 06:45 PM, Nir Soffer wrote:
>>> > $ ./profile-stats -c myscript.prof
>>> >
>>> > Wed Nov 25 10:40:11 2015myscript.prof
>>> >
>>> >  7892315 function calls (7891054 primitive calls) in 7.940
>>> seconds
>>> >
>>> >Ordered by: internal time
>>> >List reduced from 1518 to 20 due to restriction <20>
>>> >
>>> >ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>>> >  90862.6930.0006.7060.001
>>> inspect.py:247(getmembers)
>>> >   19524941.3940.0001.8800.000 inspect.py:59(isclass)
>>> >  90921.0300.0001.0300.000 {dir}
>>> >   19526420.6000.0000.6000.000 {getattr}
>>> >   19727650.5040.0000.5040.000 {isinstance}
>>> > 30.3340.1110.3340.111 {method 'perform' of
>>> > 'pycurl.Curl' objects}
>>> >   18839180.2840.0000.2840.000 {method 'append' of
>>> 'list'
>>> > objects}
>>> >  90870.2210.0000.2210.000 {method 'sort' of 'list'
>>> > objects}
>>> >  90510.1720.0006.9110.001
>>> > reflectionhelper.py:51(isModuleMember)
>>> > 10.1240.1240.3540.354 errors.py:17()
>>> > 10.0880.0880.2300.230 params.py:8()
>>> >  88790.0700.0006.9980.001
>>> params.py:367(__setattr__)
>>> > 10.0470.0475.1825.182 api.py:23()
>>> > 10.0250.0254.7434.743 brokers.py:22()
>>> > 10.0230.0230.0300.030
>>> > connectionspool.py:17()
>>> > 10.0220.0220.0530.053
>>> > lxml.etree.pyx:1(PyMODINIT_FUNC PyInit_etree(void))
>>> >   1180.0190.0004.6840.040 params.py:45277(__init__)
>>> > 50.0150.0030.0240.005 {built-in method
>>> strptime}
>>> > 10.0120.0120.0130.013 socket.py:45()
>>> >100.0110.0010.0150.002
>>> collections.py:288(namedtuple)
>>> >
>>> > So it is not the classes, it is the code inspecting them on import.
>>> >
>>>
>>> The script doesn't contain only the imports, it is also calling the
>>> server, and we know parsing the result is slow, due to the excesive use
>>> of "inspect", as I mentioned before:
>>>
>>>   [RFE][performance] - generate large scale list running to slow.
>>>   https://bugzilla.redhat.com/show_bug.cgi?id=1221238#c2
>>>
>>> In the profiling information seems to corresponds to the script before
>>> commenting out the part that lists all the VMs, as it looks like the
>>> constructor of the VM class was called 21 times (you probably have 21
>>> VMs):
>>>
>>>   21 0.004 1.308
>>> build/bdist.linux-x86_64/egg/ovirtsdk/infrastructure/brokers.py:29139(VM)
>>>
>>> Actually I only have one VM running on the server.
>> This time it contains only the import in the script, not calling the
>> server. It shows:
>> 210.0051.666  brokers.py:29139(VM)
>>
>> $./profile-stats -c myscript_contains_only_import.prof
>>
>> Thu Nov 26 09:11:59 2015myscript_contains_only_import.prof
>>
>>  5453977 function calls (5452849 primitive calls) in 5.463 seconds
>>
>>Ordered by: internal time
>>List reduced from 1083 to 20 due to restriction <20>
>>
>>ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>>  74682.0290.0004.9140.001 inspect.py:247(getmembers)
>>   13486780.9750.0001.3420.000 inspect.py:59(isclass)
>>  74740.7370.0000.7370.000 {dir}
>>   13488250.4330.0000.4330.000 {getattr}
>>   13659700.3830.0000.3830.000 {isinstance}
>>   12934550.2110.0000.2110.000 {method 'append' of 'list'
>> objects}
>>  74690.1630.0000.1630.000 {method 'sort' of 'list'
>> objects}
>>  74340.1390.0005.0820.001
>> reflectionhelper.py:51(isModuleMember)
>>  76700.0610.0005.1580.001 params.py:367(__setattr__)
>> 10.0430.0435.4635.463 api.py:23()
>> 10.0420.0420.1390.139 errors.py:17()
>> 10.0270.0275.2485.248 brokers.py:22()
>> 10.0260.0260.0970.097 params.py:8()
>>   1180.0230.0005.1870.044 params.py:45277(__init__)
>> 10.0180.0180.0250.025
>> connectionspool.py:17()
>> 10.0130.0130.0130.013 socket.py:45()
>>100.0100.0010.0130.001
>> collections.py:288(namedtuple)
>> 136370.0090.0000.009

Re: [ovirt-users] [ovirt-devel] ovirt-engine-sdk-python too slow

[Nir Soffer]

Thanks John, very interesting results.

On Thu, Nov 26, 2015 at 3:17 AM, John Hunter  wrote:

> Hi Juan,
>
> On Thu, Nov 26, 2015 at 2:15 AM, Juan Hernández 
> wrote:
>
>> On 11/25/2015 06:45 PM, Nir Soffer wrote:
>> > $ ./profile-stats -c myscript.prof
>> >
>> > Wed Nov 25 10:40:11 2015myscript.prof
>> >
>> >  7892315 function calls (7891054 primitive calls) in 7.940
>> seconds
>> >
>> >Ordered by: internal time
>> >List reduced from 1518 to 20 due to restriction <20>
>> >
>> >ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>> >  90862.6930.0006.7060.001 inspect.py:247(getmembers)
>> >   19524941.3940.0001.8800.000 inspect.py:59(isclass)
>> >  90921.0300.0001.0300.000 {dir}
>> >   19526420.6000.0000.6000.000 {getattr}
>> >   19727650.5040.0000.5040.000 {isinstance}
>> > 30.3340.1110.3340.111 {method 'perform' of
>> > 'pycurl.Curl' objects}
>> >   18839180.2840.0000.2840.000 {method 'append' of 'list'
>> > objects}
>> >  90870.2210.0000.2210.000 {method 'sort' of 'list'
>> > objects}
>> >  90510.1720.0006.9110.001
>> > reflectionhelper.py:51(isModuleMember)
>> > 10.1240.1240.3540.354 errors.py:17()
>> > 10.0880.0880.2300.230 params.py:8()
>> >  88790.0700.0006.9980.001 params.py:367(__setattr__)
>> > 10.0470.0475.1825.182 api.py:23()
>> > 10.0250.0254.7434.743 brokers.py:22()
>> > 10.0230.0230.0300.030
>> > connectionspool.py:17()
>> > 10.0220.0220.0530.053
>> > lxml.etree.pyx:1(PyMODINIT_FUNC PyInit_etree(void))
>> >   1180.0190.0004.6840.040 params.py:45277(__init__)
>> > 50.0150.0030.0240.005 {built-in method strptime}
>> > 10.0120.0120.0130.013 socket.py:45()
>> >100.0110.0010.0150.002
>> collections.py:288(namedtuple)
>> >
>> > So it is not the classes, it is the code inspecting them on import.
>> >
>>
>> The script doesn't contain only the imports, it is also calling the
>> server, and we know parsing the result is slow, due to the excesive use
>> of "inspect", as I mentioned before:
>>
>>   [RFE][performance] - generate large scale list running to slow.
>>   https://bugzilla.redhat.com/show_bug.cgi?id=1221238#c2
>>
>> In the profiling information seems to corresponds to the script before
>> commenting out the part that lists all the VMs, as it looks like the
>> constructor of the VM class was called 21 times (you probably have 21
>> VMs):
>>
>>   21 0.004 1.308
>> build/bdist.linux-x86_64/egg/ovirtsdk/infrastructure/brokers.py:29139(VM)
>>
>> Actually I only have one VM running on the server.
> This time it contains only the import in the script, not calling the
> server. It shows:
> 210.0051.666  brokers.py:29139(VM)
>
> $./profile-stats -c myscript_contains_only_import.prof
>
> Thu Nov 26 09:11:59 2015myscript_contains_only_import.prof
>
>  5453977 function calls (5452849 primitive calls) in 5.463 seconds
>
>Ordered by: internal time
>List reduced from 1083 to 20 due to restriction <20>
>
>ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>  74682.0290.0004.9140.001 inspect.py:247(getmembers)
>   13486780.9750.0001.3420.000 inspect.py:59(isclass)
>  74740.7370.0000.7370.000 {dir}
>   13488250.4330.0000.4330.000 {getattr}
>   13659700.3830.0000.3830.000 {isinstance}
>   12934550.2110.0000.2110.000 {method 'append' of 'list'
> objects}
>  74690.1630.0000.1630.000 {method 'sort' of 'list'
> objects}
>  74340.1390.0005.0820.001
> reflectionhelper.py:51(isModuleMember)
>  76700.0610.0005.1580.001 params.py:367(__setattr__)
> 10.0430.0435.4635.463 api.py:23()
> 10.0420.0420.1390.139 errors.py:17()
> 10.0270.0275.2485.248 brokers.py:22()
> 10.0260.0260.0970.097 params.py:8()
>   1180.0230.0005.1870.044 params.py:45277(__init__)
> 10.0180.0180.0250.025
> connectionspool.py:17()
> 10.0130.0130.0130.013 socket.py:45()
>100.0100.0010.0130.001
> collections.py:288(namedtuple)
> 136370.0090.0000.0090.000 {method 'lower' of 'str'
> objects}
>350.0090.0000.0350.001
> reflectionhelper.py:28(getClasses)
>173/270.0080.0000.0190.001 sre_parse.py:388(_parse)
>
>
>Ordered by: internal time
>List reduced from 1083 to 20 due to restriction <20>
>

Re: [ovirt-users] [ovirt-devel] ovirt-engine-sdk-python too slow

[Nir Soffer]

$ ./profile-stats -c myscript.prof

Wed Nov 25 10:40:11 2015myscript.prof

 7892315 function calls (7891054 primitive calls) in 7.940 seconds

   Ordered by: internal time
   List reduced from 1518 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 90862.6930.0006.7060.001 inspect.py:247(getmembers)
  19524941.3940.0001.8800.000 inspect.py:59(isclass)
 90921.0300.0001.0300.000 {dir}
  19526420.6000.0000.6000.000 {getattr}
  19727650.5040.0000.5040.000 {isinstance}
30.3340.1110.3340.111 {method 'perform' of
'pycurl.Curl' objects}
  18839180.2840.0000.2840.000 {method 'append' of 'list'
objects}
 90870.2210.0000.2210.000 {method 'sort' of 'list'
objects}
 90510.1720.0006.9110.001
reflectionhelper.py:51(isModuleMember)
10.1240.1240.3540.354 errors.py:17()
10.0880.0880.2300.230 params.py:8()
 88790.0700.0006.9980.001 params.py:367(__setattr__)
10.0470.0475.1825.182 api.py:23()
10.0250.0254.7434.743 brokers.py:22()
10.0230.0230.0300.030
connectionspool.py:17()
10.0220.0220.0530.053
lxml.etree.pyx:1(PyMODINIT_FUNC PyInit_etree(void))
  1180.0190.0004.6840.040 params.py:45277(__init__)
50.0150.0030.0240.005 {built-in method strptime}
10.0120.0120.0130.013 socket.py:45()
   100.0110.0010.0150.002 collections.py:288(namedtuple)

So it is not the classes, it is the code inspecting them on import.

Nir

On Wed, Nov 25, 2015 at 7:49 AM, John Hunter  wrote:

> Hi Nir,
>
> Attachment is my script and its profile.
> Thanks a lot about your help!
>
> On Wed, Nov 25, 2015 at 5:49 AM, Nir Soffer  wrote:
>
>> On Tue, Nov 24, 2015 at 3:49 PM, John Hunter  wrote:
>>
>>>
>>>
>>> On Tue, Nov 24, 2015 at 9:15 PM, Juan Hernández 
>>> wrote:
>>>
 On 11/24/2015 01:40 PM, John Hunter wrote:
 > Hi,
 >
 > On Tue, Nov 24, 2015 at 5:18 PM, Oved Ourfali  > wrote:
 >
 > Hi
 >
 > I discussed it with Juan (cc-ed).
 >
 > There used to be a bug in the JDBC authenticion extension that
 > artificially delayed RESTAPI responses by 5 seconds:
 >
 >   brute force prevention login delay should not be applied to
 successful
 > login requests
 >   https://bugzilla.redhat.com/1255814
 >
 > That matches the description of the issue, but in theory it has
 been
 > fixed. I would suggest him to check that he is using the right
 version
 > of the extension.
 >
 > I did not use the extension ovirt-engine-extension-aaa-jdbc, and I
 don't
 > think this bug matches my problem, because even there is only one line
 > in the python script, it still cost like 3 seconds, I don't think
 this is a
 > reasonable time as when I import other package, it cost almost no
 time.
 >
 > Can you explain why this import line costs so much time?
 >

 If you are using 3.6 then you are using ovirt-engine-extension-aaa-jdbc,
 as it is enabled by default, but looks like it isn't related to your
 problem.

 That line takes a long time to execute because it has to process two
 large Python modules: the "params" module that contains a class per each
 type used by the API (393 classes) and the "brokers" module that
 contains a class per each resource used by the API (358 classes). That
 makes a total of 751 classes. In my environment it takes 0.9 seconds,
 approx. You may want to use the python profile in your environment and
 share the results:

 $ cat > profile.py <<.
 import cProfile
 cProfile.run("from ovirtsdk.api import API")
 .

 $ python profile.py

 I won't be surprised to see this taking those 3 seconds in a slower
 environment.

 But even if this takes those 3 seconds it shouldn't be a big problem,
 because you shouldn't be running that "from ... import ..." line
 frequently. Your program should do this once only, when it starts.

 Assume that I have two functions to implement, one is to list all the
>>> vms belong
>>> to the user, and the other is to retrieve one vm's virt-viewer
>>> connection file, as
>>> far as I can see, I have to write two python scripts and import the
>>> ovirtsdk.api in both
>>> scripts, each script has to take the 3 seconds :(
>>>
>>
>> No, you have two functions, which can be in the same module, or in
>> different modules, depending on how you want to organize 

Re: [ovirt-users] [ovirt-devel] ovirt-engine-sdk-python too slow

[Juan Hernández]

On 11/25/2015 06:45 PM, Nir Soffer wrote:
> $ ./profile-stats -c myscript.prof
> 
> Wed Nov 25 10:40:11 2015myscript.prof
> 
>  7892315 function calls (7891054 primitive calls) in 7.940 seconds
> 
>Ordered by: internal time
>List reduced from 1518 to 20 due to restriction <20>
> 
>ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>  90862.6930.0006.7060.001 inspect.py:247(getmembers)
>   19524941.3940.0001.8800.000 inspect.py:59(isclass)
>  90921.0300.0001.0300.000 {dir}
>   19526420.6000.0000.6000.000 {getattr}
>   19727650.5040.0000.5040.000 {isinstance}
> 30.3340.1110.3340.111 {method 'perform' of
> 'pycurl.Curl' objects}
>   18839180.2840.0000.2840.000 {method 'append' of 'list'
> objects}
>  90870.2210.0000.2210.000 {method 'sort' of 'list'
> objects}
>  90510.1720.0006.9110.001
> reflectionhelper.py:51(isModuleMember)
> 10.1240.1240.3540.354 errors.py:17()
> 10.0880.0880.2300.230 params.py:8()
>  88790.0700.0006.9980.001 params.py:367(__setattr__)
> 10.0470.0475.1825.182 api.py:23()
> 10.0250.0254.7434.743 brokers.py:22()
> 10.0230.0230.0300.030
> connectionspool.py:17()
> 10.0220.0220.0530.053
> lxml.etree.pyx:1(PyMODINIT_FUNC PyInit_etree(void))
>   1180.0190.0004.6840.040 params.py:45277(__init__)
> 50.0150.0030.0240.005 {built-in method strptime}
> 10.0120.0120.0130.013 socket.py:45()
>100.0110.0010.0150.002 collections.py:288(namedtuple)
> 
> So it is not the classes, it is the code inspecting them on import.
> 

The script doesn't contain only the imports, it is also calling the
server, and we know parsing the result is slow, due to the excesive use
of "inspect", as I mentioned before:

  [RFE][performance] - generate large scale list running to slow.
  https://bugzilla.redhat.com/show_bug.cgi?id=1221238#c2

In the profiling information seems to corresponds to the script before
commenting out the part that lists all the VMs, as it looks like the
constructor of the VM class was called 21 times (you probably have 21 VMs):

  21 0.004 1.308
build/bdist.linux-x86_64/egg/ovirtsdk/infrastructure/brokers.py:29139(VM)

> Nir
> 
> On Wed, Nov 25, 2015 at 7:49 AM, John Hunter  > wrote:
> 
> Hi Nir,
> 
> Attachment is my script and its profile.
> Thanks a lot about your help!
> 
> On Wed, Nov 25, 2015 at 5:49 AM, Nir Soffer  > wrote:
> 
> On Tue, Nov 24, 2015 at 3:49 PM, John Hunter  > wrote:
> 
> 
> 
> On Tue, Nov 24, 2015 at 9:15 PM, Juan Hernández
> > wrote:
> 
> On 11/24/2015 01:40 PM, John Hunter wrote:
> > Hi,
> >
> > On Tue, Nov 24, 2015 at 5:18 PM, Oved Ourfali 
> 
> > >> 
> wrote:
> >
> > Hi
> >
> > I discussed it with Juan (cc-ed).
> >
> > There used to be a bug in the JDBC authenticion 
> extension that
> > artificially delayed RESTAPI responses by 5 seconds:
> >
> >   brute force prevention login delay should not be 
> applied to successful
> > login requests
> >   https://bugzilla.redhat.com/1255814
> >
> > That matches the description of the issue, but in 
> theory it has been
> > fixed. I would suggest him to check that he is using 
> the right version
> > of the extension.
> >
> > I did not use the extension 
> ovirt-engine-extension-aaa-jdbc, and I don't
> > think this bug matches my problem, because even there is 
> only one line
> > in the python script, it still cost like 3 seconds, I don't 
> think this is a
> > reasonable time as when I import other package, it cost 
> almost no time.
> >
> > Can you explain why this import line costs so much time?
> >
> 
> If you are using 3.6 then you are using
> ovirt-engine-extension-aaa-jdbc,
> as it is enabled by default, but looks like it isn't
> related to your
> problem.
> 
>

Re: [ovirt-users] [ovirt-devel] ovirt-engine-sdk-python too slow

[Nir Soffer]

On Tue, Nov 24, 2015 at 3:49 PM, John Hunter  wrote:

>
>
> On Tue, Nov 24, 2015 at 9:15 PM, Juan Hernández 
> wrote:
>
>> On 11/24/2015 01:40 PM, John Hunter wrote:
>> > Hi,
>> >
>> > On Tue, Nov 24, 2015 at 5:18 PM, Oved Ourfali > > > wrote:
>> >
>> > Hi
>> >
>> > I discussed it with Juan (cc-ed).
>> >
>> > There used to be a bug in the JDBC authenticion extension that
>> > artificially delayed RESTAPI responses by 5 seconds:
>> >
>> >   brute force prevention login delay should not be applied to
>> successful
>> > login requests
>> >   https://bugzilla.redhat.com/1255814
>> >
>> > That matches the description of the issue, but in theory it has been
>> > fixed. I would suggest him to check that he is using the right
>> version
>> > of the extension.
>> >
>> > I did not use the extension ovirt-engine-extension-aaa-jdbc, and I don't
>> > think this bug matches my problem, because even there is only one line
>> > in the python script, it still cost like 3 seconds, I don't think this
>> is a
>> > reasonable time as when I import other package, it cost almost no time.
>> >
>> > Can you explain why this import line costs so much time?
>> >
>>
>> If you are using 3.6 then you are using ovirt-engine-extension-aaa-jdbc,
>> as it is enabled by default, but looks like it isn't related to your
>> problem.
>>
>> That line takes a long time to execute because it has to process two
>> large Python modules: the "params" module that contains a class per each
>> type used by the API (393 classes) and the "brokers" module that
>> contains a class per each resource used by the API (358 classes). That
>> makes a total of 751 classes. In my environment it takes 0.9 seconds,
>> approx. You may want to use the python profile in your environment and
>> share the results:
>>
>> $ cat > profile.py <<.
>> import cProfile
>> cProfile.run("from ovirtsdk.api import API")
>> .
>>
>> $ python profile.py
>>
>> I won't be surprised to see this taking those 3 seconds in a slower
>> environment.
>>
>> But even if this takes those 3 seconds it shouldn't be a big problem,
>> because you shouldn't be running that "from ... import ..." line
>> frequently. Your program should do this once only, when it starts.
>>
>> Assume that I have two functions to implement, one is to list all the vms
> belong
> to the user, and the other is to retrieve one vm's virt-viewer connection
> file, as
> far as I can see, I have to write two python scripts and import the
> ovirtsdk.api in both
> scripts, each script has to take the 3 seconds :(
>

No, you have two functions, which can be in the same module, or in
different modules, depending on how you want to organize your code.

Python imports are done only once, on the first time you import a module
anywhere. The module is stored in sys.modules, and the next import fetch
the module object from sys.modules.

Can you share a profile of a simple script importing the Python sdk?

Do:

python -m cProfile -o myscript.prof myscript.py

The profile will be stored in myscript.prof.

For viewing the profile, I recommend to use the profile-stats tool:
https://github.com/oVirt/vdsm/blob/master/contrib/profile-stats


>
> How can I run the "from ... import ..." just once ?
>
>
>> >
>> > In addition we also know that retrieving large lists of objects
>> with the
>> > SDK is slow:
>> >
>> >[RFE][performance] - generate large scale list running to slow.
>> >https://bugzilla.redhat.com/1221238
>> >
>> > We don't have a solution for that yet.
>> >
>> > CC-ing Juan in case you have additional questions.
>> >
>> >
>> > On Mon, Nov 23, 2015 at 11:27 AM, John Hunter > > > wrote:
>> >
>> > Hi guys,
>> >
>> > I am using the ovirt-engine-sdk-python to communicate with the
>> > ovirt-engine,
>> > I am ok to list the vms but the processing time is too long,
>> > like 4.5 seconds,
>> > and this line:
>> > from ovirtsdk.api import API
>> > take almost 3 seconds.
>> >
>> > This seems a little bit longer than I expected it to be, so I am
>> > asking is there
>> > a quicker way to communicate with the ovirt-engine?
>> >
>>
>> --
>> Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
>> 3ºD, 28016 Madrid, Spain
>> Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
>>
>
>
>
> --
> Best regards
> Junwang Zhao
> Department of Computer Science 
> Peking University
> Beijing, 100871, PRC
>
> ___
> Devel mailing list
> de...@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] [ovirt-devel] ovirt-engine-sdk-python too slow

[John Hunter]

On Tue, Nov 24, 2015 at 9:56 PM, Juan Hernández  wrote:

> On 11/24/2015 02:49 PM, John Hunter wrote:
> >
> >
> > On Tue, Nov 24, 2015 at 9:15 PM, Juan Hernández  > > wrote:
> >
> > On 11/24/2015 01:40 PM, John Hunter wrote:
> > > Hi,
> > >
> > > On Tue, Nov 24, 2015 at 5:18 PM, Oved Ourfali  
> > > >> wrote:
> > >
> > > Hi
> > >
> > > I discussed it with Juan (cc-ed).
> > >
> > > There used to be a bug in the JDBC authenticion extension that
> > > artificially delayed RESTAPI responses by 5 seconds:
> > >
> > >   brute force prevention login delay should not be applied to
> successful
> > > login requests
> > >   https://bugzilla.redhat.com/1255814
> > >
> > > That matches the description of the issue, but in theory it
> has been
> > > fixed. I would suggest him to check that he is using the right
> version
> > > of the extension.
> > >
> > > I did not use the extension ovirt-engine-extension-aaa-jdbc, and I
> don't
> > > think this bug matches my problem, because even there is only one
> line
> > > in the python script, it still cost like 3 seconds, I don't think
> this is a
> > > reasonable time as when I import other package, it cost almost no
> time.
> > >
> > > Can you explain why this import line costs so much time?
> > >
> >
> > If you are using 3.6 then you are using
> ovirt-engine-extension-aaa-jdbc,
> > as it is enabled by default, but looks like it isn't related to your
> > problem.
> >
> > That line takes a long time to execute because it has to process two
> > large Python modules: the "params" module that contains a class per
> each
> > type used by the API (393 classes) and the "brokers" module that
> > contains a class per each resource used by the API (358 classes).
> That
> > makes a total of 751 classes. In my environment it takes 0.9 seconds,
> > approx. You may want to use the python profile in your environment
> and
> > share the results:
> >
> > $ cat > profile.py <<.
> > import cProfile
> > cProfile.run("from ovirtsdk.api import API")
> > .
> >
> > $ python profile.py
> >
> > I won't be surprised to see this taking those 3 seconds in a slower
> > environment.
> >
> > But even if this takes those 3 seconds it shouldn't be a big problem,
> > because you shouldn't be running that "from ... import ..." line
> > frequently. Your program should do this once only, when it starts.
> >
> > Assume that I have two functions to implement, one is to list all the
> > vms belong
> > to the user, and the other is to retrieve one vm's virt-viewer
> > connection file, as
> > far as I can see, I have to write two python scripts and import the
> > ovirtsdk.api in both
> > scripts, each script has to take the 3 seconds :(
> >
> > How can I run the "from ... import ..." just once ?
> >
>
> I don't know what technology or tools are you using to write that
> program, but if you are using Python then you don't need to import it
> twice.
>
> Yeah, I am just using Python scripts to access the RESTAPI, since I am
not skillful in python, I might have asked some stupid question.

I will try to learn more about Python.

Anyway, thanks for your patience.


> If for whatever the reason you decide to call external python scripts
> from another program then you will have to pay the price of the startup
> of the Python SDK. If that is unacceptable because of performance then
> you should look for a different way to access the RESTAPI, like sending
> XML or JSON directly, or using the Java SDK, or rbovirt, it all depends
> on the technology that you are using for your application.
>
> >
> > >
> > > In addition we also know that retrieving large lists of
> objects with the
> > > SDK is slow:
> > >
> > >[RFE][performance] - generate large scale list running to
> slow.
> > >https://bugzilla.redhat.com/1221238
> > >
> > > We don't have a solution for that yet.
> > >
> > > CC-ing Juan in case you have additional questions.
> > >
> > >
> > > On Mon, Nov 23, 2015 at 11:27 AM, John Hunter <
> zhjw...@gmail.com 
> > > >> wrote:
> > >
> > > Hi guys,
> > >
> > > I am using the ovirt-engine-sdk-python to communicate with
> the
> > > ovirt-engine,
> > > I am ok to list the vms but the processing time is too
> long,
> > > like 4.5 seconds,
> > > and this line:
> > > from ovirtsdk.api import API
> > > take almost 3 seconds.
> > >
> > > 

Re: [ovirt-users] [ovirt-devel] ovirt-engine-sdk-python too slow

[John Hunter]

Hi,

On Tue, Nov 24, 2015 at 5:18 PM, Oved Ourfali  wrote:

> Hi
>
> I discussed it with Juan (cc-ed).
>
> There used to be a bug in the JDBC authenticion extension that
> artificially delayed RESTAPI responses by 5 seconds:
>
>   brute force prevention login delay should not be applied to successful
> login requests
>   https://bugzilla.redhat.com/1255814
>
> That matches the description of the issue, but in theory it has been
> fixed. I would suggest him to check that he is using the right version
> of the extension.
>
> I did not use the extension ovirt-engine-extension-aaa-jdbc, and I don't
think this bug matches my problem, because even there is only one line
in the python script, it still cost like 3 seconds, I don't think this is a
reasonable time as when I import other package, it cost almost no time.

Can you explain why this import line costs so much time?


> In addition we also know that retrieving large lists of objects with the
> SDK is slow:
>
>[RFE][performance] - generate large scale list running to slow.
>https://bugzilla.redhat.com/1221238
>
> We don't have a solution for that yet.
>
> CC-ing Juan in case you have additional questions.
>
>
> On Mon, Nov 23, 2015 at 11:27 AM, John Hunter  wrote:
>
>> Hi guys,
>>
>> I am using the ovirt-engine-sdk-python to communicate with the
>> ovirt-engine,
>> I am ok to list the vms but the processing time is too long, like 4.5
>> seconds,
>> and this line:
>> from ovirtsdk.api import API
>> take almost 3 seconds.
>>
>> This seems a little bit longer than I expected it to be, so I am asking
>> is there
>> a quicker way to communicate with the ovirt-engine?
>>
>> --
>> Best regards
>> Junwang Zhao
>> Department of Computer Science 
>> Peking University
>> Beijing, 100871, PRC
>>
>> ___
>> Devel mailing list
>> de...@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>


-- 
Best regards
Junwang Zhao
Department of Computer Science 
Peking University
Beijing, 100871, PRC
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] [ovirt-devel] ovirt-engine-sdk-python too slow

[Juan Hernández]

On 11/24/2015 02:49 PM, John Hunter wrote:
> 
> 
> On Tue, Nov 24, 2015 at 9:15 PM, Juan Hernández  > wrote:
> 
> On 11/24/2015 01:40 PM, John Hunter wrote:
> > Hi,
> >
> > On Tue, Nov 24, 2015 at 5:18 PM, Oved Ourfali  
> > >> wrote:
> >
> > Hi
> >
> > I discussed it with Juan (cc-ed).
> >
> > There used to be a bug in the JDBC authenticion extension that
> > artificially delayed RESTAPI responses by 5 seconds:
> >
> >   brute force prevention login delay should not be applied to 
> successful
> > login requests
> >   https://bugzilla.redhat.com/1255814
> >
> > That matches the description of the issue, but in theory it has been
> > fixed. I would suggest him to check that he is using the right 
> version
> > of the extension.
> >
> > I did not use the extension ovirt-engine-extension-aaa-jdbc, and I don't
> > think this bug matches my problem, because even there is only one line
> > in the python script, it still cost like 3 seconds, I don't think this 
> is a
> > reasonable time as when I import other package, it cost almost no time.
> >
> > Can you explain why this import line costs so much time?
> >
> 
> If you are using 3.6 then you are using ovirt-engine-extension-aaa-jdbc,
> as it is enabled by default, but looks like it isn't related to your
> problem.
> 
> That line takes a long time to execute because it has to process two
> large Python modules: the "params" module that contains a class per each
> type used by the API (393 classes) and the "brokers" module that
> contains a class per each resource used by the API (358 classes). That
> makes a total of 751 classes. In my environment it takes 0.9 seconds,
> approx. You may want to use the python profile in your environment and
> share the results:
> 
> $ cat > profile.py <<.
> import cProfile
> cProfile.run("from ovirtsdk.api import API")
> .
> 
> $ python profile.py
> 
> I won't be surprised to see this taking those 3 seconds in a slower
> environment.
> 
> But even if this takes those 3 seconds it shouldn't be a big problem,
> because you shouldn't be running that "from ... import ..." line
> frequently. Your program should do this once only, when it starts.
> 
> Assume that I have two functions to implement, one is to list all the
> vms belong
> to the user, and the other is to retrieve one vm's virt-viewer
> connection file, as 
> far as I can see, I have to write two python scripts and import the
> ovirtsdk.api in both
> scripts, each script has to take the 3 seconds :(
> 
> How can I run the "from ... import ..." just once ?
>  

I don't know what technology or tools are you using to write that
program, but if you are using Python then you don't need to import it twice.

If for whatever the reason you decide to call external python scripts
from another program then you will have to pay the price of the startup
of the Python SDK. If that is unacceptable because of performance then
you should look for a different way to access the RESTAPI, like sending
XML or JSON directly, or using the Java SDK, or rbovirt, it all depends
on the technology that you are using for your application.

> 
> >
> > In addition we also know that retrieving large lists of objects 
> with the
> > SDK is slow:
> >
> >[RFE][performance] - generate large scale list running to slow.
> >https://bugzilla.redhat.com/1221238
> >
> > We don't have a solution for that yet.
> >
> > CC-ing Juan in case you have additional questions.
> >
> >
> > On Mon, Nov 23, 2015 at 11:27 AM, John Hunter  
> > >> wrote:
> >
> > Hi guys,
> >
> > I am using the ovirt-engine-sdk-python to communicate with the
> > ovirt-engine,
> > I am ok to list the vms but the processing time is too long,
> > like 4.5 seconds,
> > and this line:
> > from ovirtsdk.api import API
> > take almost 3 seconds.
> >
> > This seems a little bit longer than I expected it to be, so I am
> > asking is there
> > a quicker way to communicate with the ovirt-engine?
> >
> 

-- 
Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
3ºD, 28016 Madrid, Spain
Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] [ovirt-devel] ovirt-engine-sdk-python too slow

[Juan Hernández]

On 11/24/2015 01:40 PM, John Hunter wrote:
> Hi,
> 
> On Tue, Nov 24, 2015 at 5:18 PM, Oved Ourfali  > wrote:
> 
> Hi
> 
> I discussed it with Juan (cc-ed).
> 
> There used to be a bug in the JDBC authenticion extension that
> artificially delayed RESTAPI responses by 5 seconds:
> 
>   brute force prevention login delay should not be applied to successful
> login requests
>   https://bugzilla.redhat.com/1255814
> 
> That matches the description of the issue, but in theory it has been
> fixed. I would suggest him to check that he is using the right version
> of the extension.
> 
> I did not use the extension ovirt-engine-extension-aaa-jdbc, and I don't 
> think this bug matches my problem, because even there is only one line
> in the python script, it still cost like 3 seconds, I don't think this is a
> reasonable time as when I import other package, it cost almost no time.
> 
> Can you explain why this import line costs so much time?
> 

If you are using 3.6 then you are using ovirt-engine-extension-aaa-jdbc,
as it is enabled by default, but looks like it isn't related to your
problem.

That line takes a long time to execute because it has to process two
large Python modules: the "params" module that contains a class per each
type used by the API (393 classes) and the "brokers" module that
contains a class per each resource used by the API (358 classes). That
makes a total of 751 classes. In my environment it takes 0.9 seconds,
approx. You may want to use the python profile in your environment and
share the results:

$ cat > profile.py <<.
import cProfile
cProfile.run("from ovirtsdk.api import API")
.

$ python profile.py

I won't be surprised to see this taking those 3 seconds in a slower
environment.

But even if this takes those 3 seconds it shouldn't be a big problem,
because you shouldn't be running that "from ... import ..." line
frequently. Your program should do this once only, when it starts.

> 
> In addition we also know that retrieving large lists of objects with the
> SDK is slow:
> 
>[RFE][performance] - generate large scale list running to slow.
>https://bugzilla.redhat.com/1221238
> 
> We don't have a solution for that yet.
> 
> CC-ing Juan in case you have additional questions.
> 
> 
> On Mon, Nov 23, 2015 at 11:27 AM, John Hunter  > wrote:
> 
> Hi guys,
> 
> I am using the ovirt-engine-sdk-python to communicate with the
> ovirt-engine,
> I am ok to list the vms but the processing time is too long,
> like 4.5 seconds,
> and this line:
> from ovirtsdk.api import API
> take almost 3 seconds.
> 
> This seems a little bit longer than I expected it to be, so I am
> asking is there
> a quicker way to communicate with the ovirt-engine? 
> 

-- 
Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
3ºD, 28016 Madrid, Spain
Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] [ovirt-devel] ovirt-engine-sdk-python too slow

[John Hunter]

On Tue, Nov 24, 2015 at 9:15 PM, Juan Hernández  wrote:

> On 11/24/2015 01:40 PM, John Hunter wrote:
> > Hi,
> >
> > On Tue, Nov 24, 2015 at 5:18 PM, Oved Ourfali  > > wrote:
> >
> > Hi
> >
> > I discussed it with Juan (cc-ed).
> >
> > There used to be a bug in the JDBC authenticion extension that
> > artificially delayed RESTAPI responses by 5 seconds:
> >
> >   brute force prevention login delay should not be applied to
> successful
> > login requests
> >   https://bugzilla.redhat.com/1255814
> >
> > That matches the description of the issue, but in theory it has been
> > fixed. I would suggest him to check that he is using the right
> version
> > of the extension.
> >
> > I did not use the extension ovirt-engine-extension-aaa-jdbc, and I don't
> > think this bug matches my problem, because even there is only one line
> > in the python script, it still cost like 3 seconds, I don't think this
> is a
> > reasonable time as when I import other package, it cost almost no time.
> >
> > Can you explain why this import line costs so much time?
> >
>
> If you are using 3.6 then you are using ovirt-engine-extension-aaa-jdbc,
> as it is enabled by default, but looks like it isn't related to your
> problem.
>
> That line takes a long time to execute because it has to process two
> large Python modules: the "params" module that contains a class per each
> type used by the API (393 classes) and the "brokers" module that
> contains a class per each resource used by the API (358 classes). That
> makes a total of 751 classes. In my environment it takes 0.9 seconds,
> approx. You may want to use the python profile in your environment and
> share the results:
>
> $ cat > profile.py <<.
> import cProfile
> cProfile.run("from ovirtsdk.api import API")
> .
>
> $ python profile.py
>
> I won't be surprised to see this taking those 3 seconds in a slower
> environment.
>
> But even if this takes those 3 seconds it shouldn't be a big problem,
> because you shouldn't be running that "from ... import ..." line
> frequently. Your program should do this once only, when it starts.
>
> Assume that I have two functions to implement, one is to list all the vms
belong
to the user, and the other is to retrieve one vm's virt-viewer connection
file, as
far as I can see, I have to write two python scripts and import the
ovirtsdk.api in both
scripts, each script has to take the 3 seconds :(

How can I run the "from ... import ..." just once ?


> >
> > In addition we also know that retrieving large lists of objects with
> the
> > SDK is slow:
> >
> >[RFE][performance] - generate large scale list running to slow.
> >https://bugzilla.redhat.com/1221238
> >
> > We don't have a solution for that yet.
> >
> > CC-ing Juan in case you have additional questions.
> >
> >
> > On Mon, Nov 23, 2015 at 11:27 AM, John Hunter  > > wrote:
> >
> > Hi guys,
> >
> > I am using the ovirt-engine-sdk-python to communicate with the
> > ovirt-engine,
> > I am ok to list the vms but the processing time is too long,
> > like 4.5 seconds,
> > and this line:
> > from ovirtsdk.api import API
> > take almost 3 seconds.
> >
> > This seems a little bit longer than I expected it to be, so I am
> > asking is there
> > a quicker way to communicate with the ovirt-engine?
> >
>
> --
> Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
> 3ºD, 28016 Madrid, Spain
> Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
>



-- 
Best regards
Junwang Zhao
Department of Computer Science 
Peking University
Beijing, 100871, PRC
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users