Oleg Broytmann wrote:
On Wed, Jul 22, 2009 at 02:26:57PM -0400, Stef Telford wrote:
yes. evals appear to be  a 'bad' thing here :\
   Well, those evals are in sqlmeta.addColumn and .addJoin methods, so they
work once for every column in the table, but that's all. After the class
has been created and populated - whatever you do with rows (class instances)
- those evals are not executed.

Ah. hrm. *rubs chin* perhaps it's not the evals then. It seems that the instantiations get .. well .. 'slower' over time.

   Curiouser and curiouser. IWBN to find where the slowness is *near the
end* of the loop - i.e. when instantiation becomes really slow.

It could be purely a 'feeling' .. I don't have any numbers to back it up and I am not entirely sure -how- to benchmark that. This machine does have 8gb of ram in it, and a fairly beefy quad core. I have never seen any process get near memory exhaustion, which I could believe the calls to 'malloc' could slow down but.. yes.

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    40000    0.063    0.000    0.141    0.000 <string>:1(<lambda>)

   Can you sort by ncalls and cumtime columns, cut the first 100-200 lines
and post two tables here?

surely, please find the output attached at the bottom. It maybe also worth changing the Decimal call to using gmpy or .. something that's faster than Decimal perhaps ? I did notice a fair few cycles are spent in Decimal.__new__ .. I don't think that's the -entire- story, but it should help I would think.
        1    0.000    0.000   85.863   85.863 <string>:1(<module>)

   This is AFAIU the entire script? :)

Yes indeed.. that's the way I read the cProfile output as well :D

Regards
Stef
217419
         37287462 function calls (37280538 primitive calls) in 85.556 CPU 
seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   85.567   85.567 <string>:1(<module>)
        1    0.000    0.000   85.567   85.567 {execfile}
        1    0.221    0.221   85.567   85.567 t.t:3(<module>)
        1    0.127    0.127   83.346   83.346 sresults.py:175(__iter__)
    40001    0.308    0.000   79.876    0.002 dbconnection.py:649(next)
    40000    0.196    0.000   40.291    0.001 util.py:123(get)
    40000    0.375    0.000   39.282    0.001 main.py:872(get)
    40002   10.018    0.000   39.252    0.001 {method 'fetchone' of 
'psycopg2._psycopg.cursor' objects}
    40000    0.168    0.000   32.166    0.001 booking.py:336(_init)
  1840006   16.887    0.000   29.234    0.000 decimal.py:515(__new__)
    40000    0.132    0.000   23.477    0.001 util.py:224(_init)
    40000    0.214    0.000   23.323    0.001 main.py:912(_init)
    40000   10.475    0.000   23.069    0.001 main.py:1140(_SO_selectInit)
    40000    0.847    0.000    8.521    0.000 util.py:236(notify)
    40000    0.390    0.000    6.736    0.000 inspect.py:820(getframeinfo)
    40000    6.525    0.000    6.647    0.000 main.py:1171(__init__)
  1880365    4.854    0.000    4.854    0.000 {built-in method match}
    80417    1.025    0.000    4.195    0.000 inspect.py:381(getsourcefile)
    40000    0.533    0.000    3.903    0.000 inspect.py:454(findsource)
  7363357    3.786    0.000    3.786    0.000 {built-in method group}
        2    0.000    0.000    3.575    1.788 dbconnection.py:336(_executeRetry)
        2    3.575    1.788    3.575    1.788 {method 'execute' of 
'psycopg2._psycopg.cursor' objects}
        1    0.000    0.000    3.343    3.343 sresults.py:181(lazyIter)
        1    0.000    0.000    3.343    3.343 dbconnection.py:403(iterSelect)
        1    0.000    0.000    3.312    3.312 dbconnection.py:635(__init__)
  3381876    3.132    0.000    3.132    0.000 {setattr}
  5035316    2.900    0.000    2.900    0.000 {isinstance}
  1200000    1.856    0.000    2.769    0.000 col.py:1289(to_python)
  3362951    2.729    0.000    2.729    0.000 main.py:1673(instanceName)
    80448    0.491    0.000    1.658    0.000 inspect.py:342(getfile)
   640000    0.999    0.000    1.471    0.000 col.py:657(to_python)
        1    0.003    0.003    1.465    1.465 booking.py:4(<module>)
        2    0.003    0.001    1.318    0.659 invoice.py:4(<module>)
        1    0.000    0.000    1.309    1.309 transaction.py:5(<module>)
        1    0.001    0.001    1.308    1.308 package.py:4(<module>)
        1    0.001    0.001    1.296    1.296 sector.py:4(<module>)
   640000    0.983    0.000    1.280    0.000 col.py:573(to_python)
        1    0.001    0.001    1.250    1.250 employee.py:4(<module>)
  1840001    1.238    0.000    1.238    0.000 {method 'lstrip' of 'str' objects}
  551/279    0.004    0.000    1.207    0.004 declarative.py:81(__new__)
      273    0.016    0.000    1.174    0.004 main.py:728(__classinit__)
  1840630    1.161    0.000    1.161    0.000 {built-in method __new__ of type 
object at 0x814fa00}
        1    0.001    0.001    1.152    1.152 communication.py:4(<module>)
        1    0.001    0.001    1.128    1.128 agent.py:4(<module>)
        1    0.001    0.001    1.012    1.012 brochure.py:4(<module>)
        1    0.001    0.001    1.000    1.000 contact.py:4(<module>)
        1    0.001    0.001    0.943    0.943 feedback.py:1(<module>)
        1    0.001    0.001    0.910    0.910 trip.py:5(<module>)
     2801    0.102    0.000    0.858    0.000 main.py:307(addColumn)
    40002    0.054    0.000    0.790    0.000 sqlcursor.py:72(useMemcache)
    80432    0.130    0.000    0.737    0.000 posixpath.py:168(exists)
    40003    0.277    0.000    0.736    0.000 sqlcursor.py:31(hostSetting)
   440000    0.564    0.000    0.672    0.000 col.py:1031(to_python)
1532367/1531398    0.647    0.000    0.648    0.000 {len}
        1    0.001    0.001    0.617    0.617 commissions.py:1(<module>)
    80563    0.608    0.000    0.608    0.000 {posix.stat}
        1    0.003    0.003    0.594    0.594 service.py:6(<module>)
   321608    0.351    0.000    0.512    0.000 string.py:218(lower)
   120000    0.239    0.000    0.474    0.000 inspect.py:43(isclass)
   640000    0.472    0.000    0.472    0.000 decimal.py:718(__nonzero__)
    40139    0.062    0.000    0.460    0.000 re.py:186(compile)
40043/40000    0.155    0.000    0.436    0.000 inspect.py:408(getmodule)
   301683    0.375    0.000    0.406    0.000 {hasattr}
    40624    0.099    0.000    0.401    0.000 re.py:227(_compile)
     6390    0.317    0.000    0.328    0.000 {eval}
   440000    0.282    0.000    0.326    0.000 col.py:511(to_python)
   161288    0.210    0.000    0.316    0.000 inspect.py:35(ismodule)
        1    0.000    0.000    0.295    0.295 priceband.py:1(<module>)
        1    0.001    0.001    0.295    0.295 price.py:1(<module>)
   160000    0.197    0.000    0.291    0.000 inspect.py:139(istraceback)
    40000    0.113    0.000    0.284    0.000 
booking.py:2678(observedNewBookingNeedsOnlinePwd)
     2801    0.018    0.000    0.278    0.000 col.py:402(withClass)
   160000    0.200    0.000    0.272    0.000 inspect.py:149(isframe)
        1    0.000    0.000    0.268    0.268 sresults.py:203(count)
        1    0.000    0.000    0.268    0.268 sresults.py:189(accumulate)
        1    0.000    0.000    0.268    0.268 
dbconnection.py:407(accumulateSelect)
        1    0.000    0.000    0.266    0.266 dbconnection.py:389(queryOne)
        1    0.000    0.000    0.266    0.266 
dbconnection.py:257(_runWithConnection)
        1    0.000    0.000    0.266    0.266 dbconnection.py:379(_queryOne)
    80006    0.266    0.000    0.266    0.000 sqlcursor.py:21(hostSettings)
      142    0.002    0.000    0.265    0.002 sre_compile.py:501(compile)
    42652    0.247    0.000    0.247    0.000 {zip}
     2801    0.062    0.000    0.238    0.000 col.py:97(__init__)
    40000    0.145    0.000    0.234    0.000 
booking.py:2653(observedDocRequirementsChanged)
        3    0.003    0.001    0.214    0.071 __init__.py:3(<module>)
   120000    0.144    0.000    0.212    0.000 inspect.py:51(ismethod)
   120000    0.143    0.000    0.209    0.000 inspect.py:126(isfunction)
        1    0.003    0.003    0.207    0.207 main.py:27(<module>)
   120000    0.143    0.000    0.196    0.000 inspect.py:167(iscode)
   280932    0.186    0.000    0.191    0.000 {getattr}
        1    0.003    0.003    0.167    0.167 dbconnection.py:1(<module>)
   327248    0.165    0.000    0.165    0.000 {method 'lower' of 'str' objects}
    80417    0.162    0.000    0.162    0.000 {imp.get_suffixes}
    80008    0.153    0.000    0.153    0.000 {posix.uname}
217419
         37287474 function calls (37280550 primitive calls) in 85.977 CPU 
seconds

   Ordered by: call count

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  7363357    3.874    0.000    3.874    0.000 {built-in method group}
  5035316    2.905    0.000    2.905    0.000 {isinstance}
  3381876    3.180    0.000    3.180    0.000 {setattr}
  3362951    2.869    0.000    2.869    0.000 main.py:1673(instanceName)
  1880365    4.896    0.000    4.896    0.000 {built-in method match}
  1840630    1.196    0.000    1.196    0.000 {built-in method __new__ of type 
object at 0x814fa00}
  1840006   17.037    0.000   29.584    0.000 decimal.py:515(__new__)
  1840001    1.268    0.000    1.268    0.000 {method 'lstrip' of 'str' objects}
1532367/1531398    0.664    0.000    0.665    0.000 {len}
  1200000    1.849    0.000    2.764    0.000 col.py:1289(to_python)
   640000    0.985    0.000    1.468    0.000 col.py:657(to_python)
   640000    0.483    0.000    0.483    0.000 decimal.py:718(__nonzero__)
   640000    0.987    0.000    1.288    0.000 col.py:573(to_python)
   440000    0.566    0.000    0.675    0.000 col.py:1031(to_python)
   440000    0.286    0.000    0.329    0.000 col.py:511(to_python)
   327254    0.169    0.000    0.169    0.000 {method 'lower' of 'str' objects}
   321608    0.351    0.000    0.517    0.000 string.py:218(lower)
   301683    0.376    0.000    0.407    0.000 {hasattr}
   280932    0.185    0.000    0.190    0.000 {getattr}
   280000    0.137    0.000    0.137    0.000 sqlcursor.py:93(debugObserver)
   161288    0.209    0.000    0.317    0.000 inspect.py:35(ismodule)
   160000    0.198    0.000    0.271    0.000 inspect.py:149(isframe)
   160000    0.196    0.000    0.291    0.000 inspect.py:139(istraceback)
   135452    0.087    0.000    0.087    0.000 {method 'get' of 'dict' objects}
   133837    0.065    0.000    0.065    0.000 {method 'has_key' of 'dict' 
objects}
   120000    0.144    0.000    0.214    0.000 inspect.py:51(ismethod)
   120000    0.236    0.000    0.470    0.000 inspect.py:43(isclass)
   120000    0.142    0.000    0.209    0.000 inspect.py:126(isfunction)
   120000    0.143    0.000    0.195    0.000 inspect.py:167(iscode)
    84976    0.045    0.000    0.045    0.000 {method 'startswith' of 'str' 
objects}
    80614    0.061    0.000    0.061    0.000 {max}
    80563    0.602    0.000    0.602    0.000 {posix.stat}
    80448    0.494    0.000    1.658    0.000 inspect.py:342(getfile)
    80432    0.127    0.000    0.728    0.000 posixpath.py:168(exists)
    80417    0.161    0.000    0.161    0.000 {imp.get_suffixes}
    80417    1.012    0.000    4.180    0.000 inspect.py:381(getsourcefile)
    80008    0.155    0.000    0.155    0.000 {posix.uname}
    80006    0.263    0.000    0.263    0.000 sqlcursor.py:21(hostSettings)
    80005    0.045    0.000    0.045    0.000 sqlcursor.py:97(globalQueryCache)
    80001    0.051    0.000    0.051    0.000 {_weakref.proxy}
    44610    0.029    0.000    0.029    0.000 {min}
    43720    0.031    0.000    0.031    0.000 {method 'keys' of 'dict' objects}
    42652    0.243    0.000    0.243    0.000 {zip}
    40624    0.098    0.000    0.384    0.000 re.py:227(_compile)
    40139    0.065    0.000    0.447    0.000 re.py:186(compile)
40043/40000    0.157    0.000    0.438    0.000 inspect.py:408(getmodule)
    40003    0.035    0.000    0.035    0.000 {sys._getframe}
    40003    0.280    0.000    0.739    0.000 sqlcursor.py:31(hostSetting)
    40002    9.936    0.000   39.520    0.001 {method 'fetchone' of 
'psycopg2._psycopg.cursor' objects}
    40002    0.054    0.000    0.793    0.000 sqlcursor.py:72(useMemcache)
    40001    0.319    0.000   80.347    0.002 dbconnection.py:649(next)
    40000    0.113    0.000    0.279    0.000 
booking.py:2678(observedNewBookingNeedsOnlinePwd)
    40000    0.853    0.000    8.519    0.000 util.py:236(notify)
    40000    0.146    0.000    0.236    0.000 
booking.py:2653(observedDocRequirementsChanged)
    40000    0.036    0.000    0.037    0.000 linecache.py:33(getlines)
    40000    0.391    0.000    6.730    0.000 inspect.py:820(getframeinfo)
    40000    0.538    0.000    3.911    0.000 inspect.py:454(findsource)
    40000    0.076    0.000    0.106    0.000 main.py:251(__init__)
    40000    0.072    0.000    0.094    0.000 main.py:1679(__init__)
    40000   10.476    0.000   23.260    0.001 main.py:1140(_SO_selectInit)
    40000    0.168    0.000   32.353    0.001 booking.py:336(_init)
    40000    0.212    0.000   23.512    0.001 main.py:912(_init)
    40000    6.530    0.000    6.654    0.000 main.py:1171(__init__)
    40000    0.022    0.000    0.022    0.000 sqlcursor.py:89(useObservers)
    40000    0.052    0.000    0.079    0.000 main.py:939(_SO_loadValue)
    40000    0.192    0.000   40.484    0.001 util.py:123(get)
    40000    0.063    0.000    0.142    0.000 <string>:1(<lambda>)
    40000    0.131    0.000   23.666    0.001 util.py:224(_init)
    40000    0.040    0.000    0.040    0.000 dbconnection.py:831(__set__)
    40000    0.376    0.000   39.477    0.001 main.py:872(get)
    39578    0.017    0.000    0.017    0.000 {method 'append' of 'list' 
objects}
    18094    0.011    0.000    0.011    0.000 {method 'endswith' of 'str' 
objects}
    13411    0.006    0.000    0.006    0.000 {id}
    13408    0.060    0.000    0.066    0.000 dispatcher.py:236(getReceivers)
    10900    0.026    0.000    0.035    0.000 sre_parse.py:188(__next)
     8576    0.011    0.000    0.038    0.000 sre_parse.py:207(get)
     7917    0.005    0.000    0.005    0.000 sre_parse.py:136(__getitem__)
     6920    0.006    0.000    0.006    0.000 {method 'setdefault' of 'dict' 
objects}
     6731    0.008    0.000    0.011    0.000 main.py:165(<lambda>)
     6731    0.003    0.000    0.003    0.000 {cmp}
     6390    0.305    0.000    0.315    0.000 {eval}
     6006    0.005    0.000    0.005    0.000 {repr}
     5742    0.005    0.000    0.012    0.000 sre_parse.py:201(match)
     5394    0.003    0.000    0.003    0.000 {method 'pop' of 'dict' objects}
     5302    0.007    0.000    0.007    0.000 declarative.py:103(__get__)
     4961    0.004    0.000    0.004    0.000 main.py:1669(getterName)
     4932    0.004    0.000    0.004    0.000 main.py:1665(setterName)
     4519    0.004    0.000    0.004    0.000 {method 'items' of 'dict' objects}
     3903    0.033    0.000    0.099    0.000 dispatcher.py:278(getAllReceivers)
     3903    0.006    0.000    0.106    0.000 dispatcher.py:259(liveReceivers)
     3744    0.026    0.000    0.040    0.000 posixpath.py:56(join)
     3657    0.022    0.000    0.040    0.000 {built-in method sub}
     3624    0.002    0.000    0.002    0.000 stat.py:29(S_IFMT)
     3609    0.015    0.000    0.015    0.000 {posix.lstat}
------------------------------------------------------------------------------
_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss

Reply via email to