Oleg Broytmann wrote:
On Wed, Jul 22, 2009 at 02:26:57PM -0400, Stef Telford wrote:
yes. evals appear to be a 'bad' thing here :\
Well, those evals are in sqlmeta.addColumn and .addJoin methods, so they
work once for every column in the table, but that's all. After the class
has been created and populated - whatever you do with rows (class instances)
- those evals are not executed.
Ah. hrm. *rubs chin* perhaps it's not the evals then. It seems that the
instantiations get .. well .. 'slower' over time.
Curiouser and curiouser. IWBN to find where the slowness is *near the
end* of the loop - i.e. when instantiation becomes really slow.
It could be purely a 'feeling' .. I don't have any numbers to back it up
and I am not entirely sure -how- to benchmark that. This machine does
have 8gb of ram in it, and a fairly beefy quad core. I have never seen
any process get near memory exhaustion, which I could believe the calls
to 'malloc' could slow down but.. yes.
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
40000 0.063 0.000 0.141 0.000 <string>:1(<lambda>)
Can you sort by ncalls and cumtime columns, cut the first 100-200 lines
and post two tables here?
surely, please find the output attached at the bottom. It maybe also
worth changing the Decimal call to using gmpy or .. something that's
faster than Decimal perhaps ? I did notice a fair few cycles are spent
in Decimal.__new__ .. I don't think that's the -entire- story, but it
should help I would think.
1 0.000 0.000 85.863 85.863 <string>:1(<module>)
This is AFAIU the entire script? :)
Yes indeed.. that's the way I read the cProfile output as well :D
Regards
Stef
217419
37287462 function calls (37280538 primitive calls) in 85.556 CPU
seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 85.567 85.567 <string>:1(<module>)
1 0.000 0.000 85.567 85.567 {execfile}
1 0.221 0.221 85.567 85.567 t.t:3(<module>)
1 0.127 0.127 83.346 83.346 sresults.py:175(__iter__)
40001 0.308 0.000 79.876 0.002 dbconnection.py:649(next)
40000 0.196 0.000 40.291 0.001 util.py:123(get)
40000 0.375 0.000 39.282 0.001 main.py:872(get)
40002 10.018 0.000 39.252 0.001 {method 'fetchone' of
'psycopg2._psycopg.cursor' objects}
40000 0.168 0.000 32.166 0.001 booking.py:336(_init)
1840006 16.887 0.000 29.234 0.000 decimal.py:515(__new__)
40000 0.132 0.000 23.477 0.001 util.py:224(_init)
40000 0.214 0.000 23.323 0.001 main.py:912(_init)
40000 10.475 0.000 23.069 0.001 main.py:1140(_SO_selectInit)
40000 0.847 0.000 8.521 0.000 util.py:236(notify)
40000 0.390 0.000 6.736 0.000 inspect.py:820(getframeinfo)
40000 6.525 0.000 6.647 0.000 main.py:1171(__init__)
1880365 4.854 0.000 4.854 0.000 {built-in method match}
80417 1.025 0.000 4.195 0.000 inspect.py:381(getsourcefile)
40000 0.533 0.000 3.903 0.000 inspect.py:454(findsource)
7363357 3.786 0.000 3.786 0.000 {built-in method group}
2 0.000 0.000 3.575 1.788 dbconnection.py:336(_executeRetry)
2 3.575 1.788 3.575 1.788 {method 'execute' of
'psycopg2._psycopg.cursor' objects}
1 0.000 0.000 3.343 3.343 sresults.py:181(lazyIter)
1 0.000 0.000 3.343 3.343 dbconnection.py:403(iterSelect)
1 0.000 0.000 3.312 3.312 dbconnection.py:635(__init__)
3381876 3.132 0.000 3.132 0.000 {setattr}
5035316 2.900 0.000 2.900 0.000 {isinstance}
1200000 1.856 0.000 2.769 0.000 col.py:1289(to_python)
3362951 2.729 0.000 2.729 0.000 main.py:1673(instanceName)
80448 0.491 0.000 1.658 0.000 inspect.py:342(getfile)
640000 0.999 0.000 1.471 0.000 col.py:657(to_python)
1 0.003 0.003 1.465 1.465 booking.py:4(<module>)
2 0.003 0.001 1.318 0.659 invoice.py:4(<module>)
1 0.000 0.000 1.309 1.309 transaction.py:5(<module>)
1 0.001 0.001 1.308 1.308 package.py:4(<module>)
1 0.001 0.001 1.296 1.296 sector.py:4(<module>)
640000 0.983 0.000 1.280 0.000 col.py:573(to_python)
1 0.001 0.001 1.250 1.250 employee.py:4(<module>)
1840001 1.238 0.000 1.238 0.000 {method 'lstrip' of 'str' objects}
551/279 0.004 0.000 1.207 0.004 declarative.py:81(__new__)
273 0.016 0.000 1.174 0.004 main.py:728(__classinit__)
1840630 1.161 0.000 1.161 0.000 {built-in method __new__ of type
object at 0x814fa00}
1 0.001 0.001 1.152 1.152 communication.py:4(<module>)
1 0.001 0.001 1.128 1.128 agent.py:4(<module>)
1 0.001 0.001 1.012 1.012 brochure.py:4(<module>)
1 0.001 0.001 1.000 1.000 contact.py:4(<module>)
1 0.001 0.001 0.943 0.943 feedback.py:1(<module>)
1 0.001 0.001 0.910 0.910 trip.py:5(<module>)
2801 0.102 0.000 0.858 0.000 main.py:307(addColumn)
40002 0.054 0.000 0.790 0.000 sqlcursor.py:72(useMemcache)
80432 0.130 0.000 0.737 0.000 posixpath.py:168(exists)
40003 0.277 0.000 0.736 0.000 sqlcursor.py:31(hostSetting)
440000 0.564 0.000 0.672 0.000 col.py:1031(to_python)
1532367/1531398 0.647 0.000 0.648 0.000 {len}
1 0.001 0.001 0.617 0.617 commissions.py:1(<module>)
80563 0.608 0.000 0.608 0.000 {posix.stat}
1 0.003 0.003 0.594 0.594 service.py:6(<module>)
321608 0.351 0.000 0.512 0.000 string.py:218(lower)
120000 0.239 0.000 0.474 0.000 inspect.py:43(isclass)
640000 0.472 0.000 0.472 0.000 decimal.py:718(__nonzero__)
40139 0.062 0.000 0.460 0.000 re.py:186(compile)
40043/40000 0.155 0.000 0.436 0.000 inspect.py:408(getmodule)
301683 0.375 0.000 0.406 0.000 {hasattr}
40624 0.099 0.000 0.401 0.000 re.py:227(_compile)
6390 0.317 0.000 0.328 0.000 {eval}
440000 0.282 0.000 0.326 0.000 col.py:511(to_python)
161288 0.210 0.000 0.316 0.000 inspect.py:35(ismodule)
1 0.000 0.000 0.295 0.295 priceband.py:1(<module>)
1 0.001 0.001 0.295 0.295 price.py:1(<module>)
160000 0.197 0.000 0.291 0.000 inspect.py:139(istraceback)
40000 0.113 0.000 0.284 0.000
booking.py:2678(observedNewBookingNeedsOnlinePwd)
2801 0.018 0.000 0.278 0.000 col.py:402(withClass)
160000 0.200 0.000 0.272 0.000 inspect.py:149(isframe)
1 0.000 0.000 0.268 0.268 sresults.py:203(count)
1 0.000 0.000 0.268 0.268 sresults.py:189(accumulate)
1 0.000 0.000 0.268 0.268
dbconnection.py:407(accumulateSelect)
1 0.000 0.000 0.266 0.266 dbconnection.py:389(queryOne)
1 0.000 0.000 0.266 0.266
dbconnection.py:257(_runWithConnection)
1 0.000 0.000 0.266 0.266 dbconnection.py:379(_queryOne)
80006 0.266 0.000 0.266 0.000 sqlcursor.py:21(hostSettings)
142 0.002 0.000 0.265 0.002 sre_compile.py:501(compile)
42652 0.247 0.000 0.247 0.000 {zip}
2801 0.062 0.000 0.238 0.000 col.py:97(__init__)
40000 0.145 0.000 0.234 0.000
booking.py:2653(observedDocRequirementsChanged)
3 0.003 0.001 0.214 0.071 __init__.py:3(<module>)
120000 0.144 0.000 0.212 0.000 inspect.py:51(ismethod)
120000 0.143 0.000 0.209 0.000 inspect.py:126(isfunction)
1 0.003 0.003 0.207 0.207 main.py:27(<module>)
120000 0.143 0.000 0.196 0.000 inspect.py:167(iscode)
280932 0.186 0.000 0.191 0.000 {getattr}
1 0.003 0.003 0.167 0.167 dbconnection.py:1(<module>)
327248 0.165 0.000 0.165 0.000 {method 'lower' of 'str' objects}
80417 0.162 0.000 0.162 0.000 {imp.get_suffixes}
80008 0.153 0.000 0.153 0.000 {posix.uname}
217419
37287474 function calls (37280550 primitive calls) in 85.977 CPU
seconds
Ordered by: call count
ncalls tottime percall cumtime percall filename:lineno(function)
7363357 3.874 0.000 3.874 0.000 {built-in method group}
5035316 2.905 0.000 2.905 0.000 {isinstance}
3381876 3.180 0.000 3.180 0.000 {setattr}
3362951 2.869 0.000 2.869 0.000 main.py:1673(instanceName)
1880365 4.896 0.000 4.896 0.000 {built-in method match}
1840630 1.196 0.000 1.196 0.000 {built-in method __new__ of type
object at 0x814fa00}
1840006 17.037 0.000 29.584 0.000 decimal.py:515(__new__)
1840001 1.268 0.000 1.268 0.000 {method 'lstrip' of 'str' objects}
1532367/1531398 0.664 0.000 0.665 0.000 {len}
1200000 1.849 0.000 2.764 0.000 col.py:1289(to_python)
640000 0.985 0.000 1.468 0.000 col.py:657(to_python)
640000 0.483 0.000 0.483 0.000 decimal.py:718(__nonzero__)
640000 0.987 0.000 1.288 0.000 col.py:573(to_python)
440000 0.566 0.000 0.675 0.000 col.py:1031(to_python)
440000 0.286 0.000 0.329 0.000 col.py:511(to_python)
327254 0.169 0.000 0.169 0.000 {method 'lower' of 'str' objects}
321608 0.351 0.000 0.517 0.000 string.py:218(lower)
301683 0.376 0.000 0.407 0.000 {hasattr}
280932 0.185 0.000 0.190 0.000 {getattr}
280000 0.137 0.000 0.137 0.000 sqlcursor.py:93(debugObserver)
161288 0.209 0.000 0.317 0.000 inspect.py:35(ismodule)
160000 0.198 0.000 0.271 0.000 inspect.py:149(isframe)
160000 0.196 0.000 0.291 0.000 inspect.py:139(istraceback)
135452 0.087 0.000 0.087 0.000 {method 'get' of 'dict' objects}
133837 0.065 0.000 0.065 0.000 {method 'has_key' of 'dict'
objects}
120000 0.144 0.000 0.214 0.000 inspect.py:51(ismethod)
120000 0.236 0.000 0.470 0.000 inspect.py:43(isclass)
120000 0.142 0.000 0.209 0.000 inspect.py:126(isfunction)
120000 0.143 0.000 0.195 0.000 inspect.py:167(iscode)
84976 0.045 0.000 0.045 0.000 {method 'startswith' of 'str'
objects}
80614 0.061 0.000 0.061 0.000 {max}
80563 0.602 0.000 0.602 0.000 {posix.stat}
80448 0.494 0.000 1.658 0.000 inspect.py:342(getfile)
80432 0.127 0.000 0.728 0.000 posixpath.py:168(exists)
80417 0.161 0.000 0.161 0.000 {imp.get_suffixes}
80417 1.012 0.000 4.180 0.000 inspect.py:381(getsourcefile)
80008 0.155 0.000 0.155 0.000 {posix.uname}
80006 0.263 0.000 0.263 0.000 sqlcursor.py:21(hostSettings)
80005 0.045 0.000 0.045 0.000 sqlcursor.py:97(globalQueryCache)
80001 0.051 0.000 0.051 0.000 {_weakref.proxy}
44610 0.029 0.000 0.029 0.000 {min}
43720 0.031 0.000 0.031 0.000 {method 'keys' of 'dict' objects}
42652 0.243 0.000 0.243 0.000 {zip}
40624 0.098 0.000 0.384 0.000 re.py:227(_compile)
40139 0.065 0.000 0.447 0.000 re.py:186(compile)
40043/40000 0.157 0.000 0.438 0.000 inspect.py:408(getmodule)
40003 0.035 0.000 0.035 0.000 {sys._getframe}
40003 0.280 0.000 0.739 0.000 sqlcursor.py:31(hostSetting)
40002 9.936 0.000 39.520 0.001 {method 'fetchone' of
'psycopg2._psycopg.cursor' objects}
40002 0.054 0.000 0.793 0.000 sqlcursor.py:72(useMemcache)
40001 0.319 0.000 80.347 0.002 dbconnection.py:649(next)
40000 0.113 0.000 0.279 0.000
booking.py:2678(observedNewBookingNeedsOnlinePwd)
40000 0.853 0.000 8.519 0.000 util.py:236(notify)
40000 0.146 0.000 0.236 0.000
booking.py:2653(observedDocRequirementsChanged)
40000 0.036 0.000 0.037 0.000 linecache.py:33(getlines)
40000 0.391 0.000 6.730 0.000 inspect.py:820(getframeinfo)
40000 0.538 0.000 3.911 0.000 inspect.py:454(findsource)
40000 0.076 0.000 0.106 0.000 main.py:251(__init__)
40000 0.072 0.000 0.094 0.000 main.py:1679(__init__)
40000 10.476 0.000 23.260 0.001 main.py:1140(_SO_selectInit)
40000 0.168 0.000 32.353 0.001 booking.py:336(_init)
40000 0.212 0.000 23.512 0.001 main.py:912(_init)
40000 6.530 0.000 6.654 0.000 main.py:1171(__init__)
40000 0.022 0.000 0.022 0.000 sqlcursor.py:89(useObservers)
40000 0.052 0.000 0.079 0.000 main.py:939(_SO_loadValue)
40000 0.192 0.000 40.484 0.001 util.py:123(get)
40000 0.063 0.000 0.142 0.000 <string>:1(<lambda>)
40000 0.131 0.000 23.666 0.001 util.py:224(_init)
40000 0.040 0.000 0.040 0.000 dbconnection.py:831(__set__)
40000 0.376 0.000 39.477 0.001 main.py:872(get)
39578 0.017 0.000 0.017 0.000 {method 'append' of 'list'
objects}
18094 0.011 0.000 0.011 0.000 {method 'endswith' of 'str'
objects}
13411 0.006 0.000 0.006 0.000 {id}
13408 0.060 0.000 0.066 0.000 dispatcher.py:236(getReceivers)
10900 0.026 0.000 0.035 0.000 sre_parse.py:188(__next)
8576 0.011 0.000 0.038 0.000 sre_parse.py:207(get)
7917 0.005 0.000 0.005 0.000 sre_parse.py:136(__getitem__)
6920 0.006 0.000 0.006 0.000 {method 'setdefault' of 'dict'
objects}
6731 0.008 0.000 0.011 0.000 main.py:165(<lambda>)
6731 0.003 0.000 0.003 0.000 {cmp}
6390 0.305 0.000 0.315 0.000 {eval}
6006 0.005 0.000 0.005 0.000 {repr}
5742 0.005 0.000 0.012 0.000 sre_parse.py:201(match)
5394 0.003 0.000 0.003 0.000 {method 'pop' of 'dict' objects}
5302 0.007 0.000 0.007 0.000 declarative.py:103(__get__)
4961 0.004 0.000 0.004 0.000 main.py:1669(getterName)
4932 0.004 0.000 0.004 0.000 main.py:1665(setterName)
4519 0.004 0.000 0.004 0.000 {method 'items' of 'dict' objects}
3903 0.033 0.000 0.099 0.000 dispatcher.py:278(getAllReceivers)
3903 0.006 0.000 0.106 0.000 dispatcher.py:259(liveReceivers)
3744 0.026 0.000 0.040 0.000 posixpath.py:56(join)
3657 0.022 0.000 0.040 0.000 {built-in method sub}
3624 0.002 0.000 0.002 0.000 stat.py:29(S_IFMT)
3609 0.015 0.000 0.015 0.000 {posix.lstat}
------------------------------------------------------------------------------
_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
