I'm using virtual lazy fields, but I have a similar problem. It looks
like it's only happening with one specific virtual field though.
Here's my (abbreviated) comparison - dir is apparently slowing things
down.
With the virtual field of death enabled:
581186 function calls (580899 primitive calls) in 1.377 CPU
seconds
Without:
71321 function calls (71034 primitive calls) in 0.393 CPU
seconds
With virtual fields:
581186 function calls (580899 primitive calls) in 1.377 CPU
seconds
Ordered by: internal time
List reduced from 488 to 80 due to restriction <80>
ncalls tottime percall cumtime percall
filename:lineno(function)
39612 0.453 0.000 0.453 0.000 {dir}
606 0.317 0.001 1.108 0.002 dal.py:
4585(setvirtualfields)
8 0.128 0.016 0.128 0.016 {method 'execute' of
'psycopg2._psycopg.cursor' objects}
39205 0.065 0.000 0.089 0.000 db.py:279(readersCount)
80267 0.064 0.000 0.095 0.000 {hasattr}
41173 0.056 0.000 0.062 0.000 dal.py:3158(__getitem__)
79661 0.055 0.000 0.055 0.000 dal.py:3173(__getattr__)
41860 0.049 0.000 0.049 0.000 dal.py:3170(__setitem__)
19 0.026 0.001 0.026 0.001 {compile}
39631 0.020 0.000 0.020 0.000 {method 'update' of
'dict' objects}
79313 0.018 0.000 0.018 0.000 {getattr}
5 0.017 0.003 1.149 0.230 dal.py:1149(parse)
1 0.011 0.011 0.011 0.011 {method 'commit' of
'psycopg2._psycopg.connection' objects}
39665 0.011 0.000 0.011 0.000 {method 'items' of
'dict' objects}
42404 0.006 0.000 0.006 0.000 {method 'get' of 'dict'
objects}
1313 0.005 0.000 0.006 0.000 dal.py:4027(__eq__)
60 0.004 0.000 0.004 0.000 {open}
2415 0.004 0.000 0.005 0.000 dal.py:3755(__getitem__)
71 0.004 0.000 0.004 0.000 {nt.stat}
1785 0.003 0.000 0.003 0.000 {built-in method match}
11680 0.003 0.000 0.003 0.000 {isinstance}
26 0.002 0.000 0.005 0.000 dal.py:3621(__init__)
2440 0.002 0.000 0.002 0.000 dal.py:3444(__getitem__)
58 0.002 0.000 0.002 0.000 {method 'close' of
'file' objects}
156 0.002 0.000 0.004 0.000 dal.py:4163(__init__)
26 0.002 0.000 0.012 0.000 dal.py:400(create_table)
2415 0.001 0.000 0.001 0.000 dal.py:3450(__getattr__)
27 0.001 0.000 0.001 0.000 {cPickle.load}
254/1 0.001 0.000 0.002 0.002 html.py:701(_traverse)
2/1 0.001 0.001 0.003 0.003 template.py:525(parse)
3 0.001 0.000 0.001 0.000 {reload}
1352 0.001 0.000 0.001 0.000 dal.py:4473(__init__)
1943 0.001 0.000 0.001 0.000 dal.py:3823(__getattr__)
605 0.001 0.000 0.002 0.000 dal.py:4626(__getitem__)
261 0.001 0.000 0.003 0.000 html.py:539(__init__)
4951 0.001 0.000 0.001 0.000 {method 'startswith' of
'str' objects}
1 0.001 0.001 0.079 0.079 db.py:19(<module>)
1319 0.001 0.000 0.001 0.000 dal.py:4417(__init__)
39 0.001 0.000 0.003 0.000 rewrite.py:
182(filter_out)
26 0.001 0.000 0.020 0.001 dal.py:
3384(define_table)
1110 0.001 0.000 0.001 0.000 {method 'split' of 'str'
objects}
39 0.001 0.000 0.004 0.000 html.py:127(URL)
1 0.001 0.001 0.001 0.001 {method 'open' of
'DBEnv' objects}
664 0.001 0.000 0.001 0.000 storage.py:
58(__getattr__)
26 0.001 0.000 0.001 0.000 dal.py:
3708(_create_references)
111 0.001 0.000 0.001 0.000 dal.py:
3048(sqlhtml_validators)
456 0.001 0.000 0.001 0.000 {_codecs.utf_8_decode}
456 0.001 0.000 0.001 0.000 {method 'decode' of
'str' objects}
504 0.001 0.000 0.001 0.000 html.py:599(__getitem__)
1 0.001 0.001 1.138 1.138 validators.py:
403(build_set)
1 0.001 0.001 0.001 0.001 {method 'open' of 'DB'
objects}
606 0.001 0.000 0.003 0.000 dal.py:4633(__iter__)
220 0.001 0.000 0.001 0.000 {built-in method
findall}
123 0.001 0.000 0.001 0.000 {built-in method sub}
471 0.001 0.000 0.001 0.000 html.py:617(__setitem__)
1 0.000 0.000 0.001 0.001 template.py:
328(reindent)
1 0.000 0.000 0.001 0.001 html.py:
1612(_postprocessing)
46 0.000 0.000 0.001 0.000 dal.py:1069(represent)
608 0.000 0.000 0.000 0.000 dal.py:3176(__setattr__)
182 0.000 0.000 0.001 0.000 dal.py:3788(__setitem__)
284 0.000 0.000 0.001 0.000 re.py:229(_compile)
431 0.000 0.000 0.000 0.000 {method 'encode' of
'unicode' objects}
312 0.000 0.000 0.000 0.000 dal.py:3826(__setattr__)
2 0.000 0.000 0.000 0.000 __init__.py:290(close)
1 0.000 0.000 0.000 0.000 email.py:3(<module>)
263 0.000 0.000 0.001 0.000 {method 'join' of 'str'
objects}
956 0.000 0.000 0.001 0.000 html.py:738(_setnode)
54 0.000 0.000 0.000 0.000 {win32file.LockFileEx}
2293 0.000 0.000 0.000 0.000 {method 'append' of
'list' objects}
37 0.000 0.000 0.001 0.000 validators.py:
352(__init__)
104 0.000 0.000 0.001 0.000 ntpath.py:63(join)
456 0.000 0.000 0.001 0.000 utf_8.py:15(decode)
199 0.000 0.000 0.000 0.000 db.py:270(titleAsSlug)
1 0.000 0.000 1.141 1.141 sqlhtml.py:643(__init__)
1 0.000 0.000 0.003 0.003 tools.py:791(__init__)
1 0.000 0.000 1.377 1.377 main.py:294(wsgibase)
2597 0.000 0.000 0.000 0.000 {method 'isdigit' of
'str' objects}
218 0.000 0.000 0.000 0.000 {method 'replace' of
'str' objects}
104 0.000 0.000 0.000 0.000 db.py:
367(pages_linked_get)
104 0.000 0.000 0.000 0.000 db.py:
356(previous_pages)
Without:
71321 function calls (71034 primitive calls) in 0.393 CPU
seconds
Ordered by: internal time
List reduced from 486 to 80 due to restriction <80>
ncalls tottime percall cumtime percall
filename:lineno(function)
8 0.199 0.025 0.199 0.025 {method 'execute' of
'psycopg2._psycopg.cursor' objects}
19 0.034 0.002 0.034 0.002 {compile}
5 0.025 0.005 0.072 0.014 dal.py:1149(parse)
1 0.012 0.012 0.012 0.012 {method 'commit' of
'psycopg2._psycopg.connection' objects}
2415 0.006 0.000 0.008 0.000 dal.py:3755(__getitem__)
407 0.006 0.000 0.006 0.000 {dir}
60 0.006 0.000 0.006 0.000 {open}
2655 0.006 0.000 0.006 0.000 dal.py:3170(__setitem__)
1313 0.005 0.000 0.008 0.000 dal.py:4027(__eq__)
71 0.005 0.000 0.005 0.000 {nt.stat}
407 0.004 0.000 0.014 0.000 dal.py:
4585(setvirtualfields)
11680 0.004 0.000 0.004 0.000 {isinstance}
26 0.003 0.000 0.017 0.001 dal.py:400(create_table)
1968 0.003 0.000 0.004 0.000 dal.py:3158(__getitem__)
2440 0.003 0.000 0.003 0.000 dal.py:3444(__getitem__)
26 0.003 0.000 0.008 0.000 dal.py:3621(__init__)
1785 0.003 0.000 0.003 0.000 {built-in method match}
58 0.002 0.000 0.002 0.000 {method 'close' of
'file' objects}
2414 0.002 0.000 0.002 0.000 dal.py:3450(__getattr__)
156 0.002 0.000 0.004 0.000 dal.py:4163(__init__)
1858 0.002 0.000 0.002 0.000 {hasattr}
27 0.002 0.000 0.002 0.000 {cPickle.load}
1352 0.002 0.000 0.002 0.000 dal.py:4473(__init__)
254/1 0.002 0.000 0.003 0.003 html.py:701(_traverse)
1942 0.002 0.000 0.002 0.000 dal.py:3823(__getattr__)
1319 0.001 0.000 0.001 0.000 dal.py:4417(__init__)
261 0.001 0.000 0.004 0.000 html.py:539(__init__)
605 0.001 0.000 0.003 0.000 dal.py:4626(__getitem__)
3 0.001 0.000 0.002 0.001 {reload}
4951 0.001 0.000 0.001 0.000 {method 'startswith' of
'str' objects}
2/1 0.001 0.001 0.003 0.003 template.py:525(parse)
26 0.001 0.000 0.028 0.001 dal.py:
3384(define_table)
1 0.001 0.001 0.094 0.094 db.py:19(<module>)
1252 0.001 0.000 0.001 0.000 dal.py:3173(__getattr__)
1 0.001 0.001 0.001 0.001 {method 'open' of
'DBEnv' objects}
1 0.001 0.001 0.066 0.066 validators.py:
403(build_set)
111 0.001 0.000 0.002 0.000 dal.py:
3048(sqlhtml_validators)
26 0.001 0.000 0.002 0.000 dal.py:
3708(_create_references)
1 0.001 0.001 0.001 0.001 db_dal.py:4(<module>)
1110 0.001 0.000 0.001 0.000 {method 'split' of 'str'
objects}
39 0.001 0.000 0.003 0.000 rewrite.py:
182(filter_out)
182 0.001 0.000 0.001 0.000 dal.py:3788(__setitem__)
608 0.001 0.000 0.001 0.000 dal.py:3176(__setattr__)
1 0.001 0.001 0.001 0.001 {method 'open' of 'DB'
objects}
39 0.001 0.000 0.004 0.000 html.py:127(URL)
3199 0.001 0.000 0.001 0.000 {method 'get' of 'dict'
objects}
504 0.001 0.000 0.001 0.000 html.py:599(__getitem__)
664 0.001 0.000 0.001 0.000 storage.py:
58(__getattr__)
220 0.001 0.000 0.001 0.000 {built-in method
findall}
1 0.001 0.001 0.001 0.001 html.py:
1612(_postprocessing)
471 0.001 0.000 0.001 0.000 html.py:617(__setitem__)
606 0.001 0.000 0.003 0.000 dal.py:4633(__iter__)
312 0.001 0.000 0.001 0.000 dal.py:3826(__setattr__)
456 0.001 0.000 0.001 0.000 {_codecs.utf_8_decode}
956 0.000 0.000 0.001 0.000 html.py:738(_setnode)
456 0.000 0.000 0.001 0.000 {method 'decode' of
'str' objects}
263 0.000 0.000 0.001 0.000 {method 'join' of 'str'
objects}
123 0.000 0.000 0.001 0.000 {built-in method sub}
54 0.000 0.000 0.000 0.000 {win32file.LockFileEx}
2292 0.000 0.000 0.000 0.000 {method 'append' of
'list' objects}
2597 0.000 0.000 0.000 0.000 {method 'isdigit' of
'str' objects}
199 0.000 0.000 0.001 0.000 db.py:270(titleAsSlug)
182 0.000 0.000 0.001 0.000 dal.py:2939(cleanup)
1 0.000 0.000 0.071 0.071 sqlhtml.py:643(__init__)
284 0.000 0.000 0.001 0.000 re.py:229(_compile)
46 0.000 0.000 0.001 0.000 dal.py:1069(represent)
218 0.000 0.000 0.000 0.000 {method 'replace' of
'str' objects}
2 0.000 0.000 0.000 0.000 __init__.py:290(close)
1 0.000 0.000 0.001 0.001 template.py:
328(reindent)
37 0.000 0.000 0.001 0.000 validators.py:
352(__init__)
431 0.000 0.000 0.000 0.000 {method 'encode' of
'unicode' objects}
104 0.000 0.000 0.001 0.000 ntpath.py:63(join)
54 0.000 0.000 0.000 0.000 {win32file.UnlockFileEx}
1 0.000 0.000 0.393 0.393 main.py:294(wsgibase)
52 0.000 0.000 0.006 0.000 dal.py:352(file_open)
1 0.000 0.000 0.000 0.000 page.py:16(<module>)
904 0.000 0.000 0.000 0.000 {getattr}
1 0.000 0.000 0.000 0.000 {method 'truncate' of
'file' objects}
1 0.000 0.000 0.069 0.069 sqlhtml.py:189(widget)
456 0.000 0.000 0.001 0.000 utf_8.py:15(decode)
===================
The virtual field that causes the slowdown is here. It's certainly
slow SQL, but it doesn't execute at all in this context, and there are
plenty of other virtual fields:
class table_story_readersCount(object):
"""
Check: select count(distinct user_id) from user_story_playthrough
where story_id = 1
"""
def readersCount(self):
def lazy(story_id = self.story.id):
#print 'calculating readersCount'
return len(db(db.user_story_playthrough.story_id ==
story_id). \
select(db.user_story_playthrough.user_id,
distinct=True))
return lazy
db.story.virtualfields.append(table_story_readersCount())
Any idea what could be happening? Seems like all the {dir} calls might
be hinting at something.
On Feb 16, 11:13 am, Carlos <[email protected]> wrote:
> You can also use lazy virtual fields, such that they execute only when
> needed/called:
>
> http://www.web2py.com/book/default/chapter/06?search=lazy
>
> Carlos
