On Sunday 07 June 2009 11:13:49 Alexey Nezhdanov wrote:
> 1) When initialising, SQLField uses dir() call to find any clashes
> with reserved names.
I was a bit incorrect here. It checks for already defined fields too.
Anyways - I optimised it by commenting it out :-P
In practice I'd recommend similar approach:
1) don't check for clashes in the production environment (SQLDB init
parameter?)
2) (optional) maintain cache of names for each table. Don't use dir().
> 2) is_integer is a fast call, but with 1.1k (!) calls ...
Replaced it with my own version:
integer_pat=re.compile('[0-9]+$')
is_integer=lambda x: integer_pat.match(x)
it's about 2.3 times faster. C version would be even better.
> 3) sqlhtml_validators is an obvious bug... 8/9 of work is wasted.
> ... I suspect that dictionary building may be moved into module namespace
did that with some tweaks (same validators get reused). Dangerous approach
may introduce hidden dataflow, but that could be prevented by somehow making
these validators read-only.
Got another 9.5% model time speedup. (0.096s vs 0.106s). Though this figures
get more and more meaningless as model init time reduces it's share in total
page generation time. Probably I'll just switch to time measuring of
urllib.urlopen().
...OK, switched. Here are stats. Each line is in the format
model_init_time/urlopen_measured_time - average over 100 calls
These timings are not comparable with my previous emails because I did some
additional arrangements that should make these times more than usual, but
more stable across tests. Also application is compiled this time.
1) no optimisations
0.1219/0.1649
2) skip CREATE TABLE in SQLTable init
0.0865/0.1261
3) + three steps, described above in this mail.
0.0439/0.0855
4) + lazy tables init in model
This one is application specific. For instance in my application it causes
4 tables out of 16 to be initialised in model and 2 more - in controller. This
is for front page. Other pages will have different figures. Yet, here are
timings:
0.0194/0.0622
5) And finally, for mere comparison, timings of (4) again, but this time with
non-compiled source.
0.0208/0.1216
Some conclusions:
1) always compile your app for production. It saves you about 50% CPU.
2) With very little effort it was possible to speedup web2py app further 2.65
times, dropping page generation times from tenths of second to hundredths.
I hope at least some of that work will find it's way into mainline. I'm
attaching all patches that I used to this mail. These patches are against
quite old web2py - Version 1.61.4 (2009-04-21 10:02:50) (I didn't like how
newer versions looked like so I stick to initially installed version).
--
Sincerely yours
Alexey Nezhdanov
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
--- sql.py.old 2009-05-23 14:28:15.286720712 +0400
+++ sql.py 2009-06-07 12:38:16.509986136 +0400
@@ -1053,6 +1058,8 @@
return False
return True
+integer_pat=re.compile('[0-9]+$')
+is_integer=lambda x: integer_pat.match(x) #0m3.688s
class SQLTable(dict):
--- sql.py.old 2009-05-23 14:28:15.286720712 +0400
+++ sql.py 2009-06-07 12:38:56.777987959 +0400
@@ -487,6 +487,10 @@
"""
def __getitem__(self, key):
+ value = dict.__getitem__(self, str(key))
+ if not callable(value) or key[0]=='_' or not isinstance(value, types.FunctionType): return value
+ print 'calling',value
+ value.__call__() # That must redefine table in-place
return dict.__getitem__(self, str(key))
def __setitem__(self, key, value):
--- sql.py.old 2009-05-23 14:28:15.286720712 +0400
+++ sql.py 2009-06-07 11:54:40.977982790 +0400
@@ -331,12 +331,9 @@
}
-def sqlhtml_validators(field_type, length):
- v = {
+preset_validators = {
'boolean': [],
- 'string': validators.IS_LENGTH(length),
'text': validators.IS_LENGTH(2 ** 16),
- 'password': validators.IS_LENGTH(length),
'blob': [],
'upload': [],
'double': validators.IS_FLOAT_IN_RANGE(-1e100, 1e100),
@@ -346,10 +343,19 @@
'datetime': validators.IS_DATETIME(),
'reference': validators.IS_INT_IN_RANGE(0, 1e100),
}
- try:
- return v[field_type[:9]]
- except KeyError:
- return []
+preset_validators2 = {
+ 'string': {},
+ 'password': {},
+ }
+def sqlhtml_validators(field_type, length):
+ typ=field_type[:9]
+ try: return preset_validators[typ]
+ except:
+ try: return preset_validators2[typ][length]
+ except:
+ if typ not in preset_validators2: return []
+ preset_validators2[typ]=validator=validators.IS_LENGTH(length)
+ return validator
def sql_represent(obj, fieldtype, dbname):
--- sql.py.old 2009-05-23 14:28:15.286720712 +0400
+++ sql.py 2009-06-07 12:39:48.649986330 +0400
@@ -1607,8 +1608,8 @@
):
self.name = fieldname = cleanup(fieldname)
- if fieldname in dir(SQLTable) or fieldname[0] == '_':
- raise SyntaxError, 'SQLField: invalid field name'
+# if fieldname in dir(SQLTable) or fieldname[0] == '_':
+# raise SyntaxError, 'SQLField: invalid field name'
if isinstance(type, SQLTable):
type = 'reference ' + type._tablename
if not length and type == 'string':
--- sql.py.old 2009-05-23 14:28:15.286720712 +0400
+++ sql.py 2009-06-07 12:40:39.565985360 +0400
@@ -930,13 +930,14 @@
if self._uri == 'None':
args['migrate'] = False
return t
- sql_locker.acquire()
- try:
+ if args['migrate']:
+ sql_locker.acquire()
+ try:
query = t._create(migrate=args['migrate'])
- except BaseException, e:
+ except BaseException, e:
sql_locker.release()
raise e
- sql_locker.release()
+ sql_locker.release()
self.tables.append(tablename)
return t
