Hi again.
Here are another testing results.
First of all, let me remind you about is_integer bug. It doesn't allow
negative numbers. Correct code is:
is_integer=re.compile('-?[\d]+).match
Now. I've run more timings, more profiling and more code tweaks.
This time it was Core2 Duo @2.20GHz with 1G RAM. Ubuntu 8.04.
Testing strategy: 20 sets of 1000 GET requests to the main page. Same
application as before. For each set I have average GET request time.
For a group of 20 sets I get minimum average, global average and
maximum average. Also I have average model init time for 20k requests
(no splitting into each set).
1) Upstream sql.py
model 0.01646
min 0.02979
avg 0.03267
max 0.03597
2) Optimised SQLTable.__init__ and SQLField.__init__ (patches
attached, backwards compartible, though you may want to remove my
testing marks)
model 0.01380
min 0.02697
avg 0.03003
max 0.03189
So - about 8% speedup (by avg time) here.
3) Now lazy tables (patch attached).
Since I expect that you will not find any objections to my patches for
(2) here are cumulative results, i.e. optimised __init__s + lazy
tables
...
hmmm. It doesn't improves results significantly anymore. It looks like
these patches are interchangeable. I've run my test three times,
results are:
model 0.01009
min 0.02555
avg 0.0297105
max 0.033360
model 0.01012
min 0.02369
avg 0.02983
max 0.03340
model 0.00966
min 0.02415
avg 0.02850
max 0.03208
So it makes some improvement about 1%-5% but somehow it also makes
testing results more disperse. Do not know if it worths including.
Massimo, your opinion?
4) Also I've tried to further speedup validator generation w/o losing
backwards compartibility, but improvement was about 1% and below the
margin of error (even less noticeable than with lazy tables).
On Sun, Jun 7, 2009 at 11:45 PM, Alexey Nezhdanov<[email protected]> wrote:
> On Sunday 07 June 2009 18:41:30 mdipierro wrote:
>> I implemented a backward compatible speedup of the default validators.
>> I also realized that those many calls to is_integer could be avoided.
>> I think I fixed it.
> 0.0429/0.0844
> Somehow consistency is not kept across reboots :(
> ... about a hour later: and actually now I get about 10% noise so it become
> very hard to make any good measurements. I'll try tomorrow on better
> hardware. In the meanwhile, I'll try to profile it once again.
> ...
> ok, there are still two perfomance hogs left: SQLTable.__init__ (about 11%)
> and SQLField.__init__ (about 6,5%). These percentages are 'self' times - i.e.
> it was spent directly in lines 1099-1149, 1693-1752.
> Python profiling doesn't allow to attribute CPU consumption to individual
> lines so for now I don't have any more precise pointers. But that could be
> split into functional blocks and narrowed further down.
>
> In any case - we collected almost all low-hanging fruits. Any further
> optimisation will give gains of range 2-5%, not times anymore.
>
> Except lazy tables which still may give tens of percents. BTW - Massimo, you
> said something about SQLLazyTable class. You should implement SQLLazyField
> too then. Or may be just wait a day or two until I do deeper profiling.
>
>> Please check out the latest trunk.
>>
>> Massimo
>>
>
> P.S. Here is small fix to is_integer. We forgot about negative numbers:
> Also this version is tiny bit faster
> is_integer=re.compile('-?[\d]+).match
>
> --
> Sincerely yours
> Alexey Nezhdanov
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
--- /home/snake/Desktop/sql.py 2009-06-08 09:36:27.225210464 +0400
+++ sql.py 2009-06-08 11:32:18.285211851 +0400
@@ -1112,39 +1112,46 @@
@raise SyntaxError: when a supplied field is of incorrect type.
"""
- new_fields = []
- for field in fields:
- if isinstance(field, SQLField):
- new_fields.append(field)
- elif isinstance(field, SQLTable):
- new_fields += [copy.copy(field[f]) for f in
- field.fields if f != 'id']
- else:
- raise SyntaxError, \
- 'define_table argument is not a SQLField'
- fields = new_fields
- self._db = db
- self._tablename = tablename
- self.fields = SQLCallableList()
- self._referenced_by = []
- fields = list(fields)
- fields.insert(0, SQLField('id', 'id'))
- for field in fields:
- self.fields.append(field.name)
- self[field.name] = field
- field._tablename = self._tablename
- field._table = self
- field._db = self._db
- if isinstance(field.type,str) and field.type[:9] == 'reference':
- referenced = field.type[10:].strip()
- if not referenced:
- raise SyntaxError, 'SQLTable: reference to nothing!'
- if not referenced in self._db:
- raise SyntaxError, 'SQLTable: table does not exist'
- referee = self._db[referenced]
- if self._tablename in referee.fields:
- raise SyntaxError, 'SQLField: table %s has same name as a field in referenced table %s' % (self._tablename, referee._tablename)
- referee._referenced_by.append((self._tablename, field.name))
+ new_fields = [ SQLField('id', 'id') ] # 1
+ for field in fields: # 1
+ if isinstance(field, SQLField): # 1 0.11
+ new_fields.append(field) # 1
+ elif isinstance(field, SQLTable): # 1
+ new_fields += [copy.copy(field[f]) for f in # 1
+ field.fields if f != 'id'] # 1
+ else: # 1
+ raise SyntaxError, 'define_table argument is not a SQLField' # 1
+# fields = new_fields # 0
+ self._db = db # 0
+ self._tablename = tablename # 0 0.11
+ fields = self.fields = SQLCallableList() # 0
+ self._referenced_by = [] # 0
+# fields = list(fields) # 2
+# fields.insert(0, SQLField('id', 'id')) # 2 0.23
+ for field in new_fields: # 2
+ field_name=field.name
+ fields.append(field_name) # 3 0.58
+# self[field_name] = field # 3
+ dict.__setitem__(self, field_name, field)
+ field._tablename = tablename # 3
+ field._table = self # 3a 0.43
+ field._db = db # 3a
+ try:
+ if field.type[:9] != 'reference':
+ continue
+ except:
+# if isinstance(field.type, str) and field.type[:9] == 'reference': # 3a
+ referenced = field.type[10:].strip() # 4
+ try:
+ referee = db[referenced] # 4
+ except:
+ if not referenced in db: # 4
+ if not referenced: # 4
+ raise SyntaxError, 'SQLTable: reference to nothing!' # 4 0.19
+ raise SyntaxError, 'SQLTable: table does not exist' # 4
+ if tablename in referee.fields: # 4
+ raise SyntaxError, 'SQLField: table %s has same name as a field in referenced table %s' % (tablename, referee._tablename)
+ referee._referenced_by.append((tablename, field_name)) # 4
self.ALL = SQLALL(self)
--- /home/snake/Desktop/sql.py 2009-06-08 09:36:27.225210464 +0400
+++ sql.py 2009-06-08 11:27:23.206210851 +0400
@@ -1718,13 +1725,14 @@
raise SyntaxError, 'SQLField: invalid field name'
if isinstance(type, SQLTable):
type = 'reference ' + type._tablename
- if not length and type == 'string':
- type = 'text'
- elif not length and type == 'password':
- length = 32
- self.type = type # 'string', 'integer'
- if type == 'upload':
+ elif type == 'upload':
length = 64
+ elif not length:
+ if type == 'string':
+ type = 'text'
+ elif type == 'password':
+ length = 32
+ self.type = type # 'string', 'integer'
self.length = length # the length of the string
self.default = default # default value for field
self.required = required # is this field required
@@ -1746,10 +1754,11 @@
self.label = ' '.join([x.capitalize() for x in
fieldname.split('_')])
if requires == sqlhtml_validators:
- requires = sqlhtml_validators(type, length)
+ self.requires = sqlhtml_validators(type, length)
elif requires is None:
- requires = []
- self.requires = requires # list of validators
+ self.requires = []
+ else:
+ self.requires = requires # list of validators
def formatter(self, value):
if value is None or not self.requires:
--- /home/snake/Desktop/sql.py 2009-06-08 09:36:27.225210464 +0400
+++ sql.py 2009-06-08 10:26:42.605233069 +0400
@@ -513,13 +513,16 @@
"""
def __getitem__(self, key):
+ value = dict.__getitem__(self, str(key))
+ if not callable(value) or key[0]=='_' or not isinstance(value, types.FunctionType): return value
+# print 'calling',value
+ value.__call__() # That must redefine table in-place
return dict.__getitem__(self, str(key))
+ __getattr__ = __getitem__
def __setitem__(self, key, value):
dict.__setitem__(self, str(key), value)
- def __getattr__(self, key):
- return dict.__getitem__(self,key)
def __setattr__(self, key, value):
if key in self:
@@ -942,14 +945,14 @@
):
for key in args:
- if key != 'migrate':
+ if key not in ('migrate', 'redefine'):
raise SyntaxError, 'invalid table attribute: %s' % key
migrate = args.get('migrate',True)
tablename = cleanup(tablename)
- if hasattr(self,tablename) or tablename[0] == '_':
- raise SyntaxError, 'invalid table name: %s' % tablename
- if tablename in self.tables:
+ if tablename in self.tables and not args.get('redefine', False):
raise SyntaxError, 'table already defined: %s' % tablename
+ elif tablename[0] == '_' or getattr(SQLDB, tablename, None):
+ raise SyntaxError, 'invalid table name: %s' % tablename
t = self[tablename] = SQLTable(self, tablename, *fields)
# db magic
