Forgot about patches. Here they are.
'optimised inits v.2'
'lazy tables' (this one is modified to be applicable to trunk version of
sql.py)
On Tue, Jun 9, 2009 at 3:10 PM, Alexey Nezhdanov <[email protected]> wrote:
> Results of second set of test runs:
>
> r875 45.523ms
> r882 20.614ms
> inits v.1 18.464ms
> inits v.2 18.677ms
> inits1+lazyT 15.377ms
> inits2+lazyT 15.280ms
>
> Observed noise was 0.502ms for slowest execution (90:1
> signal-to-noise) and 0.889ms for fastest execution (16:1
> signal-to-noise).
> Given than, I think it is safe enough to say that
> "this particular application on this particular hardware/OS
> combination was sped up 2.88...3.31 times".
>
> As in last case - for completeness I'm attaching full console output
>
> I think I know what I'll do. Currently I do not want to release my
> project as any kind of open-source. But I can
> 1) Strip out any code I do not use in this perfomance testing
> 2) rename all sensitive fieldnames/variables to something like field1,
> field2, var3, var4.
> Release resulting package. It would be useless as it is, but it will
> have same perfomance parameters so could be used for testing.
>
> On Tue, Jun 9, 2009 at 1:35 PM, Alexey Nezhdanov<[email protected]> wrote:
> > Wrote 'optimised inits v2' patch. Tested everything.
> >
> > Once again I changed my testing pattern a bit.
> >
> > 1) I dumped my own timing tool in favor of much more standard
> > 'ab' (apache benchmark).
> >
> > 2) I decidedly will publish all these results in two main.
> > I just finished first round of testing. Here are results (reported
> > values are average request time in milliseconds. If anyone interested
> > - I'm attaching a full output from my console).
> > I will do all that testing again and publish same results again so
> > that the amount of noise in all this will be public and clearly
> > visible (here I'm fighting with my own temptation to rerun the tests
> > if timings seem to be 'too away' to me).
> >
> > First four lines share my app's db.py. Last two require a modified
> > db.py, but modification is cosmetical - each define_table is put into
> > separate function and I do db.tablename=create_table_tablename after
> > each function. Model init time was not measured.
> >
> > No more words. Dry figures only. Each line represents result of 10k
> requests.
> > r875 46.025ms
> > r882 21.048ms
> > inits v.1 18.918ms
> > inits v.2 18.122ms
> > inits1+lazyT 16.032ms
> > inits2+lazyT 14.391ms
> >
> > P.S. I'm surprised about last line... Noise?
> >
> > On Tue, Jun 9, 2009 at 11:12 AM, Alexey Nezhdanov<[email protected]>
> wrote:
> >> new testing:
> >>
> >> ---- SERVER KERNEL ----
> >> --prints
> >> r875 0.04898
> >> r822ini 0.03070 1.60x
> >> --silent
> >> r875 0.04914
> >> r822ini 0.03049 1.61x
> >>
> >> So I get much more consistent results on this hardware.
> >> While this is obviously not the best perfomance (my weaker box,
> >> with less RAM, troubled with video output 1280x1024,
> >> software 90deg rotation - performs BETTER), that does not matter.
> >> What does - is that I can now be sure that these results are
> >> noise-free so I can safely compare timings from various patches.
> >> Proceeding with writing 'inits v2'.
> >>
> >>
> >> On Tue, Jun 9, 2009 at 10:43 AM, Alexey Nezhdanov<[email protected]>
> wrote:
> >>> Ok. Now the confusion is resolved.
> >>> 1) Speed improvements of 70% and up that I reported yesterday are
> >>> really exist. I just reproduced a 3.47 times model speedup and 2.15
> >>> overall speedup for my app (r875 vs r822+inits).
> >>> BUT this app is atypical. I have added some time measuring code there
> >>> so it prints out two lines per each model init. So when I am testing
> >>> perfomance - screen very quickly scrolls up
> >>>
> >>> 2) Simply commenting out two print statements gives me only 1.67
> >>> overall speedup given equal other conditions. I think that processor
> >>> receive additional interrupts from videocard that in turn results in
> >>> more often checks of tasks queue.
> >>>
> >>> 3) I declare all my previous testing results spoiled by noise
> >>> generated by print statement and inappropriate kernel scheduler
> >>> setting.
> >>> I've set up yet another test box with these parameters:
> >>>
> >>> Intel Core2 Duo 2.66GHz, 2G RAM, Ubuntu 9.04, 'server' flavour kernel
> >>> 2.6.28-11.42. Initially I considered to install a 'realtime' kernel,
> >>> but it appeared to be unstable on that hardware (and afterall - it's
> >>> for sound/video processing and 'server' type is more likely to be
> >>> installed on servers).
> >>>
> >>> Will report new testing results (and finally I hope to write
> >>> 'optimised inits ver.2' patch) later today.
> >>>
> >>> On Tue, Jun 9, 2009 at 5:14 AM, mdipierro<[email protected]>
> wrote:
> >>>>
> >>>> Please try launchpad 893. I think it should be faster on GAE.
> >>>> We can do better with lazy tables but at least the validators and
> >>>> calls to getitem are eliminated.
> >>>>
> >>>> Massimo
> >>>>
> >>>> On Jun 8, 1:05 pm, Markus Gritsch <[email protected]> wrote:
> >>>>> On Mon, Jun 8, 2009 at 5:54 PM, mdipierro<[email protected]>
> wrote:
> >>>>>
> >>>>> > the web2py in trunk can execute models 2.5x faster than the current
> >>>>> > stable/production version (requires migrate=False and bytecode
> >>>>> > compiled models).
> >>>>>
> >>>>> Will this speedup also has an effect on GAE? IMO one uploads no .pyc
> files?
> >>>>>
> >>>>> Markus
> >>>> > >>>>
> >>>>
> >>>
> >>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
--- /home/snake/Desktop/sql.py 2009-06-08 09:36:27.225210464 +0400
+++ sql.py 2009-06-08 10:26:42.605233069 +0400
@@ -513,13 +513,16 @@
"""
def __getitem__(self, key):
+ value = dict.__getitem__(self, str(key))
+ if not callable(value) or key[0]=='_' or not isinstance(value, types.FunctionType): return value
+# print 'calling',value
+ value.__call__() # That must redefine table in-place
return dict.__getitem__(self, str(key))
+ __getattr__ = __getitem__
def __setitem__(self, key, value):
dict.__setitem__(self, str(key), value)
- def __getattr__(self, key):
- return dict.__getitem__(self,key)
def __setattr__(self, key, value):
if key in self:
@@ -942,14 +945,14 @@
):
for key in args:
- if key != 'migrate':
+ if key not in ('migrate', 'redefine'):
raise SyntaxError, 'invalid table attribute: %s' % key
migrate = args.get('migrate',True)
tablename = cleanup(tablename)
- if hasattr(self,tablename) or tablename[0] == '_':
- raise SyntaxError, 'invalid table name: %s' % tablename
- if tablename in self.tables:
+ if tablename in self.tables and not args.get('redefine', False):
raise SyntaxError, 'table already defined: %s' % tablename
+ elif tablename[0] == '_' or getattr(SQLDB, tablename, None):
+ raise SyntaxError, 'invalid table name: %s' % tablename
t = self[tablename] = SQLTable(self, tablename, *fields)
# db magic
--- sql.py.882 2009-06-09 14:48:24.000000000 +0400
+++ sql.py 2009-06-09 16:00:50.000000000 +0400
@@ -1112,7 +1112,7 @@
@raise SyntaxError: when a supplied field is of incorrect type.
"""
- new_fields = []
+ new_fields = [ SQLField('id', 'id') ]
for field in fields:
if isinstance(field, SQLField):
new_fields.append(field)
@@ -1120,33 +1120,38 @@
new_fields += [copy.copy(field[f]) for f in
field.fields if f != 'id']
else:
- raise SyntaxError, \
- 'define_table argument is not a SQLField'
- fields = new_fields
- self._db = db
- self._tablename = tablename
- self.fields = SQLCallableList()
- self._referenced_by = []
- fields = list(fields)
- fields.insert(0, SQLField('id', 'id'))
- for field in fields:
- self.fields.append(field.name)
- self[field.name] = field
- field._tablename = self._tablename
+ raise SyntaxError, 'define_table argument is not a SQLField'
+ getitem=dict.__getitem__
+ setitem=dict.__setitem__
+ setitem(self, '_db', db)
+ setitem(self, '_tablename', tablename)
+ fields = SQLCallableList()
+ setitem(self, 'fields', fields)
+ setitem(self, '_referenced_by', [])
+ for field in new_fields:
+ field_name=field.name
+ fields.append(field_name)
+ setitem(self, field_name, field)
+ field._tablename = tablename
field._table = self
- field._db = self._db
- if isinstance(field.type,str) and field.type[:9] == 'reference':
- referenced = field.type[10:].strip()
- if not referenced:
- raise SyntaxError, 'SQLTable: reference to nothing!'
- if not referenced in self._db:
- raise SyntaxError, 'SQLTable: table does not exist'
- referee = self._db[referenced]
- if self._tablename in referee.fields:
- raise SyntaxError, 'SQLField: table %s has same name as a field in referenced table %s' % (self._tablename, referee._tablename)
- referee._referenced_by.append((self._tablename, field.name))
-
- self.ALL = SQLALL(self)
+ field._db = db
+ field_type = field.type
+ try:
+ if field_type[:9] != 'reference':
+ continue
+ except:
+ referenced = field_type[10:].strip()
+ try:
+ referee = getitem(db, 'referenced')
+ except:
+ if not referenced in db:
+ if not referenced:
+ raise SyntaxError, 'SQLTable: reference to nothing!'
+ raise SyntaxError, 'SQLTable: table does not exist'
+ if tablename in getitem(referee, 'fields'):
+ raise SyntaxError, 'SQLField: table %s has same name as a field in referenced table %s' % (tablename, referee._tablename)
+ getitem(referee, '_referenced_by').append((tablename, field_name))
+ setitem(self, 'ALL', SQLALL(self))
def _filter_fields(self, record, id=False):
return dict([(k, v) for (k, v) in record.items() if k
@@ -1718,13 +1723,14 @@
raise SyntaxError, 'SQLField: invalid field name'
if isinstance(type, SQLTable):
type = 'reference ' + type._tablename
- if not length and type == 'string':
- type = 'text'
- elif not length and type == 'password':
- length = 32
- self.type = type # 'string', 'integer'
- if type == 'upload':
+ elif type == 'upload':
length = 64
+ elif not length:
+ if type == 'string':
+ type = 'text'
+ elif type == 'password':
+ length = 32
+ self.type = type # 'string', 'integer'
self.length = length # the length of the string
self.default = default # default value for field
self.required = required # is this field required
@@ -1746,10 +1752,11 @@
self.label = ' '.join([x.capitalize() for x in
fieldname.split('_')])
if requires == sqlhtml_validators:
- requires = sqlhtml_validators(type, length)
+ self.requires = sqlhtml_validators(type, length)
elif requires is None:
- requires = []
- self.requires = requires # list of validators
+ self.requires = []
+ else:
+ self.requires = requires # list of validators
def formatter(self, value):
if value is None or not self.requires:
