Ok, it's here http://code.google.com/p/web2py/issues/detail?id=374
Thank you for looking into this Massimo! I do not know the best way to
do this... my code is just a first reaction to making something
faster.
On Aug 11, 2:55 am, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:
> This is really interesting. Please give me some time to study it,
> meanwhile, so that I do not forget, please open an issue and post the
> code there.
>
> Massimo
>
> On Aug 10, 7:11 pm, MichaelToomim<too...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Ok. The basic idea is to allow you to define helpers methods on rows,
> > sort of like the Models of rails/django.
>
> > You use it like this... I put this in models/db_methods.py:
>
> > @extra_db_methods
> > class Users():
> > def name(self):
> > return '%s %s' % ((self.first_name or ''),
> > (self.last_name or ''))
> > def fb_name(self):
> > p = self.person()
> > return (p and p.name) or 'Unknown dude'
> > def person(self):
> > return db.people(db.people.fb_id == self.fb_id)
> > def friends(self):
> > return [Storage(name=f[0], id=f[1])
> > for f in sj.loads(self.friends_cache)]
>
> > @extra_db_methods
> > class People():
> > ... etc
>
> > These are for tables db.users and db.people. It looks up the table
> > name from the class name. For each table that you want to extend, you
> > make a class and put @extra_db_methods on top.
>
> > It's implemented with the following @extra_db_methods decorator and a
> > patch to dal.py. The decorator just traverses the class, pulls out all
> > methods, and throws them into a "methods" variable on the appropriate
> > table in dal. Then the dal's parse() routine adds these methods each
> > row object, using the python type.MethodType() routine for
> > retargetting a method from one class to another object.
>
> > The downside is extending dal with yet ANOTHER way of adding methods
> > to objects. That makes 3 apis to maintain for similar things
> > (virtualfields, computedfields, and this). And I'm not sure about the
> > names (like "extra_db_methods") for these things yet. Also I think we
> > might be able to get it even faster by being more clever with python
> > inheritance in the Row class. Right now it has roughly 10% overhead on
> > selects in my tests (uncompiled code).
>
> > At the bottom of this message is the decorator that implements the
> > same functionality using the existing virtualfields mechanism and your
> > "lazy" decorator. Its downside is a 2x to 3x overhead on selects and
> > instead of self.field you have to say self.<tablename>.field in the
> > method bodies.
>
> > def extra_db_methods(clss):
> > tablename = clss.__name__.lower()
> > if not tablename in db:
> > raise Error('There is no `%s\' table to put virtual methods in'
> > % tablename)
>
> > for k in clss.__dict__.keys():
> > method = clss.__dict__[k]
> > if type(method).__name__ == 'function' or type(method).__name__
> > == 'instancemethod':
> > db[tablename].methods.update({method.__name__ : method})
>
> > return clss
>
> > --- k/web2py/gluon/dal.py 2011-08-03 16:46:39.000000000 -0700
> > +++ web2py/gluon/dal.py 2011-08-10 17:04:48.344795251 -0700
> > @@ -1459,6 +1459,7 @@
> > new_rows.append(new_row)
> > rowsobj = Rows(db, new_rows, colnames, rawrows=rows)
> > for tablename in virtualtables:
> > + rowsobj.setmethods(tablename, db[tablename].methods)
> > for item in db[tablename].virtualfields:
> > try:
> > rowsobj =
> > rowsobj.setvirtualfields(**{tablename:item})
> > @@ -4559,6 +4560,7 @@
> > tablename = tablename
> > self.fields = SQLCallableList()
> > self.virtualfields = []
> > + self.methods = {}
> > fields = list(fields)
>
> > if db and self._db._adapter.uploads_in_blob==True:
> > @@ -5574,6 +5576,14 @@
> > self.compact = compact
> > self.response = rawrows
>
> > + def setmethods(self, tablename, methods):
> > + if len(methods) < 0: return
> > + for row in self.records:
> > + if tablename not in row: break # Abort on this and all
> > rows. For efficiency.
> > + for (k,v) in methods.items():
> > + r = row[tablename]
> > + r.__dict__[k] = types.MethodType(v, r)
> > + return self
> > def setvirtualfields(self,**keyed_virtualfields):
> > if not keyed_virtualfields:
> > return self
>
> > ---
> > And Here's the implementation using virtualfields:
>
> > def lazy(f):
> > def g(self,f=f):
> > import copy
> > self=copy.copy(self)
> > return lambda *a,**b: f(self,*a,**b)
> > return g
>
> > def extra_db_methods_vf(clss):
> > ''' This decorator clears virtualfields on the table and replaces
> > them with the methods on this class.
> > '''
> > # First let's make the methods lazy
> > for k in clss.__dict__.keys():
> > if type(getattr(clss, k)).__name__ == 'instancemethod':
> > setattr(clss, k, lazy(getattr(clss, k)))
>
> > tablename = clss.__name__.lower()
> > if not tablename in db:
> > raise Error('There is no `%s\' table to put virtual methods in'
> > % tablename)
> > del db[tablename].virtualfields[:] # We clear virtualfields each
> > time
> > db[tablename].virtualfields.append(clss())
> > return clss
>
> > You use this just like before but with @extra_db_methods_vf instead of
> > @extra_db_methods, and append <tablename> to each use of "self".
>
> > On Aug 9, 11:16 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>
> > wrote:
>
> > > let us see it!
>
> > > On Aug 9, 9:36 pm, MichaelToomim<too...@gmail.com> wrote:
>
> > > > Result: Fixed by upgrading. I was seeing this
> > > > bug:http://code.google.com/p/web2py/issues/detail?id=345
>
> > > > However, virtualfields still take more time than they should. My
> > > > selects take 2-3x longer with virtualfields enabled than without. I
> > > > implemented a little hack in the dal that adds methods to rows with
> > > > only a 10% overhead (instead of 200-300%) and can share that if
> > > > anyone's interested.
>
> > > > On Aug 8, 8:38 pm, MichaelToomim<too...@gmail.com> wrote:
>
> > > > > It turns out the speed problem is REALLY bad. I have a table with
> > > > > virtualfields of 14,000 rows. When I run raw sql:
>
> > > > > a = db.executesql('select * from people;')
>
> > > > > ...the query returns in 121ms. But when I run it through the DAL on
> > > > > only a subset of the data:
>
> > > > > a = db(db.people.id > 0).select(limitby=(0,1000))
>
> > > > > ...it returns in 141096.431ms. That's... 141 seconds. So 1000x longer
> > > > > on .1 of the database.
>
> > > > > My virtualfields are all lazy functions. I'm looking into what's
> > > > > causing it and will report back when I find out. It seems it might
> > > > > have something to do with the lazy decorator func because when I hit
> > > > > C-
> > > > > c the code is often stuck there... inside import copy or something.
>
> > > > > def lazy(f):
> > > > > def g(self,f=f):
> > > > > import copy
> > > > > self=copy.copy(self)
> > > > > return lambda *a,**b: f(self,*a,**b)
> > > > > return g
>
> > > > > Anyway, I'll send an update when I have more info.
>
> > > > > On Aug 2, 3:03 pm, MichaelToomim<too...@gmail.com> wrote:
>
> > > > > > That's way better syntax! Great idea!
>
> > > > > > On Aug 2, 2011, at 2:31 AM, Massimo Di Pierro wrote:
>
> > > > > > > We need to work on the speed. This can perhaps help the syntax:
>
> > > > > > > db=DAL()
> > > > > > > db.define_table('a',Field('b','integer'))
> > > > > > > for i in range(10):
> > > > > > > db.a.insert(b=i)
>
> > > > > > > def lazy(f):
> > > > > > > def g(self,f=f):
> > > > > > > import copy
> > > > > > > self=copy.copy(self)
> > > > > > > return lambda *a,**b: f(self,*a,**b)
> > > > > > > return g
>
> > > > > > > class Scale:
> > > > > > > @lazy
> > > > > > > def c(self,scale=1):
> > > > > > > return self.a.b*scale
>
> > > > > > > db.a.virtualfields.append(Scale())
> > > > > > > for row in db(db.a).select():
> > > > > > > print row.b, row.c(1), row.c(2), row.c(3)
>
> > > > > > > On Aug 1, 3:10 pm, MichaelToomim<too...@gmail.com> wrote:
> > > > > > >> Maybe it helps for me to explain my use-case. I mainly use
> > > > > > >> virtual fields as lazy methods, to help traverse related tables.
> > > > > > >> I was actually surprised that lazy evaluation wasn't the
> > > > > > >> default. I noticed a few implications of this:
> > > > > > >> - Large queries are slowed byvirtualfields, even if they won't
> > > > > > >> be needed, esp if they query db
> > > > > > >> - My definitions forvirtualfieldsaren't as clean as they could
> > > > > > >> be, because I have many nested "lazy" funcs in the class
> > > > > > >> definition
> > > > > > >> - We can't serialize all objects intosessionvariables
>
> > > > > > >> So really I'm just using this because it's a nicer notation to
> > > > > > >> call row.otherthing() instead of getotherthing(row). Maybe I
> > > > > > >> really want some different feature here?
>
> > > > > > >> On Aug 1, 2011, at 5:40 AM, Anthony Bastardi wrote:
>
> > > > > > >>> Note, after looking at this some more, Massimo recalled that
> > > > > > >>> the reason auth_user virtual fields were excluded from
> > > > > > >>> auth.user (and therefore from saving in thesession) is because
> > > > > > >>> some virtual fields are objects that cannot be pickled and
> > > > > > >>> therefore cannot be serialized to store in thesession. So,
> > > > > > >>> we're thinking of either creating an option to store auth_user
> > > > > > >>> virutual fields in auth.user, or maybe testing to make sure the
> > > > > > >>> virtual fields can be pickled, and excluding them if not.
>
> > > > > > >>> Anthony
>
> > > > > > >>> On Mon, Aug 1, 2011 at 5:30 AM,
> > > > > > >>> MichaelToomim<too...@cs.washington.edu> wrote:
> > > > > > >>> Awesome! I did not know there was an issue submission system.
>
> > > > > > >>> On Jul 30, 2011, at 7:02 AM, Anthony wrote:
>
> > > > > > >>>> An issue has been submitted, and this should be
>
> ...
>
> read more »
