Use the profiler to make your tests even more effective
2013/11/19 Arnon Marcus <[email protected]> > 10x Niphlod, I'll check that tomorrow... > > I can provide any code you like - this is solid and consistent. > > I am running this on my desktop-workstation which runs windows7x64. > It's an Intel Core-i7 870 w/ 8GB RAM, running Python 2.6x64 via a simple > command-line g-event server... > Don't think I have a problem there... > > The database itself sits as a stand-alone service on an old "boxx" > pizza-box-like server on our server-room, running nothing but PostgreSQL > 9.3 (x64) and PG-Bouncer (for connection pooling) - on a minimalist CentOS > 6.4 (x64) installation. > I optimized it as far as I can, with the data weighing less than a 100MB > total, I have the data-folder sitting on a mounted "tmpfs" of 3GB (a-la > RAMDISK), and the schema is heavily "index"ed,, so the queries are as fast > as they can be. > > As I said, it may be a case of simply having a very fast database-story... > > "I'd guess your usecase is either 1k rows with 50" > That's exactly what I said - yes, it's a ~1K rows query, of a JOIN of 2 > tables, selecting for about 20 columns. > That is our use-case - it can't be any different (at least not easily...) > > We are not really using this version of web2py - I just did that for > texting and posting here. > But we initially encountered this case using a very old version of web2py > (1.8.95) - in which the exact numbers where: > > 904 rows in the result-set: > - Regular select() : 2.241 seconds > - In it, the portion that the executesql took: 0.023 seconds. > That is roughly a x44 times gap... > > And as I said - Trying to convert the result-set into a simple list of > simple-class-instances, yielded similar results. > Which made it conclusive : The overhead was in the object-instanciation > and other pluming-related stuff like "__getitem__" and ".iteritems()" and > the like... > > I think the test-code you gave here is interesting - but to make it closer > to our use-case, I would have a more complex query tried. > First, have 2 tables with a foreign-key linking them. > Second, they should each have about ~20+ fields of varying types. > Then the query should be a join of them, getting most/all columns. > > What I notices, was that a "flat" query (one with no JOINs) may end-up > being MUCH heavier to parse into Rows (or Rows-like) objects, as there are > nested-loops involved, for having these intermediary Table-representing > objects, that then have to have the result-set filtered-into them, > according to each colname's "table-name" portion. > i.e: > Lets say we have 3 tables: > - Budgets: > name (string) > project (foreign-key to Projects) > - Projects: > name (string) > client (foreign-key to Clients) > - Clients: > name (string) > > Now, if we JOIN them in a query, each result in the result-set would have > to be segregated into either of the 3 table-representational-objects in > each Row object. > > A colname-attached result-record may look like this: > { > 'Budgets.name':'My budget', > 'Budgets.project':742, > 'Projects.name':'My project', > 'Projects.client':445, > 'Clients.name':'My client' > } > > A Row object's general structure would have to end-up looking like this: > { > 'Budgets':{ > 'name':'My budget' > 'project:742 > } > 'Projects':{ > 'name':'My project' > 'project:445 > } > 'Clients':{ > 'name':'My client' > } > { > > Now, multiply the tables and/or columns, and you already have a very > different story of record-filtering/segregation and object-instanciation... > Very different than the test-code you posted here. > > If you then multiply that by a 1000, you may end-up getting 4000 > object-instanciations just for this simple example (for each Row object, of > which there would be a 1000. there would be another 3 objects - meaning, > another 3000 objects...) And that is just a join of just 3 tables - we have > many queries much-much more complex than that... > > The interesting (and somewhat disappointing) factoid I got from your > results, was that PyPy was just as "slow" in un-parsed mode... ;) > > -- > Resources: > - http://web2py.com > - http://web2py.com/book (Documentation) > - http://github.com/web2py/web2py (Source code) > - https://code.google.com/p/web2py/issues/list (Report Issues) > --- > You received this message because you are subscribed to the Google Groups > "web2py-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
