Re: [sqlalchemy] ORM performance

2011-04-26 Thread Yang Zhang
On Wed, Apr 13, 2011 at 6:07 PM, Michael Bayer mike...@zzzcomputing.com wrote: To work around the instance arguments being baked in, create the query like this: query.filter(SomeClass.somerecord ==bindparam(somerecord)) The params are then added using query.params(somerecord=x). Is it

Re: [sqlalchemy] ORM performance

2011-04-26 Thread Yang Zhang
On Tue, Apr 26, 2011 at 12:32 PM, Yang Zhang yanghates...@gmail.com wrote: On Wed, Apr 13, 2011 at 6:07 PM, Michael Bayer mike...@zzzcomputing.com wrote: To work around the instance arguments being baked in, create the query like this: query.filter(SomeClass.somerecord

Re: [sqlalchemy] ORM performance

2011-04-14 Thread Chung
Thanks Michael! Re: the heavy get() workload, that's a neat trick :-) Unfortunately for our workload, we don't really know which set of gets() are going to fire, and we can't fit the entire table into memory. I was curious about whether passing in fields instead of tables into Query and

Re: [sqlalchemy] ORM performance

2011-04-14 Thread Michael Bayer
On Apr 14, 2011, at 5:07 PM, Chung wrote: Thanks Michael! Re: the heavy get() workload, that's a neat trick :-) Unfortunately for our workload, we don't really know which set of gets() are going to fire, and we can't fit the entire table into memory. I was curious about whether

[sqlalchemy] ORM performance

2011-04-13 Thread Yang Zhang
We've been finding the SA ORM to be a large component in our application execution time. Our application makes frequent queries, and most of them simply query whole rows from a single table with some filters, but no joins (therefore, very basic queries). We've found that an alarming amount of

Re: [sqlalchemy] ORM performance

2011-04-13 Thread Michael Bayer
On Apr 13, 2011, at 9:07 PM, Michael Bayer wrote: On Apr 13, 2011, at 8:48 PM, Yang Zhang wrote: We've been finding the SA ORM to be a large component in our application execution time. Our application makes frequent queries, and most of them simply query whole rows from a single table

Re: [sqlalchemy] ORM performance

2011-04-13 Thread Chung
1. Query compilation... This resulted in ~2x improvement. that seems very strange - 2x improvement inthe overall speed of your application? I've done an enormous amount of profiling - SQL compilation is miniscule compared to the SQL statement's execution itself and the fetching

Re: [sqlalchemy] ORM performance

2011-04-13 Thread Michael Bayer
On Apr 13, 2011, at 10:05 PM, Chung wrote: 1. Query compilation... This resulted in ~2x improvement. that seems very strange - 2x improvement inthe overall speed of your application? I've done an enormous amount of profiling - SQL compilation is miniscule compared to the SQL

[sqlalchemy] ORM performance questions

2009-08-06 Thread Brian Granger
Hi, I am using sqlalchemy through its orm layer. The basic flow of my program is this: 1. Download about 18000 web pages and store the raw HTML in a table 2. Iterate through each web page and use pyparsing to parse it. I then insert the results of that parsing into about a dozen different