Hi Jeremy
I was able to fetch a bit more information on the subject.
First of all, memprof back-end processing seems to be not working
right, my uploaded data got stuck there in their processing queue.
Still, data I was able to get from it locally was very valuable.
I found that replacing 'model-centric' query with 'table-centric'
query saved me about 40kb per query. Does not seem like much, but with
5 of those queries per request and 500 request in a row this change
alone reduced 'short-lived objects' total size from 600MB to 500MB.
Old query:
Sequel::Channel.
eager_graph(:subscriptions).
filter(:subscriptions__user_id => :$user_id).
select(:channels__id).
order(:id).
ungraphed
New query:
SDB[:users].
join(:subscriptions, :user_id => :id).
filter(:users__id => :$user_id).
select(:subscriptions__channel_id___id).
order(:id)
Still, this query seems to create lots of strings and hashes in
memory. It turned out that postgres driver returns back rows as hashes
(sure this is not a surprise to you, Jeremy). Therefore, for each row
it creates new Hash object (waste in this degenerate case of one
column per row), and this Hash object contains key-value pairs as
field-name-to-field-value. Field name is a string, and therefore is
repeated for each and every row, generating more wasted memory.
Replacing Sequel query with
pg = ActiveRecord::Base.connection.execute("select channel_id
from subscriptions where user_id = #{self.id} order by channel_id")
rtn.concat(pg.column_values(0).map { |n| Integer(n) })
resulted in 80kb decrease of waste per request (for one query change).
If I do that for all 5 queries, for 500 requests that is 200MB less
waste, bringing me from 500MB to 300MB.
But of course, this code is @#$ ugly. Which leads me to the following
question. Is it possible to use Sequel for query generation/
preparation/execution but... not have it parse the results and instead
hand me back PGresult object?
This would be an almost palatable compromise...
And BTW - quickly running memprof on some of the old ActiveRecord code
suggested that generic ActiveRecord-centric code is much more
wasteful, and that pretty much makes sense... but I do not know how to
explain initial better GC results for ActiveRecord-centric version of
the code. Still there is a mystery here to uncover.
And, in case you are curious, here is a top section of memprof stats
for my whole action (as you can see, it is mostly sequel with
exception of YAJL json generator and some other small chunks).
48 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/gems/yajl-
ruby-1.1.0/lib/yajl.rb:72:String
39 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/model/associations.rb:2095:Array
36 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/adapters/postgres.rb:468:String
33 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/model/associations.rb:2085:Array
25 /Users/bar/.rvm/rubies/ruby-1.8.7-p352/lib/ruby/1.8/webrick/
httpserver.rb:55:Array
24 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/model/associations.rb:2189:Array
24 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/actions.rb:705:String
22 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/adapters/postgres.rb:468:Hash
21 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/model/associations.rb:2202:Array
19 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/misc.rb:128:String
16 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/query.rb:55:Class
16 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/prepared_statements.rb:201:Hash
16 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/actions.rb:664:Hash
16 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/adapters/postgres.rb:540:Hash
16 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/adapters/postgres.rb:428:__scope__
15 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/adapters/postgres.rb:428:String
14 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/model/associations.rb:2196:Array
14 /Users/bar/.rvm/rubies/ruby-1.8.7-p352/lib/ruby/1.8/webrick/
httpserver.rb:55:__node__
12 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/model/associations.rb:2196:String
12 /Volumes/Foo/src/proj/app/models/sequel/user.rb:54:__varmap__
11 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/gems/
actionpack-3.0.10/lib/action_view/lookup_context.rb:119:String
9 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/gems/
actionpack-3.0.10/lib/action_view/lookup_context.rb:120:Array
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/gems/
sequel_pg-1.2.0/lib/sequel_pg/sequel_pg.rb:49:__scope__
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/query.rb:57:__node__
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/query.rb:57:Array
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/query.rb:56:Hash
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/query.rb:
55:Sequel::Postgres::Dataset
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/prepared_statements.rb:
26:__scope__
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/prepared_statements.rb:
125:__scope__
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/actions.rb:664:__scope__
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/actions.rb:55:__scope__
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/actions.rb:54:Array
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/dataset/actions.rb:127:Proc
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/database/query.rb:54:__scope__
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/database/query.rb:54:Array
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/database/connecting.rb:225:__scope__
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/database/connecting.rb:224:Proc
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/connection_pool/single.rb:22:__varmap__
8 /Volumes/Foo/src/proj/vendor/bundle/ruby/1.8/bundler/gems/
sequel-47e347167fd9/lib/sequel/adapters/shared/postgres.rb:
699:__scope__
......
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sequel-talk?hl=en.
