Hiya Olek, You have some interesting observations and I wanted to chime in with my thoughts:
On Dec 29, 2011, at 12:16 PM, Olek Poplavsky wrote: > So anyway, I used Sequel models and bound variables, and prepared > statements, and found that while my 'best case scenario' action > response times went down about 2-3 times, and profiling with ruby-prof > proves about same kind of improvement, overall response time under > stress DROPPED about 20%. Might be interesting to have some code to demonstrate your issues. AR vs. Sequel would be even more enlightening. > After much of investigation it all boiled down for the fact that for > one reason or another Sequel is more wasteful with memory, comparing > to my old ActiveRecord code. Essentially, my new code is generating > DOUBLE of wasted memory that needs to be garbage collected, and my > garbage collection times are starting to become large then my 'useful > computing' times. For 500 executions of my old code about 350MB of > memory would be 'spent and collected', and for new code that number > went up to 600MB, number of allocations themselves went from 7M to 9M. I have experienced the opposite of this. My switch to Sequel has kept object allocation down overall and I find that the garbage collector is much happier these days. I use Sequel on JRuby and it is amazingly efficient. That said, the JVM is much easier to tune in regards to GC - giving me an option to use one of several collectors if needed. > That problem could be overcome to some extent by using patched version > of ruby that allows fine tuning of garbage collection, and I tested > that by switching from MRI 1.8.7 to REE 1.8.7. Yes, definite > improvements, and at the end my app is faster than it ever was... but > still good chunk of time is wasted in GC, and this kind of 'solution' > is not quite right. > So, to sum it up, it appears that somewhere inside Sequel code is > generating a bunch of short-lived objects, more then it really should > (based on GC stats that I have seen, Strings and Arrays are most > likely culprit). It is not the code that generates queries, for sure, > since I am using prepared statements 100%, it is code that is related > to loading data. Short-lived objects are what you want in general. That way they can be collected. Otherwise you have the heap growing and growing. (unless I'm missing the point here) The problem here is MRI's garbage collection. In fact, it is what led Twitter to write Kiji [1]. Perhaps there are some strategies you could use to have your commonly used objects kept around and reduce the churn a bit? Again, let's see some code and we can all take a stab at it. > BTW - on a related note - Sequel::Timezones#convert_timestamp is one > of the most expensive functions there, it takes about 0.3ms to run it > just once, so for table containing 6 DateTime columns, total time > spent adjusting timezones is about 2ms, which is not so good > considering that the time to run query against database was about > 0.6ms. Jeremy has spoken about Ruby time conversions many times and led him to write home_run [2]. I think you can also turn off time conversions and do them yourself when you absolutely need it. > Keep up the good work, guys! If I am sorry about anything now, that > would be "why, oh why, I have not discovered Sequel 3 years ago???" :) We all love it. I have thought this same thing many, many times. Don [1] http://engineering.twitter.com/2011/03/building-faster-ruby-garbage-collector.html [2] https://github.com/jeremyevans/home_run -- You received this message because you are subscribed to the Google Groups "sequel-talk" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/sequel-talk?hl=en.
