Your IO Mode plugin sounds interesting too... care to share it? That and this bulk AR/virtual table both sound useful to me. Release them as plugins? : )
-jeff On 4/18/06, zdennis <[EMAIL PROTECTED]> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Michael Koziarski wrote: > >>>I currently use RoR on a few small web sites, and some internal webapps, > >>>but I spend 90% of my time with RoR with AR optimizing > >>>it's speed. > > > > > > Are you only noticing this speed problem with big batch operations > > like you mentioned in the earlier email? > > The bigger the batch, the more time it takes, so it's easier to notice big > speed issues with 500,000 records, rather then 50 > records. Alot of what rails (and AR) give the user in terms of flexibility > and ease of use comes with a huge performance penalty. > I am willing to sacrifice niceties for better performance. > > > When working with an ORM, it's fairly common to do heavy data imports, > > batch loads and the like using the database's built in tools, what > > makes you want to use AR instead of tools like mysqldump etc? > > > > There is more to my story though, but I didn't want to rant on to long in my > last email. Myself and a colleague of mine, Mark Van > Holstyn, came up with a idea for an IO Mode plugin for AR. It works > beautifully. You can turn it off and on, on the fly. It is > great for debugging rails SQL commands if you don't wan't to parse log files. > I use my AR plugin in combination with this AR IO > mode plugin to generate all of my SQL statements and I have them written to > disk. I then batch load them via a mysql command line > client. > > In one instance every few hours I receive an update from another system. Some > updates are very small, a few kb in size, other > updates it's 15Mb to 40Mb of straight data. It doesn't come formatted in SQL > statements, so I have to parse the data myself. > > It is faster for me to deal with my AR plugin API, and let AR do all of the > hard work for me, generating statements, quoting, etc. > plus I get all of the already builtin AR functionality for free should I > need to extend my work. > > It is also easy to maintain, since we use AR, ruby and the rest of rails for > alot of this system on the backend, we worry about > ruby coding and working withing ruby-based method APIs. We also get a huge > code base to use, which is nice and easy to extend. > > I am working on limiting this performance gap for mysql though, to where > things don't need to be batch loaded outside of a ruby > instance. This is limited though due to how ruby's GC work, if you want to > continuous run a daemon that does these huge data > operations, but you never restart the daemon. > > Another problem I face is the max_allowed_packet size issue. One of my next > todo's is allow the user to configure whether or not > they want it dynamically resized per that session, or if they want the AR > plugin to split up large queries into the least # of > queries to support the current max_allowed_packet size. > > And to answer your questions after 6 paragraphs, =) , I use mysqldump and > other tools where it makes sense. But I need a way to > get from data_format_a to data_format_sql, and I prefer ruby in that regard > with AR at the moment. > > Zach > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.1 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > > iD8DBQFERXuXMyx0fW1d8G0RAuKPAJ920A+gWxpX4+UhTZgKIZgw2RamWQCdFnMX > qWMGmY9yxxs4QdqzIczCP6g= > =m1Bi > -----END PGP SIGNATURE----- > _______________________________________________ > Rails-core mailing list > Rails-core@lists.rubyonrails.org > http://lists.rubyonrails.org/mailman/listinfo/rails-core > -- Jeff Lindsay http://blogrium.com/ _______________________________________________ Rails-core mailing list Rails-core@lists.rubyonrails.org http://lists.rubyonrails.org/mailman/listinfo/rails-core
