-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Michael Koziarski wrote: >>>I currently use RoR on a few small web sites, and some internal webapps, but >>>I spend 90% of my time with RoR with AR optimizing >>>it's speed. > > > Are you only noticing this speed problem with big batch operations > like you mentioned in the earlier email?
The bigger the batch, the more time it takes, so it's easier to notice big speed issues with 500,000 records, rather then 50 records. Alot of what rails (and AR) give the user in terms of flexibility and ease of use comes with a huge performance penalty. I am willing to sacrifice niceties for better performance. > When working with an ORM, it's fairly common to do heavy data imports, > batch loads and the like using the database's built in tools, what > makes you want to use AR instead of tools like mysqldump etc? > There is more to my story though, but I didn't want to rant on to long in my last email. Myself and a colleague of mine, Mark Van Holstyn, came up with a idea for an IO Mode plugin for AR. It works beautifully. You can turn it off and on, on the fly. It is great for debugging rails SQL commands if you don't wan't to parse log files. I use my AR plugin in combination with this AR IO mode plugin to generate all of my SQL statements and I have them written to disk. I then batch load them via a mysql command line client. In one instance every few hours I receive an update from another system. Some updates are very small, a few kb in size, other updates it's 15Mb to 40Mb of straight data. It doesn't come formatted in SQL statements, so I have to parse the data myself. It is faster for me to deal with my AR plugin API, and let AR do all of the hard work for me, generating statements, quoting, etc. plus I get all of the already builtin AR functionality for free should I need to extend my work. It is also easy to maintain, since we use AR, ruby and the rest of rails for alot of this system on the backend, we worry about ruby coding and working withing ruby-based method APIs. We also get a huge code base to use, which is nice and easy to extend. I am working on limiting this performance gap for mysql though, to where things don't need to be batch loaded outside of a ruby instance. This is limited though due to how ruby's GC work, if you want to continuous run a daemon that does these huge data operations, but you never restart the daemon. Another problem I face is the max_allowed_packet size issue. One of my next todo's is allow the user to configure whether or not they want it dynamically resized per that session, or if they want the AR plugin to split up large queries into the least # of queries to support the current max_allowed_packet size. And to answer your questions after 6 paragraphs, =) , I use mysqldump and other tools where it makes sense. But I need a way to get from data_format_a to data_format_sql, and I prefer ruby in that regard with AR at the moment. Zach -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFERXuXMyx0fW1d8G0RAuKPAJ920A+gWxpX4+UhTZgKIZgw2RamWQCdFnMX qWMGmY9yxxs4QdqzIczCP6g= =m1Bi -----END PGP SIGNATURE----- _______________________________________________ Rails-core mailing list Rails-core@lists.rubyonrails.org http://lists.rubyonrails.org/mailman/listinfo/rails-core
