Hi all, I have good news and bad news.
The good news is, I have a brand new bulk uploader that works with 0.6 and its limitations (unless proven otherwise). The bad news is, I've had a look at the SQL queries the Rails port is making to the DB during a mass import, and all I could think is "WTF?!?" (Rants follows) So, let's have a look at this: [2009-02-10 01:00:34.809017 #1664] User Load (0.000144) SELECT * FROM `users` WHERE (`users`.`id` = 1) [2009-02-10 01:00:34.810457 #1664] SQL (0.000144) SELECT `display_name` FROM `users` WHERE (`users`.display_name = 'ivansanchez' AND `users`.id <> 1) [2009-02-10 01:00:34.811241 #1664] SQL (0.000108) SELECT `email` FROM `users` WHERE (`users`.email = '[email protected]' AND `users`.id <> 1) WTF does this happen on every object (read: node) I upload? The API already knows who I am the second I started to upload the diff! [2009-02-10 01:00:34.863175 #1664] SQL (0.000244) SELECT `id` FROM `current_nodes` WHERE (`current_nodes`.id IS NULL) current_nodes.id is defined as NOT NULL. So, WTF? [2009-02-10 01:00:34.864872 #1664] SQL (0.000258) SELECT `id` FROM `changesets` WHERE (`changesets`.id = 44 AND `changesets`.id <> 44) ***WTF***???!!!! [2009-02-10 01:00:34.859181 #1664] Changeset Update (0.000314) UPDATE `changesets` SET `num_changes` = 18443 WHERE `id` = 44 Couldn't this wait until the diff upload is complete? (Rants kind of end) So, while I'm no expert about Rails, I do suspect that all those extra unnecessary queries are done automagically. I do know OSM will have some brand new DB server (now with 0.1 more API!) that should be able to handle the extra overhead, and that most of those queries will hit data that will be cached anyway. I also do think that there is a lot of room for optimizations (namely, removal of DB overhead), so... Is it worth to optimize the code for diff uploads? Cheers, -- ---------------------------------- Iván Sánchez Ortega <[email protected]> Q: What's the difference between Bell Labs and the Boy Scouts of America? A: The Boy Scouts have adult supervision.
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
