Hi Ian, yes this is a problem I see too. I am totally for separating data mapping and domain logic. I don't remember which threads I've mentioned this in, but the first and second Performance Tuesday threads probably have some ramblings.
I don't think the units tests are a key aspect of it - performance of those tests is very important to us for agility and dev throughput, but there are more profound and significant problems that are caused by the current structure. And, optimising the unit tests should come *after* optimising production : if we, for instance, made all our pages twice as fast, we can reasonably expect the test suite to get faster (because the test suite is exercising code paths that are now twice as fast). But the reverse isn't true: making the test suite faster is much less likely to help production performance [unless one makes the test suite faster by making production code paths faster]. A primary problem I see is that we have no high level language [other than object traversal] to describe how much of the object graph we wish to retrieve. E.g. 'Milestone X, all specs & bugs, the product and project group, all people associated with X, or in key roles on any related item, branding for any teams found'. We have a bunch of adhoc stabs in this direction; things like the recent GenericCollection, and my recent refactoring of Person.all_members. None of them are hugely satisfactory *yet* : you cannot combine different collections reliably (yet - that might be a nice thing to work towards); and its very ad hoc. Because we have no high level language to describe what a page will need, domain objects end up requesting more of the object graph in method calls and attribute access (because, until -very- recently, we didn't actually build an object graph, rather we always went back to SQL every time). As a result pages often end up issuing hundreds or thousands of queries in a very inefficient manner. And so things are slow. I see this as a limitation of Storm, our ORM : other ORM's (both in and out of the python world) have an explicit, decoupled, mapping layer. Working with Storm though, the usual idiom (at least in Launchpad - I need to spend a couple of days with Jamu in the Landscape code looking for inspiration) is to be expressing things at the SQL layer. In terms of what we should do here; I was imagining a mapping *layer*, not necessarily DAO classes, - things like GenericCollection, and BugTaskSet are, to me, mappers, and would fit in that layer. I'd love to have the layer be Storm owned and driven. Being able to express - without SQL - the cut points in object graph traversal and have a mapping layer return a resultset for the resulting graph would be -awesome-. We should, I think, be able to sensibly aim for 10 queries per page - even the busiest pages rarely have more than 10 discrete regions showing unrelated data. Salgado, in terms of being able to migrate incrementally, I see it as a fairly high risk proposition today : we haven't finished the SQLObject migration, and eager loading interactions with that are really rather nasty : things that look like they should work don't, we have silently failing methods (like __nonzero__), and and unknown set of other problems. As a strategic concern, coding for Launchpad is full of hidden complexity. I'd like use to incrementally remove and improve on that. Less global state, more things-on-contexts (e.g. Request.annotations), and clearer code in the mapping/domain area would be awesome too. To me, finishing the SQLObject migration will remove a bunch of nasty surprises and give us a saner base to go forward with experiments on mapping. To be clear: I'm ok with folk experimenting with different sorts of mapping; (I don't think a parallel class structure is necessarily all that great - in python we could be more dynamic than that). However, I think it will be very hard to do well, safely and incrementally until SQLObjectResultSet specifically, is purged. -Rob _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
