Hi, thanks for raising this. It's very much on my mind because I've been working with/on lp:kanban, which wants to regularly poll bug collections from Launchpad. Even running from another machine in the Canonical data centre (so mere ms from Launchpad) it's pretty slow. If anyone is going to work on this I would recommend you try running it and have a look at what network traffic it actually does.
A few thoughts, somewhat related to Robert's points: Nearly any practical operation on bugs is going to take O(n) round trips, because nearly anything will want both the bug and the bugtask(s) and you cannot get them both in batches. It would be good to work out a way to send them all at once to avoid potato programming. I don't know about the limitations of the mapping layer but conceptually and in terms of network representation it should not be that hard to say that tasks are value objects sent inline with the enclosing bug. Fixing this kind of thing would speed up many clients, and I expect also reduce the server load. Similarly for merge proposals. Launchpad could either take an opinion that tasks should just always be sent in line with bugs, or could have a way for the client to express "please send me all the comments too", which is the expand case. http://pad.lv/712924 shows that searchTasks is much slower than reading a collection. I have not dug into why but it may be that the batch size is smaller. I find it kind of weird that Launchpad in some ways is quite pedantic about HTTP and being restful, and then in others has very rpc-like non-restful methods to get collections and do searches. I would like if the URLs looked more like collections, ie /~mbp/bugs/status=In+Progress/ or something. It might make them more cacheable, and more understandable for client developers. I have a sense that people do not look enough at what actually goes across the wire, and things would be better if they did. If you look at almost any other web service with an api, they talk about the URLs that are generated and give examples of the XML/json that is sent back. Launchpad acts like it might as well be a black box (or black pipe), or CORBA. <http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing> If an api client wants to read for example all the open bugs assigned to ~mbp, there does not seem much point making them read multiple batches. There will just be more roundtrips and more overhead for both the client and the server, and more risk of tearing between pages. Unlimited size collections would be very good. I think many clients are going to be interested in most-recently-changed collections, either because they are drawing a timeline-like view, or because they are updating a local cache, or _could_ have a local cache if it was easy to keep it fresh. This can suffer tearing, but of a very predictable and manageable kind. More generally I agree that batching based on actual keys will be just as good, or better, for clients, and easier on the database. (Again I can't help thinking if you thought about the URLs rather than thinking about the client-specific abstraction of a Python list this might come out more naturally.) For example, lp:kanban really only wants to know about bugs that have changed since last time it was run; if lp and lplib easily and efficiently supported that it could be doing about 1 http request per update, not thousands, with something like: /~mbp/bugs/assigned/open/modified_since=123123123 I could say a bunch of stuff about what a better client (Wrested) would look like especially with caching but that's perhaps separate. You could have a smart cache that knew how to freshen itself with a query such as given above, without rereading all bugs. I think web UI batching is nearly pointless: if there are hundreds of bugs it will be an act of desperation to actually page through them. People want a more selective search. Seeing the total count is useful, and perhaps being able to page through them just to get a sense of what's there will be occasionally used. (We could get data on this by seeing what fraction of requests for a bug page have an offset other than 0.) Spiders will walk through it; perhaps there should be a spider-specific dump view grouped by date so the pages stay stable. hth Martin _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
