Re: [Launchpad-dev] notes towards async api clients
I had a bit of a go at this over the weekend. It is gratifyingly fast compared to what I expect to see with launchpadlib clients, just through doing fewer requests and not unnecessarily blocking on them. I like that a lot. The code is in http://launchpad.net/wrested and lp:wrested. For instance: ./wrestler.py https://api.launchpad.net/devel/bugs/1 ./wrestler.py https://api.launchpad.net/devel/bzr/'?ws.op=searchTasks' (Try it!) It's early days but I really like how it's shaping up. I think it is less error-prone than the approach taken by txrestfulclient, because you never get half-initialized objects: if you have the resource, it's valid. I'm finding it also nicer to work with than launchpadlib because you never get unexpected pauses: all network io is explicit. It seems to me it would be healthy for Launchpad for people to be looking at what the actual http interface is, rather than using a black-box client. https://help.launchpad.net/API/Hacking was a great resource (thanks.) The basic approach is that you ask for an object and get a deferred, which will eventually deliver the object you requested. For collections, you pass a consumer which will be fed objects as they arrive. I can see this fitting very well with what's described in https://dev.launchpad.net/LEP/WebservicePerformance. I am trying to keep a separation between Launchpad-specific policy and REST in general; though I'm not quite sure yet how many conventions are standard and how many Launchpad has made up for itself. It has a nerd-oriented gtk explorer test harness: http://www.flickr.com/photos/mbp_/5386185146/ which can open an arbitrary URL, click through links to other objects, and stream reads of collections, including very large collections like all the bugs. All these capabilities are of course also exposed in the library api. It should do, but doesn't yet: * authenticated requests * socket reuse * caching * anything to do with the WADL * (a bunch of other things in the TODO) I have mixed feelings about WADL. As a systematic way of documenting an API, and something from which you can produce apidoc html or whatever it's fine. As something applications will read _at run time_ it seems a bit strange: the application will be written assuming particular APIs are present and if they are not, or if other APIs do turn out to be present, the application is not likely to suddenly make use of them. An application that did want to cope with differing server capabilities would probably be better off just sending requests and coping with errors. But perhaps an exception to this is an explorer-type application which does want to show all the methods you, the interactive user, could possibly call if you wanted to. That leads me to think it should be something applications can opt in to, if they want to introspect the interface. It does seem wrong to me that applications should need to download over a megabyte of data when it can't really change their behavior. That leaves open the question of how a client ought to call methods, like say https://api.launchpad.net/devel/bzr/?ws.op=searchTasks. It would be ugly to have random client apps hardcode that. (It's also ugly to have, as at present in launchpadlib, them necessarily fetch https://api.launchpad.net/devel/bzr/ first when they don't want to know about the product itself, only its bug tasks.) (Incidentally, I wonder why we don't have eg a bug_tasks_collection_link on products, which seems a bit more in keeping with a REST-ish style.) So it's quite fun and I intend to continue. If someone wants to talk about it I'd like that too. -- Martin ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] notes towards async api clients
On Tue, 25 Jan 2011 12:10:43 +1100, Martin Pool m...@canonical.com wrote:
I had a bit of a go at this over the weekend. It is gratifyingly fast
compared to what I expect to see with launchpadlib clients, just
through doing fewer requests and not unnecessarily blocking on them.
I like that a lot.
The code is in http://launchpad.net/wrested and lp:wrested.
For instance:
./wrestler.py https://api.launchpad.net/devel/bugs/1
To be fair I believe lplib would perform approximately as well if you
just did lp.load('https://api.launchpad.net/devel/bugs/1').
Thanks,
James
___
Mailing list: https://launchpad.net/~launchpad-dev
Post to : launchpad-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] notes towards async api clients
On Tue, 25 Jan 2011 12:10:43 +1100, Martin Pool m...@canonical.com wrote: So it's quite fun and I intend to continue. If someone wants to talk about it I'd like that too. Thanks for this, it's good to see things progressing on this front. I think that this would make a great bottom layer for a twisted LP client library. I think that if we were to merge this and txrestfulclient then we would have the best of both worlds. I think this is a better building-block way to go, but most developers won't want to hardcode URLs everywhere, so putting it together with the higher-level stuff in txrestfulclient (which deals with WADL etc.) would allow developers to work at the level that they want. Thanks, James ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
[Launchpad-dev] notes towards async api clients
We had a chat about what would be desirable in a Twisted-based async
web client api.
Background:
* https://dev.launchpad.net/LEP/WebservicePerformance
We can get inspiration from existing Python remote-object-like libraries:
* http://www.lothar.com/tech/papers/PyCon-2003/pb-pycon/pb.html
* http://twistedmatrix.com/documents/10.0.0/api/twisted.protocols.amp.html
Some things:
* Application startup should not require an http roundtrip (assuming
we have a reasonably current wadl and auth token - true at the moment
in launchpadlib).
* If the client can reasonably be expected to know the URL of an
object ('/bzr', or /bugs/1) then it should just use this, without
doing a redundant call merely to be able to traverse through that
object.
* Anything that does a remote call should be explicit and return a Deferred.
* Therefore, traversing object links so they need to be fetched
separately ought to be something the client explictly asks for
* Nothing should implicitly do more than one remote call: for
instance, after putting an object to the server, we should not
automatically read it back in a separate http call. (It's fine of
course to build higher-level operations that do multiple calls, but
there should be a clear one-to-one layer.)
* Explicit is better than implicit: for net operations, and for
modifications of remote operations.
* Objects that do remote calls should be very obvious. (Perhaps put
them all onto a LaunchpadServer object, passing objects as
parameters.)
* Proxy objects are just value objects, with little magic. (For
instance, we don't unwrap _link fields into magic pointers.)
* Rather than mutating object fields then saving them, we might have
an explicit update operation, which can return a Deferred for its
completion.
* For collections, we will probably deliver them into a consumer
object. This can probably opt-in to see batch boundaries but
otherwise it will get all pages. It will get gotObjects() that gets
the whole batch, and the default can then split them and deliver them
one by one.
* Getting a coherent list across multiple calls is hard; the best
solution is probably to make the server fast enough that it can return
large lists, and secondarily to have higher-level client code that
will heal tearing between batch pages by matching up unique ids.
* If Launchpad adds the ws.expand parameter, we can have a way to
pass it. Rather than this pre-populating attributes that would
otherwise be lazy, it will instead add attributes to the passive
objects. For instance, if you don't ask for expansion you will get
task.bug_link; if you do expand that you will also have task.bug which
is a local reference to another real passive object.
* We could possibly provide a synchronous layer on top of this.
* This seems to be able to accommodate the
https://dev.launchpad.net/LEP/WebservicePerformance approach of
first giving you a huge set of URLs, then later doing separate batched
calls to get the details.
* Although the client code clearly shows when it's doing network io,
it doesn't need to expose exactly what HTTP method is being used,
which is an irrelevant detail. (Case in point, some conceptually-read
methods might use POST to send large parameters.)
So, sketchy, using inlineCallbacks:
me = yield lp.get_object(lp.me)
# gives you back a fully-populated passive object
print me.full_name
class ConsumeAndPrintBugs(object):
def objectReceived(self, bug):
print bug.title
done = yield lp.get_collection(me.assigned_bugs_link, ConsumeAndPrintBugs())
me = yield me.update(full_name='Martin poolie Pool')
urls = yield me.callRemote('getArchiveSubscriptionURLs')
# make a url without fetching anything
bzr_url = lp.get_object_link(lp.Products, 'bzr')
# now get it
class PrintTaskTitles(object):
def objectReceived(self, task):
# need another roundtrip to get the bug
d = lp.get_object(task.bug)
d.addCallback(self.print_bug)
def print_bug(self, bug):
print bug.title
bzr = lp.get_object(bzr)
lp.call_collection_method(bzr, 'searchTasks', PrintTaskTitles())
# we can have a convenience method that folds up all the collection
objects into one list, if you don't care to get them individually;
this gives a Deferred producing a list
all_bugs = yield lp.call_collection_method_to_list(bzr, 'searchTasks')
# time passes...
print len(all_bugs) # should be correct; and you now have all the data
--
Martin
___
Mailing list: https://launchpad.net/~launchpad-dev
Post to : launchpad-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
