Re: [Launchpad-dev] brainstorm: cheaper API collection iteration
My understanding, may be way off. Pages in collections are just links, so you can generate what you like. You can generate the bare link to the collection to get the first page, and then when that page is returned it would say next_page_link: https://api.launchpad.net/devel/some_collection?start_key=endkey_of_this_collection then when the next page is needed launchpadlib simply hits that URL. Therefore I think this would be fairly straightforward to change in the current webservice. It also looks as though just changing lazr.batchnavigator will get most of the way there, so it may be that fixing the web UI fixes the API too. In general, this is right. lazr.restfulclient follows the URL it finds in next_collection_link without caring what that URL is. launchpadlib doesn't mess with the collection links at all, and lazr.restfulclient never uses previous_collection_link. So if you come up with a better way to paginate records, you can just change next_collection_link and the client will adapt. I know of two exceptions, both optimizations in the code that handles slices (_get_slice() in lazr.restfulclient). 1. If you ask for a slice like launchpad.bugs[:76], lazr.restful gets the first page (which happens to have 75 entries), and then this code runs: if more_needed 0 and more_needed first_page_size: # An optimization: it's likely that we need less than # a full page of entries, because the number we need # is less than the size of the first page we got. # Instead of requesting a full-sized page, we'll # request only the number of entries we think we'll # need. If we're wrong, there's no problem; we'll just # keep looping. page_url = self._with_url_query_variable_set( page_url, 'ws.size', more_needed) So, we take next_collection_link, and we hack ws.size to only give us (in this case) one additional entry. We don't need another 75 entries, we only need one. I think this will continue to work no matter what next_collection_link looks like, so long as ws.size continues to work. Worst case, we can simply remove the optimization. 2. When you ask for a slice like 'launchpad.bugs[70:200]', this code runs: # No part of this collection has been loaded yet, or the # slice starts beyond the part that has been loaded. We'll # use our secret knowledge of lazr.restful to set a value for # the ws.start variable. That way we start reading entries # from the first one we want. first_page_size = None entry_dicts = [] page_url = self._with_url_query_variable_set( self._wadl_resource.url, 'ws.start', start) This secret knowledge of lazr.restful would be invalidated by the change. We could stop supporting this syntax in later versions of launchpadlib, or we could just remove the optimization: make lazr.restfulclient load subsequent pages from the beginning until it has 200, and then perform the slice. I did a quick search of Launchpad for references to ws.start and ws.size. It shows up a lot in tests, but that's about it. In particular, it doesn't seem to be used in the Javascript code (though client.js has support for it). I did see some batch navigation code in picker.js. It looks like Launchpad code, not web service code, but it may have to be changed for the same reasons as the web service needs to be changed. Leonard ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Instead of authorizing individual applications against the Launchpad web service, let's authorize the Ubuntu desktop as a whole
We support KDE via Kubuntu and many Ubuntu developers use it; gnome-keyring seems rather specific; can we, as Sidnei suggests, use an abstraction layer to provide the benefits [currently only encryption?] to those environments too? This is an excellent idea. I've asked Benji to implement this before landing his launchpadlib branch. We *have* had credentials for key members of Ubuntu compromised in the past (through them pasting a openid cookie IIRC) and that route remains unaltered. The change that you're doing makes oauth roughly the same as openid cookies. I can see what you're talking about but I'm not sure what exactly happens in these incidents. (Part of this is likely my lack of knowledge about OpenID.) Can you give more details about this compromise, or point me to someone who can? Did someone paste a URL containing their OpenID secret, or the cookie itself? (Like from an HTTP dump?) I'm curious about this because I hadn't really considered _accidental_ security breaches, even though those are the most common kind. I'm going to give this some thought and see if there are any easy countermeasures. This list may not be the best place to have this discussion, but I'd like to offer three preliminary ideas: 1. My change doesn't change our use of OAuth very much. Once it hits the network, a DESKTOP_INTEGRATION token is exactly the same as a WRITE_PRIVATE token. The procedure for authorizing a token is exactly the same as before; it just happens less often. And I don't think the distinction between a file on disk and a key in the GNOME keyring is relevant here. So if we have a problem, I don't think it's a _new_ problem. 2. I have no idea if you can revoke an OpenID cookie, but it's easy to revoke an OAuth token if you accidentally publicize the secret. It's also easy to get a new token to replace the one you revoked. 3. We never put a token secret into a URL. We frequently put a token *key* into a URL, but it's the key to a *request* token. The main attack here is that Alice intercepts Bob's +authorize-request URL, waits for him to sign the request token, and then tries to exchange the request token for the signed access token, before Bob gets around to doing the exchange himself. This is a real attack, but it's not new. Perhaps we should set a vision for what we'd like to achieve here: Single sign on (web, casual apps, daemons like Ubuntu One), persistent credentials (if the user wants - I would!), easy authorisation of new apps (single clicky clicky dialog, in some trusted computing base), timeouts (defaulting on for high privilege levels), powerful per-app controls (like the openid prompt which says what is being disclosed), and only one protocol needed (rather than two, managed separately but with equivalent issues). Let's do this, but let's keep the vision classified in tiers or something. Persistent credentials is trivial. Only one protocol may not be 100% possible, but the possible part is worth doing. Trusted computing base seems like a project we could devote an entire Ubuntu release to, assuming it's possible at all. If we're aiming so high that trusted computing base is actually on the table, then I'll stop shooting down ideas that assume it, and start putting them in the appropriate tier. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Instead of authorizing individual applications against the Launchpad web service, let's authorize the Ubuntu desktop as a whole
After talks with the USSO desktop team, Kees Cook, Ubuntu developer Didier Roche, and Kubuntu developer Johnathan Riddell, I'm back to update this thread. I know a lot more about the GNOME keyring, I've made some tweaks, and I think I can present my design in a more convincing way. Restatement of the status quo = On the server side, Launchpad credentials are associated with a consumer key (the name of the application that requested the credential). Each credential has an associated access level which limits what the application can do under your name. On the client side, launchpadlib's login_with() method caches credentials unencrypted in your ~/.launchpadlib/[host]/credentials directory. The get_token_and_login() method can be used to get a token without writing it to /credentials, but most applications use login_with(), and most of the ones I've seen use get_token_and_login() also write the token to disk somewhere. This setup works great when integrating other websites into Launchpad, because you don't have any control over (eg.) Facebook, and because third-party websites don't all share a data store. Although third-party websites can use launchpadlib, they can't use login_with(), which is designed for desktop use. This setup doesn't make much sense when integrating desktop applications running in a GNOME environment. You have control over the environment. Any application running as you can easily access your credentials and act as the union of all your OAuth tokens. Not only that, login_with() stores the tokens unencrypted on disk. Unless you've enabled home directory encryption, if your hard drive is stolen, all of your credentials will be compromised. Restatement of my design Now I'll restate my design as a two-step process. I hope I can make at least the first step sound uncontroversial. Step 1: Store credentials in the GNOME keyring -- The first improvement is to change login_with() to start storing Launchpad credentials in the GNOME keyring (if available), with disk storage as a fallback. This will prevent the stolen-hard-drive attack. With Launchpad credentials in the GNOME keyring, a suitably paranoid end-user can also take countermeasures against certain other attacks by moving their Launchpad credentials out of the login keyring (where they will be stored by default) and into a keyring for which they've set an idle timeout. You can do this from the Passwords and Encryption Keys desktop accessory. Idle timeout sidebar ~~~ Setting an idle timeout means that if you unlock a keyring and then walk away from your computer for N seconds, the keyring will automatically lock back up. The next time any application wants access to one of the keys in that keyring, you'll have to type your unlock phrase again. This stops an attack where someone stops by your running computer when you're not around, and snoops on your passwords. It also offers a little protection against malware that tries to dump an unlocked keyring: if you suddenly get a password dialogue out of nowhere, you'd do well to be suspicious. It doesn't protect against a keylogger. If you set the idle timeout to 0 seconds, you will have to type your unlock password *every* time an application wants access to *any* key in that keyring. If you're careful, this will give you complete protection against malware that tries to dump a keyring. It still won't protect against a keylogger, though. Step 2: Authorize the desktop, not the application -- Setting an idle timeout offers you some additional options for security. But even if you set your idle timeout to zero, you have no way of ensuring that the 'apport' credential is only used by the apport application.[Footnote 1] Any authorized application can access your entire keyring. The second improvement is to acknowledge this, by relaxing the requirement that each desktop app have its own individually authorized OAuth token, while maintaining that requirement for integrated websites. This should have the side effect of solving the problem I'm actually trying to solve--getting third-party developers to accept the token authorization process instead of trying to hack around it. The desktop authorization will be done through a one-time browser open, as described in my earlier email. Desktop authorization can be done permanently or on a time-limited basis, and revoked at any time from the Launchpad website. Token expiration or revocation will cause another browser open. (About) as good as it gets == Why am I basing this design around a system (the GNOME keyring) that has obvious shortcomings? Because what we're doing now is worse, and because there's nothing lying around that's better than the GNOME keyring. It's not hard to come up with a better system than the one I propose. But my
[Launchpad-dev] Instead of authorizing individual applications against the Launchpad web service, let's authorize the Ubuntu desktop as a whole
Introduction: Can We Do As Well As Ubuntu One? == Let's say you boot up your new Maverick system, fire up Rhythmbox, and decide to buy some music from the Ubuntu One store. What happens? An application window pops up on top of Rhythmbox, asking you for your Ubuntu SSO login and password so that you can get access to Ubuntu One. Being a new user, you don't have an Ubuntu SSO account, so you get a chance to create one. Once you have an account, you accept the Ubuntu One TC, and the application window secretly uses your authorization to store a token on your computer. This token allows applications to use Ubuntu One under your name. It's stored in the GNOME keyring. You never see the token, but it's there, and usable by other applications. You buy your music, and then you get to work, writing some notes in Tomboy. Once you're done, you decide to sync your notes to the cloud using the Ubuntu One integration features of Tomboy. What happens? Well, nothing happens. The Ubuntu One token is available to *any* application that uses Ubuntu One. You authorized Rhythmbox, but Tomboy can use the token too. Tomboy grabs the token you created from Rhythmbox from the GNOME keyring and syncs your notes. The Control Panel, desktopcouch, cloudserver, and every other application that uses Ubuntu One can grab this token from the GNOME keyring, without asking you, and access your Ubuntu One account. Uh-oh! Rhythmbox just crashed while it was playing your music! Now a dialogue box pops up, asking you if you'd like to report a bug. You would like to report a bug, so you go through a process that's pretty similar to the process you went through when you wanted to buy some music: A browser window pops up, and you're asked to log in to Launchpad, the site that hosts the bug reports. You're able to log in using the Ubuntu SSO account you just created, and then you get a message like this: The application identified as apport wants to access Launchpad on your behalf. Will you allow apport access to your Launchpad account? * Yes. * No, thanks. You click Yes, and Launchpad grants your computer a token that lets apport use Launchpad on your behalf, without asking you every time. This token is stored in the GNOME keyring, right next to your Ubuntu One token. But of course you don't see the tokens. You just see that apport has done its job and now there's a Rhythmbox bug in Launchpad with your name on it. Now you're curious about Launchpad, so you poke around the site. You decide that Launchpad is the perfect site to host the little open source project you're working on, so you start a project and import your source tree. And you discover there's an application called Ground Control, which lets you integrate your Launchpad work into your desktop. You install Ground Control, start it up, and... your browser opens again. The application identified as Ground Control wants to access Launchpad on your behalf. Will you allow Ground Control access to your Launchpad account? * Yes. * No, thanks. What happened? How come Tomboy didn't need to ask you for permission to access your Ubuntu One account, but Ground Control had to ask you for permission to access your Launchpad account? This is a question that our users have been asking (albeit not in this form) since the Launchpad web service was first released. We don't have a good answer. It turns out there *is* no good answer. It's time to change this. Authorize the entire desktop at once I have a branch in review that needs some discussion. Here's the merge proposal: https://code.edge.launchpad.net/~leonardr/launchpad/rename-grant-permissions I'm changing the social norms around Launchpad's OAuth token authorization protocol, *as it's used on the Ubuntu desktop*, to make it more like the system used by Ubuntu One. Instead of authorizing individual applications (apport, Ground Control, etc.), the end-user will authorize the Ubuntu desktop itself. The protocol itself is unchanged; I'm simply reducing the number of times the end-user has to go through it. Robert pulled the brake on my branch because he has legitimate concerns about my process: I didn't use this list to hash out the design, and the design has changed radically over the past couple months. Fair enough. Let me present the stable design in public, tell you who I've discussed it with, and try to convince you that it's good on the merits. The ultimate purpose of this branch is to stop third-party developers from writing horrible hacks that bypass the OAuth token authorization protocol. These people are my customers. I can't stop them through technical means, any more than music companies can stop *their* customers from sharing songs with each other. But I can get my customers back on my side, by improving the experience of the legitimate alternative. I went through a couple iterations of a design in consultation with these
Re: [Launchpad-dev] Instead of authorizing individual applications against the Launchpad web service, let's authorize the Ubuntu desktop as a whole
Thanks for your responses. I have a couple follow-up questions for Kees and Robert. First, Robert, one of your concerns is with making the web service safe for privileged users to use. I believe it's possible to configure the GNOME keyring so that the keyring is never permanently unlocked. Any attempt to access a key from the keyring will require that you enter your keyring password. If you didn't just do something that requires accesss to one of your keys, you don't enter your password. I imagine it would be very annoying to live with this kind of setup. But that's the setup I'd live with if I were totally serious about protecting my SSH keys and OAuth tokens from malware. How close would this come to making you feel that your OAuth tokens were safe? And, Kees, how much would this actually improve security? Using sensibly short timeouts for session tokens as a default is good. I'm not opposed to making forever available for those that want it, but it should probably not be the default. We do plan to add timeouts for session tokens in two circumstances. (These are both mentioned in the design document I linked to last time, http://pastebin.ubuntu.com/499131/) Here's the one that pertains to my last two questions: There are a couple actions (uploading SSH/GPG keys) that are so fraught with peril that we currently don't publish them on the web service at all. We plan to publish these operations but only make them accessible through time-limited OAuth tokens that have a new permission level. Normal desktop integration permission will not be good enough to use applications that upload new SSH keys, like Quickly. You will have to do the browser dance and get a brand new time-limited token every time you want to upload a new SSH key. Now, suppose we take existing web service operations like toggle whether this bug is a security bug or toggle this bug's public flag. Things that we're kind of worried about, but not so worried that we were afraid to publish them at all. And we put those in the same category as upload a new SSH key. A category that forces you to do the browser dance every 15 minutes as long as you're doing that kind of thing. And added some code that kept those keys from being stored in the GNOME keyring, making it more difficult for malware to grab them. Robert, how far would *this* go towards addressing your concerns about making the web service safe for privileged users? Think about what you're most afraid of when it comes to malware acting as a privileged user. Are you afraid of things that only privileged users tend to do? Things like uploading SSH keys and changing the security status of bugs? Or are you afraid of things that everybody does all the time, but that can be especially damaging when privileged users do them? This brings me to my third question. Imagine that the existing system really did work the way I imagined it could, before I found out about the GNOME keyring. Imagine that when you grant an OAuth token to apport, apport is the only application that can see or use that token. You have to authorize every one of your desktop apps individually. It works like a dream. Here's the problem: my developers will not put up with that. They don't put up with it now, and they won't suddenly start putting up with it if the security benefits ever become real. Pretty much every third-party developer (and at least one internal developer) has responded to our OAuth token authorization protocol by hacking around it, creating some native-GUI way of asking the user for their Launchpad username and password, so that their users don't have to do the browser dance. That's the whole reason I'm trying to change this system--my customers simply refuse to make their users do the browser dance before using an application. Making the tokens time-limited, or making the permissions more fine-grained, will just make things worse, as developers try harder to find workarounds. The system we have now, for all its shortcomings, is intolerable to my developers. I'm trying to find a system that's at least as secure and that can get developer buy-in. My proposed design seems congruent with existing Ubuntu best practices when it comes to your SSH keys and the data you've got stored in Ubuntu One. The only realistic alternative I can think of is to enforce draconian rules like we won't let your program into Ubuntu unless it does the browser dance properly, but you can tell your users to blame us for the bad user experience. And now, finally, my third question. Suppose we were to move the scary activities into a time-limited token, a token that required a browser dance every fifteen minutes but which ordinary users would rarely need to acquire. Would it then be below the scary threshold to manage your ordinary activities on the web service with a single, desktop-wide token? (And, to restate my second question, would there be enough non-scary activities left that an ordinary user would not find
Re: [Launchpad-dev] tuesday - performance day!
Now, I'd like to ask for some advice. There seem to be three ways of
fixing this:
- (somehow - I don't know the magic) tell lazr.restful to skip some
attributes (e.g. karma, coc_signed etc) for representations.
https://bugs.edge.launchpad.net/launchpad-foundations/+bug/251284 -
makes note of things being removed from representations, but I don't
know whats involved in doing that (yet). Or I may be misunderstanding
and actually its an all-or-nothing solution. Leonard, if you could
comment here (or there) and help enlighten me, I'd really love that :)
lazr.restful will publish an attribute in representations only if it is
tagged with the exported() decorator. If you remove the exported()
decorator from an attribute that attribute will instantly disappear.
But unless the performance problem is catastrophically bad, you want to
un-export an attribute only in the latest version, so as to preserve
backwards compatibility. For that, let me quote my earlier email to
Julian:
---
And here's how to get rid of queue_status in the latest version while
maintaining it in the old versions.
queue_status = exported(
Choice(
title=_('Status'),
vocabulary=BranchMergeProposalStatus, required=True,
readonly=True,
description=_(The current state of the proposal.)),
('devel', dict(exported=False)))
---
So you call exported(), but then you take it back, as of a specific
version of the web service.
Leonard
signature.asc
Description: This is a digitally signed message part
___
Mailing list: https://launchpad.net/~launchpad-dev
Post to : launchpad-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] How do I support older API calls when changing the underlying model?
On Mon, 2010-07-26 at 14:55 -0400, James Westby wrote: On Mon, 26 Jul 2010 12:16:48 -0400, Leonard Richardson leonard.richard...@canonical.com wrote: @operation_removed_in(devel) @operation_for_version(devel) Do these need to be updated next time devel is snapshotted, to create 2.0 or whatever it will be? Yes, exactly. When we snapshot a release we'll go through and replace all the devel with the next version name. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] How do I support older API calls when changing the underlying model?
I'm in the situation where I'm wanting to split up a status field on the
branch
merge proposal into two. However there is an existing API that is used to
fetch merge proposals.
OK, let me quote the part of the system I think you're changing.
From IHasMergeProposals:
@operation_parameters(
status=List(
title=_(A list of merge proposal statuses to filter by.),
value_type=Choice(vocabulary=BranchMergeProposalStatus)))
@call_with(visible_by_user=REQUEST_USER)
@operation_returns_collection_of(Interface) # Really
IBranchMergeProposal.
@export_read_operation()
def getMergeProposals(status=None, visible_by_user=None):
From IBranchMergeProposal:
queue_status = exported(
Choice(
title=_('Status'),
vocabulary=BranchMergeProposalStatus, required=True,
readonly=True,
description=_(The current state of the proposal.)))
How do we continue to support the older methods when changing the underlying
model? Should we even try?
Sometimes it's possible to determine that no one is using the old
version of a named operation, so you can change it without pain, or that
only a couple people are using it and you can coordinate a change with
them.
However, this change doesn't look too difficult, so let's try
maintaining the backwards compatibility.
I'm wanting to break the queue_status member into merge_status and
review_status. I was also considering re-using the
BranchMergeProposalStatus enum for the merge_status with changed values.
However I feel that this is more likely to break something.
I'm at a bit of a loss to know where to start with supporting the old method
where the method directly gets values from an enumerated type I'm changing.
Should I instead of reusing the BranchMergeProposalStatus enum, create two
new
ones (instead of just one new one)?
You should. All of the versions of the web service coexist in a single
application. Launchpad now has three enumerated types: one old one and
two new ones. They all need to be present.
How do I have a method that is only supported in an old API version?
With the @operation_removed_in() decorator. Here's the old method,
renamed to getMergeProposalsOLD internally, published as
getMergeProposals externally, and removed in the latest version of the
web service:
@operation_removed_in(devel)
@operation_parameters(
status=List(
title=_(A list of merge proposal statuses to filter by.),
value_type=Choice(vocabulary=BranchMergeProposalStatus)))
@call_with(visible_by_user=REQUEST_USER)
@operation_returns_collection_of(Interface) # Really
IBranchMergeProposal.
@export_as(getMergeProposals)
@export_read_operation()
def getMergeProposalsOLD(status=None, visible_by_user=None):
Here's the new method, introduced in the latest version of the web
service.
@operation_parameters(
merge_status=List(
value_type=Choice(
Vocabulary=BranchMergeProposalMergeStatus,
review_status=List(
value_type=Choice(
Vocabulary=BranchMergeProposalReviewStatus)))
)
@call_with(visible_by_user=REQUEST_USER)
@operation_returns_collection_of(Interface) # Really
IBranchMergeProposal.
@export_read_operation()
@operation_for_version(devel)
def getMergeProposals(merge_status=None, review_status=None,
visible_by_user=None):
This way the 'getMergeProposals' named operation will have the existing
behavior in 'beta' and '1.0' (because it's mapped to
getMergeProposalsOLD()), and new behavior in subsequent versions
(because it's mapped to getMergeProposals()).
And here's how to get rid of queue_status in the latest version while
maintaining it in the old versions.
queue_status = exported(
Choice(
title=_('Status'),
vocabulary=BranchMergeProposalStatus, required=True,
readonly=True,
description=_(The current state of the proposal.)),
('devel', dict(exported=False)))
Leonard
signature.asc
Description: This is a digitally signed message part
___
Mailing list: https://launchpad.net/~launchpad-dev
Post to : launchpad-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Quickly and Launchpad
Agree. My argument about client-side security is that there is little point forcing the user to do an interaction through a web browser rather than through any other client-side program. I have completely capitulated on this point. Even so, once Launchpad becomes an OpenID consumer, a web browser will be the only tool general enough to complete the Launchpad authentication process, since that process might involve a redirect to Livejournal or Facebook and a trip through whatever HTML forms they throw at you. With the desktop credential-granting service, you'll go through the web browser only once, to create a GRANT_PERMISSIONS OAuth token for the credential-granting application itself. The credential-granting application then uses the GRANT_PERMISSIONS token to grant additional tokens _using the web service itself_, without going through the Launchpad authentication process. 2. If we really consider scenario 4 and scenario 1 equally insecure, we might as well go for the superior usability of scenario 4. Sorry, I'm not sure what API to create a new OAuth token means. API on the desktop credential-granting service? I don't think you should be able to use a less-privileged token to make a WRITE_SENSITIVE_DATA one. Yes, API to create a new OAuth token means a new feature of the Launchpad web service which will allow the credential-granting service to create new OAuth tokens without making the end user re-authenticate with Launchpad. The launchpadlib code would look something like this: new_token = launchpad.me.oauth_access_tokens.new( consumer=consumer, access_level=WRITE_PRIVATE) This action requires the GRANT_PERMISSIONS access level. GRANT_PERMISSIONS is, by definition, the most privileged access level. The only application that should ever have GRANT_PERMISSIONS is the desktop credential-granting service. I like the idea of making WRITE_SENSITIVE_DATA tokens expire so that old tokens found on backups or abandoned drives won't work anymore. But I don't think that will help, because typically the GRANT_PERMISSIONS token will still be available, and an attacker can just use it to create a new WRITE_SENSITIVE_DATA token. The two solutions to this problem: 1. Make the GRANT_PERMISSIONS token also expire after a while. 2. Prohibit the GRANT_PERMISSIONS token from creating WRITE_SENSITIVE_DATA tokens, even though it can create other kinds of tokens. Both of these solutions require opening the web browser more often, at seemingly random intervals. Not, at this point because the web browser is more trustworthy than the permission-granting app, but because the web browser is the one that can negotiate a redirect to a Facebook login. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Quickly and Launchpad
I can write some code that simply reads the token from ~/.launchpadlib/credentials/quickly and attempts to use it to add a new GPG key if you are in ~ubuntu-core-dev, and so gets a secret key that can sign packages that can end up on millions of machines. A very tagetted attack like that would hard to prevent, because there is clearly a lot of desire. However, as one of the people that could be exploited in this manner I am wary of anything that makes it possible. OK, we can make WRITE_SECURITY_SENSITIVE a time-limited token that must be reacquired, say, once an hour. Or we can make it a one-shot token that must be acquired anew every time you want to perform one of these actions. However, such a system destructively interferes with our plan for a desktop credential management app. Let me explain. I'm now working on adding *another* OAuth access level, GRANT_PERMISSIONS. This access level is only officially available to the credential management app. The point of GRANT_PERMISSIONS is to make it possible to create new OAuth tokens without opening the user's web browser. The credential management app will ask the end-user to grant or deny access within a native GUI window rather than within the web browser. You point out that it's easy for a malicious application to get another application's OAuth credentials. This is true whether the credential's access level is WRITE_SECURITY_SENSITIVE or GRANT_PERMISSIONS. (Hiding the GRANT_PERMISSIONS credential somewhere other than the file system won't help: in the worst case, a malicious application can impersonate the Launchpad credential management app over DBus.) If you have GRANT_PERMISSIONS, you can grant yourself any other level of access, including WRITE_SECURITY_SENSITIVE. It's no secret that I think the desktop credential management app, although superior from a UI standpoint, is insecure. Up to this point the counter-argument has prevailed that malicious client code on an Ubuntu desktop is rare, so we shouldn't worry about it. I think this counter-argument has an additional premise that has just been revealed: malicious client code on an Ubuntu desktop is rare, *and if it does exist, the worst it can do is screw up your own system/Launchpad account*. With GRANT_PERMISSIONS plus the ability to upload GPG keys, once malicious code gets on an Ubuntu system it can easily infect thousands of other systems. The obvious solution is to prohibit GRANT_PERMISSIONS from being used to grant WRITE_SECURITY_SENSITIVE. This means that WRITE_SECURITY_SENSITIVE cannot be obtained through the desktop credential management app. Every time you want WRITE_SECURITY_SENSITIVE, you will have to open the user's web browser and tell them: Quickly wants to do a very dangerous thing with your Launchpad account. Because the thing Quickly wants to do is so dangerous, 1) you need to make doubly certain that you trust Quickly with your Launchpad account, 2) you need to authorize Quickly in your web browser because we can't even trust the Launchpad credential manager in this situation. Since WRITE_SECURITY_SENSITIVE is either a time-limited or a one-shot authorization, the end-user will find themselves authorizing in the web browser every time they want to do something dangerous, plus authorizing in the Launchpad credential app every time they start using a new app, plus using the web browser to authorize the Launchpad credential manager itself. To summarize, a totally new user will need to do the following to upload a GPG key to Launchpad: 1. Authorize the Launchpad credential manager using the web browser. 2. Authorize Quickly for WRITE_PUBLIC using the credential manager. 3. Authorize Quickly for WRITE_SECURITY_SENSITIVE using the web browser. 4. Upload the SSH key. Or, if WRITE_SECURITY_SENSITIVE is time-limited rather than one-shot, you could implement this: 1. Authorize Quickly for WRITE_SECURITY_SENSITIVE using the web browser. 2. Upload the SSH key before the credential expires. (In the second case, once the credential expires, the end-user will completely lose the ability to use Quickly. Next time the end-user wants to use Quickly, they will need to once again authorize Quickly for WRITE_SECURITY_SENSITIVE, or else authorize the Launchpad credential manager and then use the credential manager to authorize Quickly for WRITE_PUBLIC .) Is this acceptable? Is there a better solution? Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Quickly and Launchpad
That is true but to me the main point is that if there is malicious code running within your desktop machine, you have bigger problems than your Launchpad account. Sure; the question I'm trying to answer is whether malicious code running on your desktop machine also means bigger problems for lots of innocent people. Can you unpack the logic there? Do you mean that if malicious code gets onto an Ubuntu system of a user who can write to the main archive or a popular PPA, it can propagate to thousands of other machines. That is true, but orthogonal to whether there is an API to manipulate credentials. I'm not sure what you mean by API to manipulate credentials. I don't know whether you mean API to upload a new GPG key or API to create a new OAuth token without user input. Forgive the verbiage that follows; I want to be really precise about this. Let's take Bob to be a user who can write to the main archive of a popular PPA, and let's stipulate that Bob has malware on his computer. OK, what are the scenarios? 1. The current scenario. There is no API to upload a new GPG key and no API to create a new OAuth token. Can the malware on Bob's computer reproduce itself in the PPA? I guess, but it would have to include a keylogger (to sniff the passphrase to Bob's existing GPG key) or something fairly sophisticated. 2. There is an API to create a new OAuth token, but no API to upload a new GPG key. This is identical to scenario #1. The malware can exploit the API to create new OAuth tokens for itself, but it can't use the API to do what it really wants to do--reproduce. 3. There is an API to upload a new GPG key, but no API to create a new OAuth token. Now an exploit is trivial. The malware can grab Bob's existing OAuth token, impersonate one of his other applications, and upload a new GPG key. We can stop this from working by making upload a new GPG key require a single-purpose, one-use OAuth token. 4. There is an API to upload a new GPG key, and an API to create a new OAuth token. Now it doesn't help to require a single-purpose, one-use OAuth token, because the malware can create new OAuth tokens whenever it wants. The only way to plug the trivial exploit is to prohibit create a new OAuth token from creating the kind of OAuth token that can be used to upload a new GPG key. If we take the hard line that once there's malware on your system, you're screwed, then all four of these scenarios are identical--the difference between fairly sophisticated exploit and trivial exploit is not worth considering. I don't know if this is what you mean by [The existence of an exploit] is orthogonal to whether there is an API to manipulate credentials. Is it? If we take this hard line, then there's no reason to make WRITE_SENSITIVE_DATA tokens temporary or prohibit the OAuth token generator from generating them--we're just making the malware's job marginally easier. I think users are likely to need WRITE_SENSITIVE_DATA fairly rarely, so I don't have a problem with requiring these tokens to be acquired through the web browser. But: 1. The reaction to 'tokens through the web browser' has been incredibly negative. Others have spent a lot of time hacking around it, and I've spent a lot of time trying to broker and implement a compromise. Reintroducing the browser at (what will appear to the end-user as) random intervals isn't good for the user experience--you could argue it's worse than our current strategy of using the browser all the time. 2. If we really consider scenario 4 and scenario 1 equally insecure, we might as well go for the superior usability of scenario 4. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Quickly and Launchpad
However, for maverick, we need to go further and help the user to create and get his full environment, which means: - uploading his gpg key (for uploading his package) if he doesn't have one - upload his ssh key (for bzr) - creating one ppa if he hasn't one - this imply make him sign the Code Of Conduct in a simple way. There's no reason these functions can't be published through the web service (except of course lack of person-hours). But, it might be dangerous to publish 'upload GPG key' and 'upload SSH key'. For instance, allowing a web service client to upload a new GPG key could allow a malicious client to create its own GPG key, upload it as yours, and then upload packages 'signed' by you. In a sense this problem already exists--a malicious client can currently post atrocious comments in your name, etc. But this is more worrisome because 1) anything to do with GPG or SSH triggers one's security Spidey-sense, 2) a malicious app that uploads packages in your name can infect other peoples' computers. If we were to implement these features today, they would be available to clients that were granted the OAuth access level WRITE_PRIVATE or WRITE_PUBLIC. But almost every nontrivial web service client needs to write the dataset, which means it needs WRITE_PRIVATE or WRITE_PUBLIC. That leaves quite a lot of scope for abuse. But let's say we create a new level of access WRITE_SECURITY_SENSITIVE. Clients with WRITE_PRIVATE can do everything they can do now, but they can't do especially sensitive things like upload SSH keys. Clients with WRITE_SECURITY_SENSITIVE can do everything a WRITE_PRIVATE client can do, and can also upload SSH keys. A client like Quickly would need to ask for, and be granted, WRITE_SECURITY_SENSITIVE to function properly. If there are any existing activities protected with WRITE_PRIVATE clients, that we want to protect with WRITE_SECURITY_SENSITIVE instead, we could do that, at the cost of breaking existing clients until the end-user granted them the new permission. We can't prevent malicious clients from existing, but we can reduce the amount of trust the end-user needs to put in any particular client. If you grant WRITE_PUBLIC or WRITE_PRIVATE to most of your apps, and only grant WRITE_SECURITY_SENSITIVE to Quickly, then at least you can rest assured that your other apps aren't going to upload fake GPG keys to your account. You only have to worry about whether Quickly is well-audited. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Adapters API
What we would like to do is avoid registry importing from so many of the other apps, to avoid the circular dependencies. To do this we are currently looking at having ISomething interfaces in the apps (bugs, code, soyuz etc.) that define an interface relevant to them. This is standard practice for a lot of things, see IBugTarget, IHasBranches etc. When the issue of circular dependencies first came up, we decided to stumble along until we could use a planned newer version of Zope which would allow us to use some kind of deferred reference. I didn't follow this development very closely, so I don't know if this feature actually made it into Zope, but if so it seems like a simpler solution to your problem than creating a lot of adapters. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Can we generate the apidoc directory at build time?
Since the apidoc directory under version control is empty, can remove it from the repository and generate it at build time? That would eliminate these merge conflicts once and for all. It would solve our accidental deletion problems too. I don't have a problem with this. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] API issue moving branches
While its true that this is a particular concern, many scripts are written without knowledge of it - and for things like hydrazine designed to be single user, putting fingers in the ears and going la-la-la seems to be the best answer at the moment (and thats what it does). The lack of a thread-safe cache has recently caused problems in groundcontrol (a GUI application). Since using a prepopulated cache is the single biggest performance/bandwidth win we have, I'll definitely be taking a look at this soon. Hopefully we can contribute the thread-safe cache back to httplib2. Hopefully the reason httplib2 doesn't already have a thread-safe cache is that most people don't really need one, not that it's especially difficult to give it one. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] API issue moving branches
I don't intend to put you on the spot - that is, if you don't have an answer, its fine. But it would comfort me to know that some of the following are intended to be addressed: - the transaction vs non-transactional model: * for instance: in LPAPI's, how do you create a set of objects that temporarily violate DB constraints, which appserver code can do easily. As a concrete example, rename two objects A and B to be B and A, safely * or call methods on altered objects without saving them (saving them means you can't check you have achieved what you wanted to achieve) * LPAPI's cannot do repeated reads, so any appserver code that depends on that can't really be ported to LPAPI's. - speed I feel like we're going around in circles with more detail on each go-round. I'll re-state the problem and the plan as I understand it: 1. There are enormous conceptual and practical obstacles standing in the way of a fully unified API that is also a web service we're happy using. Huge obstacles! 2. Nonetheless, I would like to try to get through those obstacles with a unified API, because the alternative is to design and implement two parallel APIs. 3. In the cases where we can't get to a unified API, we have a fallback plan: two parallel APIs. This fallback plan is *annoying* and requires a lot of duplication of effort, and refactoring that doesn't give you the satisfying feeling you ought to get when you refactor something. But that's what it means to have two parallel APIs. Look at it in terms of manpower. The web service team is me. One person. Not only am I responsible for the lazr.restful and lazr.restfulclient frameworks, I have de facto responsibility for the concrete instantiation of those frameworks: the Launchpad web service itself. The *only way I can survive* is to ensure that the complexity of the Launchpad web service remains a very small fraction of the complexity of Launchpad. If we start writing parallel APIs, the complexity of the Launchpad web service will start to approach the complexity of Launchpad. This will not scale. This is why I am so obsessed with finding solutions that work by reducing the complexity of the Launchpad web service and/or increasing the complexity of lazr.restful. * It takes 13 seconds to log into launchpad, find out that there are some pending merges for bzr and quit hydrazine again. This may not seem like a long time, but really - it is, for something that should be a single HTTP request [I say that because in other web services API's *it is* a single HTTP request]. 13 seconds is an INCREDIBLY LONG TIME. It is TOTALLY UNACCEPTABLE. Let's figure out how to fix it. The first step is to profile. I added timing code to lazr.restfulclient and wrote come code that retrieves the pending merges for bzr, basically this: Launchpad(...).projects['bzr'].getMergeProposals() The results: Request for https://api.edge.launchpad.net/1.0/bzr took 0.82 seconds. Request for https://api.edge.launchpad.net/1.0/bzr?ws.op=getMergeProposals took 3.03 seconds. OK, there are two HTTP requests, not one. Fortunately, the first request is unnecessary: it happens because you can't get a reference to launchpad.projects['bzr'] without making an HTTP request to /bzr. Now, we've already solved this problem for collections--note that we were able to get a reference to launchpad.projects without making an HTTP request to /projects. Similarly, we can have launchpadlib return a shim object that only requests /bzr when you try to access some data specific to that project. If you just want to invoke a named operation on the project, that request will never be made. The downside of this solution is that if you write launchpad.projects['nosuchproject'], you won't find out right away the project doesn't exist. You'll only find out when you try to access project.owner or invoke project.getMergeProposals(). This seems like an acceptable price to pay for not making that extra request. I filed bug 583318 to deal with this. We can implement this solution or not, but I wouldn't say it has anything to do with the server-side design of Launchpad. It's a problem with the way the client translates the programmer's desires into HTTP requests. The getMergeProposals call itself is pretty slow on the server-side, but optimizing it shouldn't require changing the API. I don't know what launchpadlib setup you used to get your 13 seconds number, but it was either a new setup or an old setup. Let's say it was a new setup. You made two HTTP requests and it took 13 seconds (due to latency, that's longer than it took me to make the same two requests). With an old setup, you would have made four HTTP requests and with those latency numbers it would have taken 25 seconds or even longer. I've spent the past month or so improving performance and this is the result. You may have already seen https://dev.launchpad.net/Foundations/Webservice/Performance but it has detailed information and measurements
Re: [Launchpad-dev] API issue moving branches
I think you may be going a little far in assuming that he would have been making 2 http requests in a new setup. That requires using a shared cache, which is unacceptable to some applications without writing their own cache implementation right now. (The alternative is fixing the bug in httplib2 or lazr.restfulclient and waiting for it to be widely deployed.) Can you elaborate? I'm not sure which bug you're talking about. I know that shared caches are problematic when you're running launchpadlib in a multithreaded environment. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] API issue moving branches
And there is the issue I think: what makes a great web services API is fundamentally different to what makes a great in-app-server API. The latency and bandwidth constraints are radically different. I think one of three things will happen when we try refactoring a given part of the data model. 1. It works fine. There is no fundamental reason why the internal API has to be different from the public API. For small things like setStatus, I think we will find almost all the time that we can refactor the code into one superior API. (Apart from backwards compatibility problems.) 2. The refactoring succeeds, but some unimportant part of the resulting data model cannot be made efficient or comprehensible enough to justify putting it in the web service. For instance, we may know that Launchpad will run a certain inefficient query very rarely, but if we publish it we have no control over how often it's run. In this case, we would proceed with the refactoring and not expose the less important part. The public data model will be a subset of the internal data model. We know we need special features to deal with the greater latency of the web service, but I'm not convinced these features have anything to do with specific data models. The ability to group objects into ad hoc collections and perform operations on the collection (or however we want to implement this) can be implemented in lazr.restful. We shouldn't need to implement this ability separately for bugs and for people. 3. The refactoring fails. For complex systems like the HWDB, it may be possible to design an internal data model _and_ a public data model, but fundamentally impossible to make the same design serve both purposes. In this case we would need to retreat from the annotation-based model and create our own Entry classes. I don't think this is incredibly likely, but life is complicated, so who knows? We have this option. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] API issue moving branches
I find this quite hard to do since the API exposes our model objects. I'd love to have a separate layer, like we have with the browser. In theory, you are 100% free to write your own Entry classes instead of having them generated from annotated model objects. As an example, the web service defined in lazr.restful doc/webservice.txt creates its Entry classes from scratch. But, we would rather not have two different data models (one for internal use, one for the web service). We'd rather have one, and when we first designed the web service we had a specific policy for making sure we only had one. We developed the annotation technique to make it really easy for the Launchpad developers to publish a web service for their field of responsibility, without having to learn a lot about web service design or make a lot of changes. The price we paid was a coupling of the web service model to the underlying data model. There's no free lunch. You either put some work into designing your web service, or you get a web service that reflects the work you already did. The original idea was that when we ran into a mismatch between the underlying data model and the web service data model, we would refactor the underlying data model. We would think to ourselves: design X is going to confuse our users; it would be nice to publish design Y instead, and then we'd start using design Y _everywhere_. Because why should we have a confusing design when our users get a nice design? The web service would be a way to make piecemeal improvements to our underlying design. Yeah, the named operation setTarget(package, project) is confusing. But the internal method setTarget(self, package, project) is also confusing, in exactly the same way! If we leave it in place internally and patch over it for the web service (which is what we talk about when we talk about annotations), we're letting our users have the good stuff but we're still using the not-so-good stuff internally. It sounds like we are approaching the point where we either need to start refactoring, or decide that the refactoring idea won't work after all and we need to fall back to writing our own Entry classes. In this particular instance (setTarget) I think we can resolve the problem with, at most, a minor change to the underlying data model. But let's take another example. When I look at the HWDB data model (as published through the web service) I have absolutely no idea what is going on. If I wanted to write a web service script using the HWDB it would take me days just to get started. I'm sure the domain experts know what is going on, and I'm also sure that if they put some work into client use cases they could come up with a public-facing data model that was simpler. At this point, I'd suggest we refactor the existing HWDB data model into the simpler data model, rather than have to keep two different data models in our heads. Why not? It's a lot of work, but not that much more work than writing a whole other data model, and at the end our application will be better. Because the public-facing data model is less efficient? That's a reason not to publish at all--it's a bad idea to publish an inefficient data model and hope our users don't use it too much. Are the two data models just totally irreconcilable for some other reason? In that case, it probably makes more sense to go ahead and create our own Entry classes. I hope it doesn't feel like I'm suddenly springing this on you. This has been our plan for quite a while, but up to this point we've been focusing on creating a web service with a solid foundation that publishes all the Launchpad artifacts. Now is a good time to start talking about the best way to *improve* the design of the web service. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] API issue moving branches
I agree with Francis, but I'd also like to add that it seems hard to expose 'never both' over API's AIUI - that is, setTarget doesn't permit both package and project to be set, but individual fields would permit that. If the individual fields have setTarget() as their mutator, then any attempt to set both 'package' and 'project' would fail, just as any attempt to invoke setTarget(package=foo, project=bar) would fail. Any attempt to set 'package' when 'project' is already set would implicitly clear out 'project'. I take Francis's general point that we should think about the API that will feel most natural to the end-user, rather than publishing our internal API directly. I also take his specific point that having a single, polymorphous 'target' field two fields named operation. You can set 'target' to a package or a project, if you need to see what kind of object the target is you can GET it, if you want there to be no target you can set a single field to None. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] API issue moving branches
Not surprising, but how can we tell the restfulclient that the url for the entry has changed? Named operations don't really have hooks for this kind of thing. OTOH if you set a regular field (possibly through a mutator) and that happens to change the object's URL, lazr.restful knows to send a redirect to the new URL. I would like to get rid of that named operation altogether if possible. setTarget() is a method that sets _either_ project _or_ source_package. Both of these are normal fields of the branch, and if you made them directly settable the API would be simpler. The only part of setTarget()'s logic the end-user needs to know is if the branch is ever missing both project and source_package, it gets automatically moved into the junk namespace of the branch owner. That information can go into the descriptions of project and source_package. But setTarget sets _either_ project or source_package. lazr.restful prohibits one field from having two mutators, but you might be able to declare a single method with different fixed arguments as the mutator for two different fields. Realistically, though, this has never been tried before and strains the capabilities of the system. A more realistic solution might be to implement setattr for 'project' and 'source_package' in Branch. This would let us treat them like normal fields without having to do any lazr.restful hacks. On top of this, I can implement something in lazr.restful that checks the URL of an object before and after invoking a POST named operation. But this named operation needs special treatment is a code smell. It indicates that the system is more complex than it needs to be. In this case, if the user thinks I want to move this branch to project foo, they have to look at the apidoc, discover that setting the project is a special case of setting the target, and invoke a named operation called setTarget. Ideally they would be able to set .project=foo and save. Summary: I should extend the 'check if the URL has changed' code to run after a POST named operation, but I don't think we need to publish this particular named operation, and turning project_link and sourcepackage_link into read-write fields will also solve your problem. Leonard ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Want feedback on LaunchpadSharp
Manish, Launchpad provides a WADL file describing the API for the web-service. I wrote a WADL-C# converter called wadlsharp[1] . Actually with minor modifications it can even create VB code but that is not the current target. wadlsharp is just a library and I am yet to write it's documentation. I created wadlsharp just to create the C# client code so that customizing the generated C# code can as easy as possible. I have released v0.1 of wadlsharp. Get the code from bzr[2] or download the tarball[3] First, I'm grateful that you're taking the time to write a .NET client for the web service. However, I think your current design will cause trouble later on for you and your users. Let me explain. I don't know any C#, so I don't understand what happens inside wadlsharp, but I believe your problems start there, with the generation of C# client code for later customization. I'm looking at LpClientProxy.cs in lpsharp, and it looks like it's the output of wadlsharp, or some wadlsharp-aided transformation of the WADL served by Launchpad on a specific date. I believe this design is a mistake, for three reasons. All three reasons are variants on a single reason: changes to the web service are not automatically reflected in the client. To see the changes, you must upgrade lpsharp. 1. Let's imagine that the Launchpad web service changes in a backwards-incompatible way. Scripts based on launchpadlib will break if they use the feature that changed, but launchpadlib itself will still function, and rewriting the launchpad-based script will fix the problem. Scripts that do not use the feature will still work correctly. From what I can see about the way lpsharp works, lpsharp scripts that do not use the modified feature will still work. But, lpsharp scripts that do use the feature will break, and these scripts _cannot be fixed_ without upgrading lpsharp and _then_ rewriting the script. Because the lpsharp code was generated from an old version of the WADL, lpsharp has no way to find out what the new service looks like. In real life, we take great pains not to make backwards-incompatible changes to the web service. So this scenario is unlikely to happen. But I can't rule it out, especially since it can happen by accident. If it does happen by accident, your users (unlike launchpadlib users) won't be able to use a temporary hack to get their scripts working: that would require a new lpsharp release, and you don't want to do a new release just to deal with a temporary problem on the server side. 2. OK, now imagine that we add a backwards-compatible improvement to the web service. You don't really have to imagine this because we do it all the time. Looking at the service_root_json class in LpClientProxy.cs, I see members like: ... private string _pillars_link; private string _translation_import_queue_entries_collection_link; private string _bug_trackers_collection_link; private string _countries_collection_link; ... For (almost) any one of these links, there was a time in the past year when the web service did not publish that link. The same is true for (almost) all the named operations defined in LPClientProxy.cs as people_findPerson, etc. When we add anything new to the web service, existing installations of launchpadlib will pick up on it. Existing installations of lpslash will not. This is not a catastrophe--obviously scripts written before a feature was published will not break just because they use a lpslash that doesn't know about that feature. But it will be annoying to your users, and it will tie lpslash's upgrade schedule to Launchpad's. Your users will also be effectively unable to use the bleeding-edge 'devel' version of the web service, which changes on a regular basis and which may break its own backwards compatibility from day to day. Which brings me to... 3. Currently there are three versions of the Launchpad web service: beta, 1.0 and devel. Each publishes a slightly (or drastically) different WADL document. This is how we're able to make the strong promises we make about backwards compatibility (see #1). launchpadlib's Launchpad constructor takes a version name, and reads (from the server or from a cache) the appropriate WADL document at startup. If there are 5 versions of the web service, there can be up to 5 cached WADL documents on the user's hard drive, but there's only one copy of launchpadlib. Five pieces of data, one piece of code. By contrast, LpClientProxy.cs is derived from a specific version of the WADL as it was served on a specific date. If there are 5 versions of the web service, there need to be 5 versions of LpClientProxy.cs in a lpslash release. That's five similar pieces of code packaged with lpslash, which need to be updated whenever the web service is updated (once a month, in the limiting case). This isn't a showstopper (unless you consider the inability to access the new features of 'devel' a showstopper), but it violates Don't Repeat Yourself and it'll be
Re: [Launchpad-dev] Adapters API (was Re: Exposing newCodeImport)
I guess you could implement IBranchTarget(obj).name and IBranchTarget(obj).newCodeImport() on the client-side too. As sugar for this case, or are you speaking about something else? I meant as a more general sugar. And, since my IRC backlog indicates I didn't make it clear enough here, I'm not saying Go forth and implement this solution, I'm discussing possible solutions to an interesting problem. It would be wise to seek Gary or Leonard's advice on this topic. Please don't implement anything special on the client side. I'm not totally opposed to a solution based on client-side conversion (as opposed to my earlier server-side include the adapters mechanism) but I want to be involved with that design and I don't have time right now. If you need to do something right now, I suggest implementing and exporting branch_target_name and branch_target_import() on the objects you want to see them on. If we do the server-side solution I proposed earlier, switching over will be invisible to the client. If we do a client-side solution, we can make branch_target_name and branch_target_import() disappear as of the first web service version to support the client-side solution. Leonard ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] webservice futures
Hi Gary, Leonard, How will the API's version numbers track or match up with Ubuntu release? Will there be a new API version for every Ubuntu release? Is there a wiki page on dev.lp.net that documents the mapping? Barry, There's no guarantee there will be a new API version for every Ubuntu release. There's also no necessary relationship between the version numbers and the name of the first Ubuntu release to use it by default--I think that would confuse more than it would help. The only relationship between API versions and Ubuntu releases is that we can't deprecate an API version until the EOL of the last Ubuntu release to use it by default. If you go to the apidoc index page on edge (https://edge.launchpad.net/+apidoc/), it now shows a list of all the active versions, and the description of each version shows the last Ubuntu release to use it. Leonard ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] Bug heat and the API
I really think that that excluding bug heat from the etag calculation is the right thing to do at this point. It's not like this information is up-to-date anyway. Bug heat isn't a real time data, so it shouldn't be included in the Etag. As a general solution to the problem, Bjorn's suggest seems like a good one, but on this particular problem, simply remove it from the calculation. And it's probable that simply having a way to exclude some fields from the Etag would solve 80% of our problems very cheaply. For bug heat specifically, perhaps removing it from ETag is appropriate. However, do bear in mind that folk may be getting stale responses many months later if no other fields change: caches can be quite stubborn; must-revalidate will, after all, look at the ETag. OK, I've finally had some time to think about this. 1. If a field is important enough to include in the representation, it's important enough to include in the ETag. As Robert points out, omitting a read-only field from the ETag will stop clients from knowing when the value changes. 2. I like Bjorn's idea of a two-part ETag, where both parts are checked for GET requests/caching, but only one part is checked for PATCH/PUT requests. But, I don't think this would comply with the HTTP standard. There are two types of ETags, strong and weak. A strong ETag changes whenever the entity (the entity-body or any entity-headers) changes in any way. A weak ETag changes only on semantically significant changes, and not when insignificant aspects of the entity change. These quotes are from section 13.3.3. Consider a two-part ETag, where one part describes the state of the read-only fields and the other part describes the state of the read-write fields. Taken together, the whole thing is a strong ETag. Look at only the second part and you have a weak ETag. But (13.3.3 again) The only function that the HTTP/1.1 protocol defines on [ETags] is comparison. There's no looking at the second half of an ETag. The whole thing has to match. OK, so let's define another function on ETags, another weak comparison function which only looks at the second half of the ETag. That goes beyond HTTP/1.1 but it doesn't contradict the standard. Now: We would like to validate GET requests using the strong comparison function (looking at the strong ETag), so that even minor changes will invalidate the cache and the client will always have the most up-to-date information. We would like to validate PATCH requests by using the weak comparison function, so that irrelevant changes to the resource don't affect the success of the PATCH. But, the HTTP standard says we have to do the opposite. Section 14.26 says The weak comparison function can only be used with GET or HEAD requests. The IETF draft that defines PATCH says, Clients wishing to apply a patch document to a known entity can first acquire the strong ETag of the resource to be modified, and use that Etag in the If-Match header on the PATCH request to verify that the resource is still unchanged. As I read the standards, if bug_heat has changed and I try to PATCH an unrelated field, the correct behavior is that my PATCH should fail. I don't think this makes sense for us given the way we implement PATCH, but it does make sense in general. I don't think anything bad will happen if we implement the two-part ETag. The whole ETag string is in fact a strong ETag, so anything that treats it as a strong ETag will still work. We'll just have some server-side magic that secretly implements a weak comparison function to make PATCH requests work. And if anyone actually tries to PATCH bug_heat they'll still get an error. I'd like to hear other peoples' readings of the standards, especially Robert's. Leonard ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] launchpadlib release?
Hey, We're actually going to do a launchpadlib release today. (It's ready to release, but I'm not doing it myself because gary and salgado want to test the release process, which itself uses launchpadlib: https://dev.launchpad.net/HackingLazrLibraries#Releases) So, I am developing launchpadlib and lazr.restfulclient, but as Gary says I'm focusing intently on the server side, which means the client side (launchpadlib and lazr.restfulclient) suffers. 1. I went through the launchpadlib review queue: https://code.edge.launchpad.net/launchpadlib/+activereviews There were a lot of branches in there, but almost all of them had been merged (there was a time when Launchpad thought launchpadlib was managed by PQM, even though it wasn't, so a merged branch didn't get picked up as 'merged') or disapproved and not marked 'rejected'. I see one very small branch that slipped through the cracks in June, and a branch from July that I thought had landed but now seems to have been reverted--I'm not sure what's up with that, and I'll investigate today. 2. I also looked at the lazr.restfulclient review queue: https://code.edge.launchpad.net/lazr.restfulclient/+activereviews I see two branches from this month, both from non-Launchpad developers. One of the branches was reviewed but the reviewer forgot to land it. I was the first Launchpad developer to look at the other one. I'm taking care of both of those today. 3. There were no pending branches for wadllib or lazr.restful. So I should definitely pay closer attention to the review queues, but I don't think there's a huge backlog. I'll add a section to https://dev.launchpad.net/HackingLazrLibraries on the review and landing process, because it is a little different from Launchpad, but it's also easier. If you're a Launchpad dev, you should feel comfortable developing launchpadlib or lazr.restfulclient. The reviews don't need to go through me--other Launchpad developers review my branches all the time, and as jml points out, there were changes committed (by allenap on behalf of others) to launchpadlib while I was on vacation. With gary trying out the release process today, there's now no reason I have to do the releases, either. I'm interested in the idea of a predictable release schedule for launchpadlib and lazr.restfulclient. I don't think we release only once a month makes sense, because that would slow down the rate at which improvements to these packages make it into Launchpad. But a policy of we release _at least_ once a month would avoid the problem I think jml is complaining about, where a useful launchpadlib change lands on November 26 and there's no release until December 17, when I need to do a release so I can land a Launchpad branch. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
Re: [Launchpad-dev] When will the table of contents reach the +apidoc page?
I don't want to sound impatient, but I would like to know when the fix for bug 325367 (The API docs should have a table of contents) will appear on https://edge.launchpad.net/+apidoc/? It would really improve the navigation on that page. I asked Karl Fogel about it (as he helped me to get the needed changes for it merged) and his reply sounded like it might happen around LP 3.0 release, but now he doesn't know himself either. Karl's change went into launchpadlib, not Launchpad itself. (1.5.2, the most recent launchpadlib release, has it.) To get the change into Launchpad we need to bring Launchpad's launchpadlib dependency up to date. I don't know how we go about doing that, but that's what we have to do. At this point, since Launchpad is open source, it might make more sense to move the XSLT file back into Launchpad. It was moved into launchpadlib so that third parties could use it. Leonard signature.asc Description: This is a digitally signed message part ___ Mailing list: https://launchpad.net/~launchpad-dev Post to : launchpad-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
