Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
IMHO, a well-documented WADL + XSD would say a thousand words (maybe more)... And can serve as a basis for automated testing as well. I understand that the v3 API draft is perhaps not at that stage yet; but, would like to see a WADL + XSD set as soon as the concepts are solidified. Liem -Original Message- From: openstack-bounces+liem_m_nguyen=hp@lists.launchpad.net [mailto:openstack-bounces+liem_m_nguyen=hp@lists.launchpad.net] On Behalf Of Mark Nottingham Sent: Tuesday, June 12, 2012 8:43 PM To: Gabriel Hurley Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [keystone] v3 API draft (update and questions to the community) On 13/06/2012, at 1:24 PM, Gabriel Hurley wrote: Totally agree with all of Jay's points, and I also couldn't agree more with Mark on the importance of being crystal clear, and not operating on just a common understanding which is quickly misunderstood or forgotten. Ideally I'd like to see an OpenStack API feature contract of some sort... essentially a document describing the FULL list of features, how those parameters are controlled and how they would interact, and what a project should do if they do not implement an API feature (hopefully only for technical reasons such as Keystone paging with LDAP or swift with complex DB-esque operations). This isn't saying we should have a unified API spec, I'm talking solely about a contract for the features all APIs should strive to support. This would be a big project, but everyone would then have a common agreement about what the user experience of interacting with OpenStack should be. The project APIs as they stand are siloed and stunningly inconsistent, and I'd love to work toward fixing that. Absolutely. One of my other projects is to rewrite the API as a proper specification (in a style similar to an Internet-Draft, not that we'd necessarily publish it as one). I should have something to show soon; if you're interested in helping out, that'd be great. Cheers, My two cents, - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Mark Nottingham Sent: Tuesday, June 12, 2012 7:20 PM To: Jay Pipes Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [keystone] v3 API draft (update and questions to the community) On 13/06/2012, at 3:31 AM, Jay Pipes wrote: This isn't necessarily true. Nova's compute layer goes through a number of steps to ensure a semi-transactional nature to certain operations like resizing. Certain times a query needs to indicate that it intends to make a reservation of resources (see quota/reservation system now .. this is the SELECT FOR UPDATE paradigm) and other times, the query doesn't care about such things. In the latter case, there aren't expectations that the list returned is 100% accurate according to the state of the database at a particular timestamp of when the transaction occurred. In this case, filters and optimistic pagination works perfectly fine, IMHO. That might work, but we need to be crystal-clear about the semantics of what we're giving back; having it understood between OpenStack projects isn't good enough. I.e., we're not building the APIs just for Horizon; they're for lots of folks, and subtle semantics -- even when well-documented, much less when they're not -- are often misunderstood. Cheers, -- Mark Nottingham http://www.mnot.net/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Mark Nottingham http://www.mnot.net/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
+1 Ideally I'd like to see an OpenStack API feature contract of some sort... essentially a document describing the FULL list of features, how those parameters are controlled and how they would interact, and what a project should do if they do not implement an API feature (hopefully only for technical reasons such as Keystone paging with LDAP or swift with complex DB-esque operations). This isn't saying we should have a unified API spec, I'm talking solely about a contract for the features all APIs should strive to support.We were discussing just that on the docs meeting earlier in the week. Whether it is a formal spec, or just documentation, is certainly open to debate. We may have need of both. This applies across all projects, IMO.Cheers,Christopher FerrisIBM Distinguished Engineer, CTO Industry and Cloud StandardsMember, IBM Academy of TechnologyIBM Software Group, Standards Strategyemail: chris...@us.ibm.comTwitter: christo4ferrisphone: +1 508 234 2986-openstack-bounces+chrisfer=us.ibm@lists.launchpad.net wrote: -To: "openstack@lists.launchpad.net" openstack@lists.launchpad.netFrom: Gabriel Hurley <gabriel.hur...@nebula.com>Sent by: openstack-bounces+chrisfer=us.ibm@lists.launchpad.netDate: 06/12/2012 11:28PMSubject: Re: [Openstack] [keystone] v3 API draft (update and questions to the community)Totally agree with all of Jay's points, and I also couldn't agree more with Mark on the importance of being crystal clear, and not operating on just a "common understanding" which is quickly misunderstood or forgotten.Ideally I'd like to see an OpenStack API feature contract of some sort... essentially a document describing the FULL list of features, how those parameters are controlled and how they would interact, and what a project should do if they do not implement an API feature (hopefully only for technical reasons such as Keystone paging with LDAP or swift with complex DB-esque operations). This isn't saying we should have a unified API spec, I'm talking solely about a contract for the features all APIs should strive to support.This would be a big project, but everyone would then have a common agreement about what the user experience of interacting with OpenStack should be. The project APIs as they stand are siloed and stunningly inconsistent, and I'd love to work toward fixing that.My two cents, - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Mark Nottingham Sent: Tuesday, June 12, 2012 7:20 PM To: Jay Pipes Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [keystone] v3 API draft (update and questions to the community) On 13/06/2012, at 3:31 AM, Jay Pipes wrote: This isn't necessarily true. Nova's compute layer goes through a number of steps to ensure a semi-transactional nature to certain operations like resizing. Certain times a query needs to indicate that it intends to make a reservation of resources (see quota/reservation system now .. this is the SELECT FOR UPDATE paradigm) and other times, the query doesn't care about such things. In the latter case, there aren't expectations that the list returned is 100% accurate according to the state of the database at a particular timestamp of when the transaction occurred. In this case, filters and optimistic pagination works perfectly fine, IMHO. That might work, but we need to be crystal-clear about the semantics of what we're giving back; having it understood between OpenStack projects isn't good enough. I.e., we're not building the APIs just for Horizon; they're for lots of folks, and subtle semantics -- even when well-documented, much less when they're not -- are often misunderstood. Cheers, -- Mark Nottingham http://www.mnot.net/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp___Mailing list: https://launchpad.net/~openstackPost to : openstack@lists.launchpad.netUnsubscribe : https://launchpad.net/~openstackMore help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
I'd love to review it when you're ready for review, or collaborate on it prior to a public offering. My laundry-list of features that an ideal API would contain is ever-growing, though I do try to temper it with reality. We should compare notes sometime. All the best, - Gabriel -Original Message- From: Mark Nottingham [mailto:m...@mnot.net] Sent: Tuesday, June 12, 2012 8:43 PM To: Gabriel Hurley Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [keystone] v3 API draft (update and questions to the community) On 13/06/2012, at 1:24 PM, Gabriel Hurley wrote: Totally agree with all of Jay's points, and I also couldn't agree more with Mark on the importance of being crystal clear, and not operating on just a common understanding which is quickly misunderstood or forgotten. Ideally I'd like to see an OpenStack API feature contract of some sort... essentially a document describing the FULL list of features, how those parameters are controlled and how they would interact, and what a project should do if they do not implement an API feature (hopefully only for technical reasons such as Keystone paging with LDAP or swift with complex DB-esque operations). This isn't saying we should have a unified API spec, I'm talking solely about a contract for the features all APIs should strive to support. This would be a big project, but everyone would then have a common agreement about what the user experience of interacting with OpenStack should be. The project APIs as they stand are siloed and stunningly inconsistent, and I'd love to work toward fixing that. Absolutely. One of my other projects is to rewrite the API as a proper specification (in a style similar to an Internet-Draft, not that we'd necessarily publish it as one). I should have something to show soon; if you're interested in helping out, that'd be great. Cheers, My two cents, - Gabriel -Original Message- From: openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Mark Nottingham Sent: Tuesday, June 12, 2012 7:20 PM To: Jay Pipes Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [keystone] v3 API draft (update and questions to the community) On 13/06/2012, at 3:31 AM, Jay Pipes wrote: This isn't necessarily true. Nova's compute layer goes through a number of steps to ensure a semi-transactional nature to certain operations like resizing. Certain times a query needs to indicate that it intends to make a reservation of resources (see quota/reservation system now .. this is the SELECT FOR UPDATE paradigm) and other times, the query doesn't care about such things. In the latter case, there aren't expectations that the list returned is 100% accurate according to the state of the database at a particular timestamp of when the transaction occurred. In this case, filters and optimistic pagination works perfectly fine, IMHO. That might work, but we need to be crystal-clear about the semantics of what we're giving back; having it understood between OpenStack projects isn't good enough. I.e., we're not building the APIs just for Horizon; they're for lots of folks, and subtle semantics -- even when well-documented, much less when they're not -- are often misunderstood. Cheers, -- Mark Nottingham http://www.mnot.net/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Mark Nottingham http://www.mnot.net/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
Mark, Apparently you must have missed my lightning talk at the Essex summit... ;-) (http://gabrielhurley.github.com/slides/openstack/apis_like_orms/index.html) Filtering, pagination, and many other API features are *critical* for a rich dashboard experience. If you want to talk specifics, the entire Horizon team would be happy to have a long chat with you. That said, we have also considered the case you propose where you effectively request everything and handle it on the client-side... however, I see that as a tremendously lazy solution. On the service-provider end you have access to powerful database methods that can do these operations in fractions of the time the client-side can (especially with good indexes, etc.). And if you've ever worked in mobile applications you'll know that minimizing data across the wire is crucial. The only argument I've heard in favor of that is basically it's easier for us not to add API features. To speak on the specific feature of pagination, the problem of 'corruption' by simultaneous writers is no excuse for not implementing it. You think Google, Facebook, Flickr, etc. etc. etc. don't have this problem? If you consume their feeds you'll notice you can fetch offset-based pagination with ease. You'd never expect to see a navigation element at the bottom of Google search results that said take me to results starting with the letter m. None of this is a case of someone might use it. The Horizon team has been loudly asking for these features for 8+ months now. And not just from Keystone but from all the projects. I have a list a mile long of API features we need to really deliver a compelling experience. I was just adding some items to it today, in fact. The rest of your points I have no strong feelings on and generally agree, but when it comes to API features... I feel *very* strongly. All the best, - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Mark Nottingham Sent: Monday, June 11, 2012 10:27 PM To: Joseph Heck Cc: openstack@lists.launchpad.net (openstack@lists.launchpad.net) Subject: Re: [Openstack] [keystone] v3 API draft (update and questions to the community) On 11/06/2012, at 6:58 AM, Joseph Heck wrote: First - what's the current thought of support for PATCH vs PUT in updating REST resources? Are there any issues with clients being able to use a PATCH verb? It's not something I'm super familiar with, so I'm looking for feedback from the community here. Ideally, I'd like to support the semantics of the PATCH HTTP verb, and possibly just assert no support for the PUT verb to be clear about intended functionality. Is that going to throw anyone for a loop? I answered a question about PATCH before; don't want to repeat myself, but it should be workable. Happy to chat more about it if you have specific questions. Second - filtering/searching for resources. The draft includes a section labelled Query By Name, which is probably mis-labelled, as it's intended to cover the general idea of passing in query parameters to general listing resource endpoints to filter the result set. The API endpoints across all the resources are defined as plurals, with the idea that specificity comes later in the URI (for referencing a single resource), or that we could add on these query parameters to restrict/filter by resource type. I'm in the middle of doing some log analysis and other research about how the APIs are used at Rackspace. It's too early to share results (although I do intend to, in some form, because the idea is to inform future API design), but one of the things that's very noticeable is how (extremely!) little pagination and filtering seem to be used in anger. In fact, if you take a look at the libraries, you'll find that they often don't use or even support filtering or pagination; e.g., libcloud doesn't, AFAICT. So, it's worth having a think about what the use cases actually are; both filtering and pagination are usually ways to save one or more of: a) client-side work b) server-side work c) bandwidth / latency One interesting exercise would be to estimate the largest number of users (or whatever else you'd be listing) that a reasonable deployment would put in a single response, triple it, do a dummy serialisation in JSON, and then gzip it, so that you can estimate the size, see how long it takes to parse on the client, etc. From what I've seen (in OpenStack as well as in other APIs that have nothing to do with Cloud), API designers tend to overestimate the utility of pagination and especially filtering (somebody might use it), but users just ignore them, doing all of the work on the client side, except in extreme circumstances (e.g., VERY large responses / very high latency). Unless you have strong use cases
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
On 12/06/2012, at 6:24 PM, Gabriel Hurley wrote: Mark, Apparently you must have missed my lightning talk at the Essex summit... ;-) (http://gabrielhurley.github.com/slides/openstack/apis_like_orms/index.html) Until I clone myself, I think that's going to happen pretty regularly; missing interesting sessions at the summit seems to happen too much :( Filtering, pagination, and many other API features are *critical* for a rich dashboard experience. If you want to talk specifics, the entire Horizon team would be happy to have a long chat with you. Love to. Like I said, I'm still doing my research. That said, we have also considered the case you propose where you effectively request everything and handle it on the client-side... however, I see that as a tremendously lazy solution. That's not the word I'd choose, but OK. What's concerning me here is that if someone has a new requirement for how the data is presented in Horizon (or any other control panel / app taking this approach), it's going to need to be reflected in the APIs, which is going to make them balloon out pretty quickly. It'd be interesting to get a sense of how other API consumers are planning to use these features (or not); like I said, I didn't see it in the log data. On the service-provider end you have access to powerful database methods that can do these operations in fractions of the time the client-side can (especially with good indexes, etc.). Oh, I don't know. CPU on the server side is often scarce, especially as you pile on features. Lots of clients means lots of power on the client side, in aggregate, and it also puts the responsibility for scaling expensive things in the hands of the party who's asking for them. And if you've ever worked in mobile applications you'll know that minimizing data across the wire is crucial. Absolutely. But, I'd question if these features are actually going to help. Many mobile clients are going to want to show all of a user's VMs in a list, for example, and paginating them won't help unless the sort order is *exactly* aligned with how the client wants to present them (and from what I saw, the proposed API didn't offer sorting at all). The only argument I've heard in favor of that is basically it's easier for us not to add API features. Well, I'd put it slightly differently. The question is how much footprint our API should have; if we try to make everybody happy, it'll be absolutely huge, and have corresponding issues -- everything from security to documentation to server-side load, and so forth. However, this being HTTP, somebody can (and will!) slap an intermediary or client library in front of it to do interesting things, even if we don't provide them out of the box. So I think the question really is where we, as a community, want to concentrate our efforts. To speak on the specific feature of pagination, the problem of 'corruption' by simultaneous writers is no excuse for not implementing it. You think Google, Facebook, Flickr, etc. etc. etc. don't have this problem? If you consume their feeds you'll notice you can fetch offset-based pagination with ease. You'd never expect to see a navigation element at the bottom of Google search results that said take me to results starting with the letter m. Strangely, I have; I wrote an RFC about it. None of this is a case of someone might use it. The Horizon team has been loudly asking for these features for 8+ months now. OK, that's good to know -- I didn't know that, but now I do. Look forward to discussing it more. And not just from Keystone but from all the projects. I have a list a mile long of API features we need to really deliver a compelling experience. I was just adding some items to it today, in fact. The rest of your points I have no strong feelings on and generally agree, but when it comes to API features... I feel *very* strongly. All the best, - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Mark Nottingham Sent: Monday, June 11, 2012 10:27 PM To: Joseph Heck Cc: openstack@lists.launchpad.net (openstack@lists.launchpad.net) Subject: Re: [Openstack] [keystone] v3 API draft (update and questions to the community) On 11/06/2012, at 6:58 AM, Joseph Heck wrote: First - what's the current thought of support for PATCH vs PUT in updating REST resources? Are there any issues with clients being able to use a PATCH verb? It's not something I'm super familiar with, so I'm looking for feedback from the community here. Ideally, I'd like to support the semantics of the PATCH HTTP verb, and possibly just assert no support for the PUT verb to be clear about intended functionality. Is that going to throw anyone for a loop? I answered a question about PATCH before; don't want to repeat myself
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
P.S. the X-Subject-Token stuff is breaking HTTP; you need to either put the token (or a facsimile for it) in the URL, or put Vary: Subject-Token in EVERY response those resources generate. The former is preferred; this is over TLS, right? Sorry I didn't see that earlier. P.P.S If it's not too late, drop the X- from that header! http://tools.ietf.org/html/draft-ietf-appsawg-xdash-05 Mark - could you open a bug against Keystone for the X-Subject-Token breaking HTTP with the relevant details? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
On 06/12/2012 04:24 AM, Gabriel Hurley wrote: Mark, Apparently you must have missed my lightning talk at the Essex summit... ;-) (http://gabrielhurley.github.com/slides/openstack/apis_like_orms/index.html) Filtering, pagination, and many other API features are *critical* for a rich dashboard experience. If you want to talk specifics, the entire Horizon team would be happy to have a long chat with you. Yes and no. The reality is that it is a trade off. Server side, you pay by doing a network round trip, and having to determine ahead of time the mechanisms for sorting, caching, paging, etc. The real problem is that the parsing of the results can blow out the stack space in the browser. That said, we have also considered the case you propose where you effectively request everything and handle it on the client-side... however, I see that as a tremendously lazy solution. On the service-provider end you have access to powerful database methods that can do these operations in fractions of the time the client-side can (especially with good indexes, etc.). And if you've ever worked in mobile applications you'll know that minimizing data across the wire is crucial. The only argument I've heard in favor of that is basically it's easier for us not to add API features. At the expense of loading your Database. Serverside paging and filtering both require one of two things: caching or additional Database queries, and both increase your server footprint. For small datasets, or for limited queries, this is not a problem, but for scalability you want to limit the work you do on the server. For Keystone using the LDAP backend, caching and pagination are extremely expensive, and not something I would like to support. an LDAP query is not guaranteed to come back in any particular order, so you can't just do the SQL trick of executing the query at offset + window size. You have to do the equivalent of a Cursor, and this places serious load on the LDAP server, the Keystone server, and possibly impacts other apps dependand on LDAP. To speak on the specific feature of pagination, the problem of 'corruption' by simultaneous writers is no excuse for not implementing it. You think Google, Facebook, Flickr, etc. etc. etc. don't have this problem? If you consume their feeds you'll notice you can fetch offset-based pagination with ease. You'd never expect to see a navigation element at the bottom of Google search results that said take me to results starting with the letter m. There is a major difference. We are working with data that has to be ACID. Google, Facebook and flickr do not. Before you migrate a VM, you need to know if the host meets the criteria for the VM. If it changes between when you check and when you reserve the space for the VM, you have just over committed. Get it right eventually does not work for management apps. None of this is a case of someone might use it. The Horizon team has been loudly asking for these features for 8+ months now. And not just from Keystone but from all the projects. I have a list a mile long of API features we need to really deliver a compelling experience. I was just adding some items to it today, in fact. The rest of your points I have no strong feelings on and generally agree, but when it comes to API features... I feel *very* strongly. Note that I am not saying don't do pagination as I agree, it is essential for good user experience. What I am stating is that we need to be smart about the techniques and technologies we choose, as there is always an upside and a downside. All the best, - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Mark Nottingham Sent: Monday, June 11, 2012 10:27 PM To: Joseph Heck Cc: openstack@lists.launchpad.net (openstack@lists.launchpad.net) Subject: Re: [Openstack] [keystone] v3 API draft (update and questions to the community) On 11/06/2012, at 6:58 AM, Joseph Heck wrote: First - what's the current thought of support for PATCH vs PUT in updating REST resources? Are there any issues with clients being able to use a PATCH verb? It's not something I'm super familiar with, so I'm looking for feedback from the community here. Ideally, I'd like to support the semantics of the PATCH HTTP verb, and possibly just assert no support for the PUT verb to be clear about intended functionality. Is that going to throw anyone for a loop? I answered a question about PATCH before; don't want to repeat myself, but it should be workable. Happy to chat more about it if you have specific questions. Second - filtering/searching for resources. The draft includes a section labelled Query By Name, which is probably mis-labelled, as it's intended to cover the general idea of passing in query parameters to general listing resource endpoints
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
On 06/12/2012 04:24 AM, Gabriel Hurley wrote: Mark, Apparently you must have missed my lightning talk at the Essex summit... ;-) (http://gabrielhurley.github.com/slides/openstack/apis_like_orms/index.html) Filtering, pagination, and many other API features are *critical* for a rich dashboard experience. If you want to talk specifics, the entire Horizon team would be happy to have a long chat with you. That said, we have also considered the case you propose where you effectively request everything and handle it on the client-side... however, I see that as a tremendously lazy solution. On the service-provider end you have access to powerful database methods that can do these operations in fractions of the time the client-side can (especially with good indexes, etc.). And if you've ever worked in mobile applications you'll know that minimizing data across the wire is crucial. The only argument I've heard in favor of that is basically it's easier for us not to add API features. To speak on the specific feature of pagination, the problem of 'corruption' by simultaneous writers is no excuse for not implementing it. You think Google, Facebook, Flickr, etc. etc. etc. don't have this problem? If you consume their feeds you'll notice you can fetch offset-based pagination with ease. You'd never expect to see a navigation element at the bottom of Google search results that said take me to results starting with the letter m. None of this is a case of someone might use it. The Horizon team has been loudly asking for these features for 8+ months now. And not just from Keystone but from all the projects. I have a list a mile long of API features we need to really deliver a compelling experience. I was just adding some items to it today, in fact. The rest of your points I have no strong feelings on and generally agree, but when it comes to API features... I feel *very* strongly. As do I. Couldn't agree more. Pagination and filtering are critical components to the API, which is why we've tried as much as possible in Glance to support as many types of filters as we can. Best, -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
On 06/12/2012 12:21 PM, Adam Young wrote: On 06/12/2012 04:24 AM, Gabriel Hurley wrote: That said, we have also considered the case you propose where you effectively request everything and handle it on the client-side... however, I see that as a tremendously lazy solution. On the service-provider end you have access to powerful database methods that can do these operations in fractions of the time the client-side can (especially with good indexes, etc.). And if you've ever worked in mobile applications you'll know that minimizing data across the wire is crucial. The only argument I've heard in favor of that is basically it's easier for us not to add API features. At the expense of loading your Database. Serverside paging and filtering both require one of two things: caching or additional Database queries, and both increase your server footprint. For small datasets, or for limited queries, this is not a problem, but for scalability you want to limit the work you do on the server. This is actually incorrect for relational databases. Passing filter and pagination/offset parameters in the API allows *more efficient* queries to be executed on the database server (given proper indexing, of course...). Not passing in these parameters in the API means more full table scans, more rows transmitted over the wire, and more work done by the database server. For Keystone using the LDAP backend, caching and pagination are extremely expensive, and not something I would like to support. This is a problem with LDAP, not with SQL backends, which are specifically built for querying in this manner. That said, however, there *are* certain filters that LDAP can deal with pretty well, right? Things like limiting to a specific OU can allow LDAP to winnow results, correct? an LDAP query is not guaranteed to come back in any particular order, so you can't just do the SQL trick of executing the query at offset + window size. You have to do the equivalent of a Cursor, and this places serious load on the LDAP server, the Keystone server, and possibly impacts other apps dependand on LDAP. If this is the case, then one solution might be to raise NotImplementedError in the case of when an API filter is not supported by a backend and leave it up to the client to retry the full set of results request and do the filtering/pagination itself? To speak on the specific feature of pagination, the problem of 'corruption' by simultaneous writers is no excuse for not implementing it. You think Google, Facebook, Flickr, etc. etc. etc. don't have this problem? If you consume their feeds you'll notice you can fetch offset-based pagination with ease. You'd never expect to see a navigation element at the bottom of Google search results that said take me to results starting with the letter m. There is a major difference. We are working with data that has to be ACID. Google, Facebook and flickr do not. Before you migrate a VM, you need to know if the host meets the criteria for the VM. If it changes between when you check and when you reserve the space for the VM, you have just over committed. Get it right eventually does not work for management apps. This isn't necessarily true. Nova's compute layer goes through a number of steps to ensure a semi-transactional nature to certain operations like resizing. Certain times a query needs to indicate that it intends to make a reservation of resources (see quota/reservation system now .. this is the SELECT FOR UPDATE paradigm) and other times, the query doesn't care about such things. In the latter case, there aren't expectations that the list returned is 100% accurate according to the state of the database at a particular timestamp of when the transaction occurred. In this case, filters and optimistic pagination works perfectly fine, IMHO. None of this is a case of someone might use it. The Horizon team has been loudly asking for these features for 8+ months now. And not just from Keystone but from all the projects. I have a list a mile long of API features we need to really deliver a compelling experience. I was just adding some items to it today, in fact. The rest of your points I have no strong feelings on and generally agree, but when it comes to API features... I feel *very* strongly. Note that I am not saying don't do pagination as I agree, it is essential for good user experience. What I am stating is that we need to be smart about the techniques and technologies we choose, as there is always an upside and a downside. Sure, agreed. :) Best, -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
The X-Subject-Token solution is definitely not valid HTTP, in that it implies that two otherwise identical requests for GET /tokens would return two completely different results (hence the need for a Vary header, as we include for X-Auth-Token). I have a slightly more proper (and complicated) solution in mind if we want to continue with the current token architecture, but I'd much rather see PKI deprecate the idea of centralized token validation. Either way, I don't think a bug needs to be opened because it's not implemented in keystone today anyway (it was implemented in legacy, and wasn't ported to redux). -Dolph On Tue, Jun 12, 2012 at 11:10 AM, Joseph Heck he...@mac.com wrote: P.S. the X-Subject-Token stuff is breaking HTTP; you need to either put the token (or a facsimile for it) in the URL, or put Vary: Subject-Token in EVERY response those resources generate. The former is preferred; this is over TLS, right? Sorry I didn't see that earlier. P.P.S If it's not too late, drop the X- from that header! http://tools.ietf.org/html/draft-ietf-appsawg-xdash-05 Mark - could you open a bug against Keystone for the X-Subject-Token breaking HTTP with the relevant details? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
On Tue, Jun 12, 2012 at 3:24 AM, Gabriel Hurley gabriel.hur...@nebula.com wrote: To speak on the specific feature of pagination, the problem of 'corruption' by simultaneous writers is no excuse for not implementing it. You think Google, Facebook, Flickr, etc. etc. etc. don't have this problem? If you consume their feeds you'll notice you can fetch offset-based pagination with ease. You'd never expect to see a navigation element at the bottom of Google search results that said take me to results starting with the letter m. Maybe OT, but the reason Swift doesn't support offset-based pagination is because it doesn't scale well enough. That probably doesn't apply to everyone, though. - Mike ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
On 13/06/2012, at 3:31 AM, Jay Pipes wrote: This isn't necessarily true. Nova's compute layer goes through a number of steps to ensure a semi-transactional nature to certain operations like resizing. Certain times a query needs to indicate that it intends to make a reservation of resources (see quota/reservation system now .. this is the SELECT FOR UPDATE paradigm) and other times, the query doesn't care about such things. In the latter case, there aren't expectations that the list returned is 100% accurate according to the state of the database at a particular timestamp of when the transaction occurred. In this case, filters and optimistic pagination works perfectly fine, IMHO. That might work, but we need to be crystal-clear about the semantics of what we're giving back; having it understood between OpenStack projects isn't good enough. I.e., we're not building the APIs just for Horizon; they're for lots of folks, and subtle semantics -- even when well-documented, much less when they're not -- are often misunderstood. Cheers, -- Mark Nottingham http://www.mnot.net/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
Totally agree with all of Jay's points, and I also couldn't agree more with Mark on the importance of being crystal clear, and not operating on just a common understanding which is quickly misunderstood or forgotten. Ideally I'd like to see an OpenStack API feature contract of some sort... essentially a document describing the FULL list of features, how those parameters are controlled and how they would interact, and what a project should do if they do not implement an API feature (hopefully only for technical reasons such as Keystone paging with LDAP or swift with complex DB-esque operations). This isn't saying we should have a unified API spec, I'm talking solely about a contract for the features all APIs should strive to support. This would be a big project, but everyone would then have a common agreement about what the user experience of interacting with OpenStack should be. The project APIs as they stand are siloed and stunningly inconsistent, and I'd love to work toward fixing that. My two cents, - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Mark Nottingham Sent: Tuesday, June 12, 2012 7:20 PM To: Jay Pipes Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [keystone] v3 API draft (update and questions to the community) On 13/06/2012, at 3:31 AM, Jay Pipes wrote: This isn't necessarily true. Nova's compute layer goes through a number of steps to ensure a semi-transactional nature to certain operations like resizing. Certain times a query needs to indicate that it intends to make a reservation of resources (see quota/reservation system now .. this is the SELECT FOR UPDATE paradigm) and other times, the query doesn't care about such things. In the latter case, there aren't expectations that the list returned is 100% accurate according to the state of the database at a particular timestamp of when the transaction occurred. In this case, filters and optimistic pagination works perfectly fine, IMHO. That might work, but we need to be crystal-clear about the semantics of what we're giving back; having it understood between OpenStack projects isn't good enough. I.e., we're not building the APIs just for Horizon; they're for lots of folks, and subtle semantics -- even when well-documented, much less when they're not -- are often misunderstood. Cheers, -- Mark Nottingham http://www.mnot.net/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [keystone] v3 API draft (update and questions to the community)
On 13/06/2012, at 1:24 PM, Gabriel Hurley wrote: Totally agree with all of Jay's points, and I also couldn't agree more with Mark on the importance of being crystal clear, and not operating on just a common understanding which is quickly misunderstood or forgotten. Ideally I'd like to see an OpenStack API feature contract of some sort... essentially a document describing the FULL list of features, how those parameters are controlled and how they would interact, and what a project should do if they do not implement an API feature (hopefully only for technical reasons such as Keystone paging with LDAP or swift with complex DB-esque operations). This isn't saying we should have a unified API spec, I'm talking solely about a contract for the features all APIs should strive to support. This would be a big project, but everyone would then have a common agreement about what the user experience of interacting with OpenStack should be. The project APIs as they stand are siloed and stunningly inconsistent, and I'd love to work toward fixing that. Absolutely. One of my other projects is to rewrite the API as a proper specification (in a style similar to an Internet-Draft, not that we'd necessarily publish it as one). I should have something to show soon; if you're interested in helping out, that'd be great. Cheers, My two cents, - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Mark Nottingham Sent: Tuesday, June 12, 2012 7:20 PM To: Jay Pipes Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [keystone] v3 API draft (update and questions to the community) On 13/06/2012, at 3:31 AM, Jay Pipes wrote: This isn't necessarily true. Nova's compute layer goes through a number of steps to ensure a semi-transactional nature to certain operations like resizing. Certain times a query needs to indicate that it intends to make a reservation of resources (see quota/reservation system now .. this is the SELECT FOR UPDATE paradigm) and other times, the query doesn't care about such things. In the latter case, there aren't expectations that the list returned is 100% accurate according to the state of the database at a particular timestamp of when the transaction occurred. In this case, filters and optimistic pagination works perfectly fine, IMHO. That might work, but we need to be crystal-clear about the semantics of what we're giving back; having it understood between OpenStack projects isn't good enough. I.e., we're not building the APIs just for Horizon; they're for lots of folks, and subtle semantics -- even when well-documented, much less when they're not -- are often misunderstood. Cheers, -- Mark Nottingham http://www.mnot.net/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Mark Nottingham http://www.mnot.net/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [keystone] v3 API draft (update and questions to the community)
First a thank you to everyone who's swung by to read (and some comment) on the
V3 draft at
https://docs.google.com/document/d/1s9C4EMxIZ55kZr62CKEC9ip7He_Q4_g1KRfSk9hY-Sg/edit?pli=1.
It's been immensely useful.
To clear up a bit of confusion I caused (sorry Jay!) - there were *no* example
responses included in the document, although some of the pieces certainly
looked like they might be. I put in placeholders for example responses to be
added to make it more explicit, and cleaned up my formatting so that I was
consistent (and hopefully through that, more clear)
This morning/afternoon I went through and tried to provide answers and feedback
to most of the outstanding comments - the folks who commented will see the
results through the Google doc responses as they have notifications defined. I
made a few changes to the draft - in particular, identifing what the primary
key in the resource attributes would/should be from a REST perspective, and
crossing out or adding a few attributes here and there based on feedback. I
also tried to make some spelling fixes where I missed earlier.
I've also added an open points discussion near the top of the document, and I'd
like to raise a few of those issues here on the mailing list.
First - what's the current thought of support for PATCH vs PUT in updating REST
resources? Are there any issues with clients being able to use a PATCH verb?
It's not something I'm super familiar with, so I'm looking for feedback from
the community here. Ideally, I'd like to support the semantics of the PATCH
HTTP verb, and possibly just assert no support for the PUT verb to be clear
about intended functionality. Is that going to throw anyone for a loop?
Second - filtering/searching for resources. The draft includes a section
labelled Query By Name, which is probably mis-labelled, as it's intended to
cover the general idea of passing in query parameters to general listing
resource endpoints to filter the result set. The API endpoints across all the
resources are defined as plurals, with the idea that specificity comes later in
the URI (for referencing a single resource), or that we could add on these
query parameters to restrict/filter by resource type.
Would it be better to make each of the query parameters explicit in the API
beyond the pagination?
Third - there's a general workflow question about how to go from username +
password to a token scoped to a specific tenant. There are a few suggestions
outstanding:
1) make a default tenant concept on a user, and when the user authenticates and
gets a token, it's initially scoped to that specific tenant *unless* the
authentication request also explicitly passed in an alternative tenant_id.
1a) The user could then look up any additional tenant_id from the
/users/{userid}/tenants resource path.
1b) a variation of this could be to include a list of all tenants the
user is associated with in the token resource when it's returned, with the
token scoped by default to whatever is set in the user's default tenant
attribute.
2) when any authentication that is just handed in a username+password and no
tenent_id, hand back a list of tokens, all authenticated for the user.
Fourth - there's an outstanding suggestion that the token resource, in
particular, may be much better suited to including whole resources back,
instead of resource IDs or atom link references to those resources. That's
generally how /token is behaving in the v2 API, and I'm leaning towards making
that explicit in the V3 API as well. Right now the APi defines only that the
ID's will be returned, not representations of the whole resource.
Thoughts? Feedback?
-joe
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
