Re: Charm store API proposal, new version
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 To echo what Aaron says, in any distributed system, it is almost always a mistake not to design for bulk api calls. Even if you don't think they are needed for the initial use cases, they will almost always be needed at some point later. It is trivial to use a bulk call with a single value, but it is not trivial to go the other way. This topic has been discussed previously in the juju-core space. I had thought that William had (correctly) mandated the bulk calls were to be used for our distributed apis. On 16/07/14 00:39, Aaron Bentley wrote: On 14-07-15 10:17 AM, roger peppe wrote: The most significant change is that all endpoints refer just to a single charm or bundle, rather than being bulk calls as they were before. That sounds like the opposite of what juju-reports wants. Does it really mean that we would need to make N requests to retrieve info on N charms? What was the reason for changing this? Aaron -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iJwEAQECAAYFAlPGPQsACgkQCJ79BCOJFcb3KgP/fwTB7j+4aR4bFU5F9EU9HUOk HE/IuyEaNrgybDAingF3+ingPJDvYu1/Q2W+DLVVUXjWqosEJbu8NG4qEd/ev5C+ BagEiHD53Ht0jsGZFR6wC6CVVYr3P4u79uO6NsNT6OAEoQi/cXtTws2SC5zpvrCd qsOoQsGhfzSaJmQA8gs= =HThz -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Charm store API proposal, new version
... To echo what Aaron says, in any distributed system, it is almost always a mistake not to design for bulk api calls. Even if you don't think they are needed for the initial use cases, they will almost always be needed at some point later. It is trivial to use a bulk call with a single value, but it is not trivial to go the other way. This topic has been discussed previously in the juju-core space. I had thought that William had (correctly) mandated the bulk calls were to be used for our distributed apis. Just tossing in my agreement here. If you just provide single resource endpoints, it makes it really hard to ever improve your request efficiency. And while *current* clients may only ever pass one, if it is meant to be an API that we will use for a long time we should try to keep in mind future proofing. I will agree that we went a little overboard internally. We knew that the Client api should have been bulk but wasn't so we wrote all of the internal APIs as bulk even though it isn't really able to be used. However, if we can update Client, then at least things are consistent and useful across the board. Anyway, I do know that Launchpad suffered a lot having been written as one-by-one and it made it really hard to write efficient clients that interacted with LP's web API. It would be good to at least allow for the possibility of people wanting to build external consumers. John =:- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Charm store API proposal, new version
On Tue, Jul 15, 2014 at 7:05 PM, Richard Harding rick.hard...@canonical.com wrote: It is listed under known clients in the spec, and we mentioned your request down below. What we lack is your specific use cases as no one working on the spec is knowledgeable about how you're using the api. Besides what others have said, requiring everyone to not only review their own usage of the existing public APIs, but to justify their cases in a convincing way, as an attempt to prevent you from breaking the existing use cases is a pretty bad approach to API compatibility. gustavo @ http://niemeyer.net -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Charm store API proposal, new version
On Wed, Jul 16, 2014 at 12:10 PM, Richard Harding rick.hard...@canonical.com wrote: I'm not against having any bulk api calls but we've got a handful of clients and the one use case we've found for them is Aaron's work which I think we can better address with our current scheme anyway as he only needed the bulk api call to get data he'd already have on hand. I'd suggest at least one other use case -- namely, that juju-core wants to know which deployed charms have updates available, and it's annoying to issue N requests for N charms. But the major drivers are: 1) consistency, so that clients across the board can just *always know* that they can issue a given request in N-plicate and never have to mess around hunting for which variant of a given api call offers what they're looking for; 2) expressivity, because you can always express a single request in a bulk call but can't go the other way; 3) agility(?), in that bulk calls have scope for server-side optimisations that would be very very hard to apply across a list of single calls, and we'd rather address performance issues via implementation tweaks than API churn; and 4) past experience, both others' (launchpad) and our own, leading us to believe that it's generally nicer to work with chunky APIs than chatty ones, despite the slightly steeper initial learning curve. ...not to mention that IMO most of the arguments for single calls represent a failure of imagination. I certainly agree that it's not always obvious why someone might want to discover the latest revisions of N charms at once, or the metadata for M other charms that may or may not be directly related to one another, or reporting info for a bunch of others (or, in juju-core terms, why we might want to find out the life statuses of a bunch of entities, or why the set of state server addresses might vary according to what agent needs to connect, or why a client might want provisioning info for a whole bunch of new machines at a time). But simple prudence dictates that we pay the minimal costs of enabling such behaviours early, in the interest of providing a rich experience for all our clients, rather than scrambling to fix them up later as we discover the use cases that always somehow seem terribly predictable in hindsight -- and in doing so take on the long-term costs of uglified and non-orthogonal APIs (or, perhaps even worse, failing to do so and leaving our clients hanging because we're focused on other things). Cheers William -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Charm store API proposal, new version
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14-07-15 10:17 AM, roger peppe wrote: The most significant change is that all endpoints refer just to a single charm or bundle, rather than being bulk calls as they were before. That sounds like the opposite of what juju-reports wants. Does it really mean that we would need to make N requests to retrieve info on N charms? What was the reason for changing this? Aaron -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTxT04AAoJEK84cMOcf+9hTY4IALfbWrq6VydINsAzHha6Az1+ RhtpJpS74VBbnBSzFK3LgeHk1uLGLNa73VklKYqibKcSHe6TPUhfwpSldxHoSrXR KGq86SdjojCnFXMxUw1QBXqd4QXv6h19znO89AwgDK8Crw14IB3ZrQvmfNbyZWex WmJjlgtvcuImVLrzBntF1UQcHzzz5ycXI2nzzhVEGTlD+z2Eg+8SfSoagbMuZ6Zv q4I0gNEJcktnoHoYGYvx98umN57td5nj9u9R79c500ZQz7kIE7f4y07Hy7MxK714 gEzoFo4rnbgZJBCdoI3FiH4BqxP5uzWoj2iAG9UiMlPN6nFZUsJnZKjjcvS+nvM= =6gFz -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Charm store API proposal, new version
On Tue, 15 Jul 2014, Aaron Bentley wrote: On 14-07-15 10:17 AM, roger peppe wrote: The most significant change is that all endpoints refer just to a single charm or bundle, rather than being bulk calls as they were before. That sounds like the opposite of what juju-reports wants. Does it really mean that we would need to make N requests to retrieve info on N charms? What was the reason for changing this? The change is because we went through the known clients and could not come up with use cases where some client had a known list of charm ids they wanted data for, but they knew those ids ahead of time. In the gui, cli, charmtools, etc, we perform queries by a single entity id. The only exception was search, which is a different animal all together and has its own api points for this use. Aaron, I'd like to see if we can get your full use case details and make sure we address them. You had requested an all charms/bundles api call in the last revision of the doc and we're looking at addresssing that through search results. The entity type is a direct filter attribute on a search call. You're also able to describe what metadata about a charm/bundle you'd like to see in a search result. So the thought was that we'd limit the complexity of the api to that known use case. Can you describe your exact use case as a client of the api and we'll look at how the spec/design can be improved to help your use case? -- Rick Harding Juju UI Engineering https://launchpad.net/~rharding @mitechie -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Charm store API proposal, new version
On Tue, 15 Jul 2014, Aaron Bentley wrote: I am surprised that juju-reports was not considered a known client. I certainly made many comments on the first draft of the original proposal. It is listed under known clients in the spec, and we mentioned your request down below. What we lack is your specific use cases as no one working on the spec is knowledgeable about how you're using the api. Our use case is we want to be able to synchronize public data from the charm store into our DB. We need to provide the statistics that show adoption rates to Mark Ramm and Mark Shuttleworth. We cannot forsee exactly which questions will be asked. We need to be able to add new statistics without friction, so we need our own copy of this data. One way to do this would be to have an API that accepts a timestamp and returns the metadata for all charm and bundle revisions created on or after that timestamp. Right now we do this instead: 1. Retrieve a list of all charms from charmworld 2. Determine the tip revisions of all charms from the charm store 3. Determine which revisions exist but we haven't seen (including non-tip revisions) 4. Determine the revision ids of all the new revisions 5. Use the revision-ids to determine the publication dates of all the new revisions Steps 2, 4, 5 use multiple-id queries. Right now, we're showing number of charms published here: http://reports.vapour.ws/scorecard We'll need to do the same for bundles. We've also been asked to display the number of running environments. What specific metadata do you need? I know we need the list of older revisions of charms for the juju-gui. I'm going to make sure that's listed as a potential metadata request in the api. (It's missing currently) Given that data, when you get the latest version of a charm, you'd get a list of previously published versions to facilitate the upgrade/downgrade UI. It seems this would help you with the publish counts? Most of the other data around revisions and bugs we'll do our best at, but will be sacrificed information so that users can publish a charm or bundle from any vcs. What other metadata should be make sure is addressed? You had requested an all charms/bundles api call in the last revision of the doc and we're looking at addresssing that through search results. The doc doesn't say that search can be used to list all charms and bundles. If you meant it to be used that way, please say so in the doc ;-) Sure thing, thanks for pointing it out. Search provides results for only the most recent revisions. It can stand in for step 1 2 of our algorithm, but in order to implement 4 and 5 efficiently, we'd need to do multi-id queries. With the previous rev metadata we're missing but also required by the juju-gui I think we can resolve this with the current api implementation. Let me know if there's more we need to address. Thanks for reviewing the doc and providing feedback. We're very much hoping to build out a strong consistent api we can use for a long time going forward. -- Rick Harding Juju UI Engineering https://launchpad.net/~rharding @mitechie -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
