Let's work out some examples about promoting APIs and how the repos should work.
Example: I have a provider-specific API that extends the compute abstraction in a meaningful way. Currently I would have to: 1. Make a PR for the API in one of the labs 2. Make a PR for the abstraction in jclouds/jclouds 3. (Same as 1. in a way) Make a PR in other labs repos for other providers (as it is likely new abstraction features would have to be supported in other providers - some of these might be created by other people, obviously - or these features might already be supported) 4. Review and merge all of these. 5. Make PRs to promote the provider-specific APIs to jclouds/jclouds and merge those. 6. Make PRs to clean up labs repos. Does this make sense? Am I missing something? How can we improve this? ________________________________________ From: Ignasi Barrera [n...@apache.org] Sent: Monday, December 15, 2014 3:22 PM To: dev@jclouds.apache.org Subject: Re: [DISCUSS] What is the criteria for something to be in the jclouds/jclouds repo? I can agree that giving full control over provider specific APIs can be a good thing, but always as a secondary goal. As you say, abstractions and working on the missing ones should be the high priority. Adding provider specific features to complete the abstractions is also a common sense thing; and at this concrete point in time, keep growing or promoting the APIs that are not related to any abstraction don't benefit the project as a whole. I think the entire point of this discussion is to set up a baseline that allows us to "move forward". To do that, we need to have a common criteria and a common vision of which should be the next steps to evolve the project, and it seems we agree that the high priority should be to work on the abstractions and focus on them. This means that specially the PMC and committers should work to make this happen, and focus on this high priority tasks. Setting a rule that we need an abstraction to promote the APIs will help us have under control what we promote and help us focus on the tasks we have defined as high priority. If we keep putting our major efforts in things that are not the high priority (apis not related to an abstraction), we won't achieve the main purpose of "moving forward". This is not black or white, and I don't think "we will never promote an API that doesn't have an abstraction" is an statement we should do. However, at this point in time, we should *really enforce* any promoted provider and API to fit in an abstraction. We have limited resources, and we need to focus in the high priority tasks. If we don't dedicate these few resources to the high priority tasks, we won't succeed. So, "will having an abstraction be a mandatory requirement forever and ever?" I'd say No. But that will only make sense if we work *starting today* together on the APIs that actually fit on those abstractions, and start building and discussing the ones that could be added. We should prioritise that, and make that a *real* thing, leaving the unrelated apis/providers aside. Otherwise we will be at the same point and won't be able to move forward, regardless of what we promote. On 15 December 2014 at 18:49, Zack Shoylev <zack.shoy...@rackspace.com> wrote: >>> Every single API in jclouds must be aligned with the project's purpose, and > that means *portability*. > > jclouds also tries to do portability "while giving you full control to use > cloud-specific features." > >>> if someone contributes a patch adding support for Google Drive > > I was not part of that discussion, but it should be possible to add a > blobstore abstraction for this, no? > >>> So, once again, jclouds is *not* a cloud aggregator for any API labelled > "cloud". It is about portable code, and every single API and provider > should meet the project goals. > > jclouds allows you to write portable applications, but also allows you the > flexibility to use cloud-specific features. From a user perspective, you need > to have both. You want to be able to build infrastructure using high-level > code that works with all the clouds, but occasionally you will need the > flexibility to do provider-specific things (even just because this is how > providers differentiate themselves). > > > I wanted to focus on the user perspective for this discussion. In the end, > users want things to work well, with less code, and with whatever providers > they choose. > Because jclouds is modular, users don't care if jclouds "is not an API > aggregator of any API that is labelled "cloud"." - users want jclouds to work > well for whatever providers and clouds they choose. > As such, figuring out the criteria of what should be in jclouds/jclouds is > not really something that affects users directly, but has to be done to make > development easier, saner, and to reduce current and future technical debt. > If we had infinite resources, it would make sense, for users, to include and > release all cloud APIs and their abstractions (where such abstractions make > sense) as part of jclouds. > > Part of this is about balance. Currently we have too many cloud-specific APIs > that could be abstracted, but are not. Clearly some of the high-priority work > in jclouds is working on abstractions. That's the case at the moment. It > might be a knee-jerk reaction to limit jclouds just to abstractable APIs > because of this situation, when in the future we might also want to give > users the flexibility to also use provider-specific features or even > provider-specific APIs that might be useful when used together with > higher-level, more abstract code. > > This is our current mission statement: > "Apache jclouds® is an open source multi-cloud toolkit for the Java platform > that gives you the freedom to create applications that are portable across > clouds while giving you full control to use cloud-specific features." > > Should this be morphed into: > "Apache jclouds® is an open source multi-cloud toolkit for the Java platform > that gives you the freedom to create applications that are portable across > clouds"? Is this better for users? > > Thanks, and let's keep the discussion going. Figuring things out will help > both contributors and users. > > ________________________________________ > From: Ignasi Barrera [n...@apache.org] > Sent: Wednesday, December 10, 2014 12:59 AM > To: dev@jclouds.apache.org > Subject: RE: [DISCUSS] What is the criteria for something to be in the > jclouds/jclouds repo? > > I'd never do point 2 and would remove those APIs. As I've said several > times in the thread, jclouds is not an API aggregator of any API that is > labelled "cloud". > > Every single API in jclouds must be aligned with the project's purpose, and > that means *portability*. It does not make sense to keep adding APIs that > don't fit in the project's purpose. If we do that we would be facing the > exact same issues we are facing now (I mentioned them in my previous email) > regardless of the number of repos we use. > > To put a concrete example: jclouds is *not* the Rackspace SDK nor is the > OpenStack java SDK. Perhaps Rackspace adopts jclouds as its SDK, but > jclouds isn't the Rackspace SDK. And this difference is important. Perhaps > not every single API in Rackspace makes sense in jclouds, as it would > hardly be modelled in a portable way. In that case, we shouldn't add it, as > it does not benefit jclouds as a whole and doesn't serve the jclouds > purpose. > > Let's take another example: if someone contributes a patch adding support > for Google Drive (which was discussed in the ML or IRC), providing an API > to store documents, should we merge it? That API does not fit in the > BlobStore abstraction as it does not manage blobs (raw object data) but > documents. The answer is no, because jclouds is not about that (ir is just > a single, isolated API to store documents out there). Without alignment > with what jclouds targets, we shouldn't add it. > > So, once again, jclouds is *not* a cloud aggregator for any API labelled > "cloud". It is about portable code, and every single API and provider > should meet the project goals. If not, we should not add them. > El 10/12/2014 01:54, "Zack Shoylev" <zack.shoy...@rackspace.com> escribió: > >> If we take this to its logical conclusion, it sounds like we can separate >> APIs into two categories: >> >> 1. Those that have an abstraction or can be abstracted in the future (to >> be in jclouds/jclouds). Most APIs would be in here. >> 2. Those that provide provider-specific features only, and cannot be >> abstracted. There should only be a few here, but this means a few per >> provider, so overall this category might contain a large number of APIs. >> >> If we were to organize the repos using the two categories above, we can do >> something like (suggestion): >> >> 1. jclouds/jclouds (Same as now, with more APIs - including beta APIs - >> promoted. Regular release schedule) >> 2. jclouds/providers (All provider-specific APIs that will never be >> abstracted. Release together with 1. This also needs a better name, as it >> only contains APIs that are not abstractable). >> 3. jclouds/labs (very experimental only - will not be released) >> >> Note that I am strongly in favor of fewer repos for us to manage. Maybe we >> should not even have 2. >> >> What would you change in the description above? >> Thanks! >> -Zack >> ________________________________________ >> From: Ignasi Barrera [ignasi.barr...@gmail.com] >> Sent: Monday, December 08, 2014 4:19 PM >> To: dev@jclouds.apache.org >> Subject: Re: [DISCUSS] What is the criteria for something to be in the >> jclouds/jclouds repo? >> >> 1. So once an api meets the criteria of having a steward, being >> modern, and having a proper test suite, it can immediately be >> promoted? >> >> I really think that those APIs need to have an abstraction, or a >> potential abstraction that is being developed, but something real, >> tangible, that ensures that it is aligned with the essence of what >> jclouds is. >> >> 2. Can you elaborate on this point? Did you have some criteria in mind >> for what “makes sense” means? >> >> There are several features more than one cloud provider offers >> (networking, load balancers and databases, are the obvious examples) >> that could have an abstraction. Not every feature in every cloud >> provider may make sense in others, and thus are not suitable to have >> an abstraction. In that case, we should be asking ourselves what value >> adding such an "independent" API gives to jclouds. We have provider >> specific APIs to *complete* the functionality "offered by the >> abstractions", but I really don't think is a good idea to add "entire >> independent apis" that won't have an abstraction because they're too >> specific to a single cloud provider. When I say "makes sense" I'm >> saying to ask ourselves if that API really makes sense in the global >> jclouds picture. If it makes sense, let's work on getting those >> features available in a portable way to other providers too. If that >> doesn't make sense, then that API doesn't make sense to jclouds. >> >> 3. Are you saying that if there’s an api that would never be a good >> fit for abstraction, that it has no chance of being promoted? >> >> Yes. I really believe we should be doing that. As I said, jclouds is >> not an API placeholder to every single API that qualifies itself as >> "cloud". jclouds is all about abstractions, and about providing users >> a way to write portable code. Of course we have provider specific >> features, but as I mentioned before, those should serve the purpose of >> completing the abstractions, but never be independent and standalone >> providers/apis. That has proven to be difficult to maintain. >> considerably increase the codebase complexity and make it more >> difficult to evolve jclouds itself. >> >> 4. Are you saying that if an abstraction doesn’t happen in some amount >> of time, that any apis related to that lack-of-abstraction would be >> removed? >> >> Well, this is open source, and "time" is relative. The complexity of >> abstractions can be very different: databases seems to be a simple, >> but networking can be really complex. What I'm saying is that we >> should *work* on those abstractions and *work in a direction to make >> those abstractions" something real. But we can't just keep adding more >> "satellite" APIs and forget about the rest. We should be thinking >> about abstractions and woking on them, and that's what matters, not >> time. Take the database APIs as an example. It has been commented more >> than once that it would be nice to have an abstraction. What actions >> have been taken in that direction? None. Any proposal? Nope. That's >> what we have to change. >> >> >> >> On 8 December 2014 at 16:01, Everett Toews <everett.to...@rackspace.com> >> wrote: >> > On Dec 2, 2014, at 2:31 PM, Ignasi Barrera <n...@apache.org> wrote: >> > >> >> This said, the absence of an abstraction (take network or databases as >> >> an example) shouldn't be a blocker. Abstractions can be proposed in >> >> the mailing list, and even better discussed in a PR! We are all >> >> looking to improve jclouds to make it better for the users, and that >> >> means making it more portable. Let's work on adding an abstraction >> >> when that makes sense! But things have to be *real* and that work has >> >> to be done. Without it, it does not make sense to promote/keep such >> >> apis/providers. >> > >> > I think having everything in jclouds/jclouds could possibly work but we >> would need to be more specific about the above. Otherwise I’m concerned >> we’d wind up in the same place 6 months down the road. >> > >> > If this were to be the path we go down, I have some clarifying >> questions. These are open questions for anyone to answer. >> > >> > “the absence of an abstraction shouldn’t be a blocker” >> > >> > 1. So once an api meets the criteria of having a steward, being modern, >> and having a proper test suite, it can immediately be promoted? >> > >> > >> > “Let's work on adding an abstraction when that makes sense!” >> > >> > 2. Can you elaborate on this point? Did you have some criteria in mind >> for what “makes sense” means? >> > >> > >> > "it does not make sense to promote/keep such apis/providers.” >> > >> > These are two different concerns and should be broken out. >> > >> > "it does not make sense to promote such apis/providers.” >> > >> > 3. Are you saying that if there’s an api that would never be a good fit >> for abstraction, that it has no chance of being promoted? >> > >> > "it does not make sense to keep such apis/providers.” >> > >> > 4. Are you saying that if an abstraction doesn’t happen in some amount >> of time, that any apis related to that lack-of-abstraction would be removed? >> > >> > >> > Everett >>
