I also want to reiterate that I'm not bashing on the current design. It absolutely serves a purpose, has many happy users and delivers on many of the goals initially set out.
But, things have changed hugely in 5 years. -- John >> "primitives for waiting, paginating, retrying, etc" < absolutely. Each driver does it differently today and this is a requirement across all. Most APIs should give page count on the first request or support a HEAD request so pages should be able to be fetched in parallel. Jed >> "A data-driven approach to generating bindings is really, really interesting" +1. I'd spent a bit of time trying to understand how Microsoft build their Python SDK and it's almost entirely automated now, see https://github.com/Azure/azure-sdk-for-python/blob/master/swagger_to_sdk_config.json for how they use autorest to create bindings. I have honestly tried to build out the Azure driver and it improve it. It ended up costing me $100's and I broke the API. --- Another project for us to look into is Terraform. Terraform doesn't do any abstraction, but it does more closely declare the dependencies between infrastructure components. Within Libcloud today there isn't a way to declare which infrastructure assets any ex_ methods are provisioning, fetching or updating. What attributes those assets should have and their behaviours or dependencies. Therefore, trying to do something like Terraform would be hugely challenging. A colleague of mine did try on this project - https://github.com/DimensionDataCBUSydney/plumbery On Fri, Mar 9, 2018 at 12:18 PM, Jed Smith <j...@jedsmith.org> wrote: > + a million to John, and I’m actually in the same boat. I stepped away > due to a conflict, though I technically remain on the PMC (it seems). > Every time I’ve wanted to come back the current design stops me, and > that’s not speaking negatively, just that I have a difficult time > approaching modern libcloud. > I work for a company much more supportive of open source now, and I’d > love to put my volunteer hours into libcloud again. I agree with > everything architectural discussed in this thread, and moving that > direction would rouse me from hibernation. > What would I add to the discussion is that the individual layers should > strive for requests-level simplicity and thought. I agree the ex_ > approach is offputting. I think Python, especially modern Python, is > expressive enough to do this aspect well. > A data-driven approach to generating bindings is really, really > interesting. If that ends up being too bold for libcloud, I might help > explore that separately. That’s a really interesting idea. > -J > > > On Thu, Mar 8, 2018, at 19:18, John Carr wrote: > > Hi Anthony, > > > > I used to be an active contributor to libcloud, and am > > technically still> a committer although I don’t think I’ve been active > at all since you > > stepped into the fold. I just wanted to start by saying thanks > > for your> hard work! > > > > I fully support your proposal. It will take a while to be ready > > to fully> exploit py3, we should get started. > > > > For me what went wrong in the current codebase is that the common > > interfaces weren’t common enough. Even within the simple use > > cases there> were often paper cuts (inconsistencies in exceptions > raised). And the> ex_ approach was off-putting too. > > > > Ultimately I ended up only targeting AWS and it was a no > > brainer to use> botocore. > > > > An asyncio libcloud could be really compelling - but it should learn a> > few tricks from boto/botocore. > > > > I think there should be a low level API that targets the actual API as> > closely as possible. It should provide some commonality - > > primitives for> waiting, paginating, retrying, etc. Maybe some well known > > exceptions. It> should probably map to basic types rather than exposing > XML. > > Fundamentally AWS APIs are inconsistent but these botocore > > utilities and> patterns make up for it mostly. > > > > High level abstractions should be built on top of this layer > > and only do> what is actually common between APIs - no iffy ex_ params. > > > > I wonder if we can make libcloud more attractive to providers > > wanting to> write a python sdk? > > > > One idea I had a long time ago was to see how well the botocore data > > driven model could extend to other providers. I don’t really think we> > have the resources to do this. But the thinking was we could generate> > requests, asyncio and twisted bindings from the same data and > > potentially share the data with an node cloud driver for example. > > > > Cheers > > John > > > > > On 8 Mar 2018, at 23:16, anthony shaw <anthonys...@apache.org> > > > wrote:> > > > > Hi All, > > > > > > The "cloud market" when Apache Libcloud was conceived in 2010 > > > is very> > different to how it looks today, some trends we are seeing > > > > > > 1- IaaS (our compute API) is one of many features in public clouds. > > > Amazon,> > Azure and GCP have 100's of individual services now [1]. > Text-to- > > > speech,> > functions, automation, API gateways, it's impossible to > keep up > > > 2- Private Clouds have seen a continued decline but are still > > > popular [2]> > 3- The advent of containers means it is now "easier" to > deploy an > > > application to multiple clouds > > > 4- The big 3 public clouds, Amazon, Azure and Google make up most > > > of the> > cloud market [3] if you compare Apache Libcloud downloads > with > > > boto (the> > native AWS Python client) downloads, it's a massive > delta. Boto is > > > in the> > top 10 most popular PyPi packages > > > > > > In terms of users, I've pulled 3 snapshots of PyPi downloads, > > > January 2016,> > 2017 and 2018 [4] > > > Annual downloads of Apache-Libcloud have seen a slight increase, > > > but the> > 2016 - 2,778,687 > > > 2017 - 2,958,591 > > > > > > Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is > > > a huge> > drop. [4] > > > > > > *I would like to propose a drastic (depending on your perspective) > > > plan to> > take Apache Libcloud through to 2020* > > > > > > *1. Focus on specific use cases* > > > > > > Lack of consistency between implementations of the base class > > > means in> > practicality it's difficult to have abstracted > applications. > > > > > > Apache Libcloud should (imo) come with a toolbox of utilities > > > to both> > demonstrate and validate cloud abstraction use cases, such > as: > > > - Migrating storage objects from Cloud X to Cloud Y > > > - Splitting an application workload across multiple clouds > > > > > > *2. Improve performance by adopting asyncio* > > > > > > In almost all use cases, Libcloud would benefit from non-blocking > > > calls.> > Listing VM's requires multiple calls for the pages, uploading > > > storage> > objects can be done in multiple futures, deleting DNS > records > > > would be> > better done in async. > > > > > > I'm suggesting we introduce a Python 3.5+ only API, move to > > > requests-futures or aiohttp for the base HTTP client. Yes, *I am > > > suggesting> > we drop Python 2 support in the future*. > > > > > > I've been researching how we could make this switch without breaking > > > 64% of> > our users.. > > > Pip now has a way to choose versions based on Python runtimes > > > https://hackernoon.com/phasing-out-python-runtimes- > gracefully-956f112f33c4> > We could have apache-libcloud 3+ for Python > 3.5 users and then > > > maintain> > 2.3 patches for Python 2.7 and 3.4 users. > > > > > > I think we can come up with a way of continuing to maintain the > > > existing> > code base for 2.7 users but move forward with a new API for > > > async and> > Python 3.5+ users. > > > > > > The downloads [4] show that Python 2.7 is still the majority of > > > users but> > this is declining quickly and by 2020 the tables will > have turned. > > > We need> > to be ready for that. > > > > > > *3. Change our positioning on dependencies of 3rd party packages* > > > > > > We aren't seeing enough community contribution to keep up with > > > the rapid> > pace at which Microsoft, Amazon and Google are changing > their APIs. > > > Google> > have been contributing to their driver for years. We haven't > seen > > > that from> > either Amazon or Microsoft. Alibaba have contributed to > theirs, > > > many other> > cloud providers have contributed, but it seems to be > after APIs are> > changed, not in advance. > > > > > > > > > * 1. Docker is another example, that API changes almost every > > > month. The> > driver we have is unstable and doesn't support API > versioning > > > correctly. *> > Please consider these options, > > > > > > I would like to hear from users how you are currently using Apache > > > Libcloud> > and how you are using it > > > > > > NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly> > > because they represent +80% of the cloud market [3]. This is a > > > community> > developed project, has no affiliation to those vendors > and these > > > are my> > opinions, not those of the project. > > > > > > [1] https://www.amazonaws.cn/en/products/ > > > [2] > > > http://www.computerweekly.com/news/450280991/IDC-research- > predicts-gradual-decline-in-on-premise-hardware-spend-as- > cloud-adoption-rises> > [3] > > > https://www.crn.com.au/news/microsoft-ate-into-aws-market- > share-in-q4-481240> > [4] > > > https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7- > vPIYPer2VFtSo_Q/edit?usp=sharing> > >
