On 9 Mar 2018, at 01:18, Jed Smith <j...@jedsmith.org> wrote: > > A data-driven approach to generating bindings is really, really > interesting. If that ends up being too bold for libcloud, I might help > explore that separately. That’s a really interesting idea. > -J >
Here’s a prototype I started working on in 2015. I didn’t get very far, but if you check out the tests I did have one code base supporting sync and async. I had a middleware-esque approach where the data described which serialisers, request signers, etc to mash together. https://github.com/Jc2k/libcloudcore <https://github.com/Jc2k/libcloudcore> John > > On Thu, Mar 8, 2018, at 19:18, John Carr wrote: >> Hi Anthony, >> >> I used to be an active contributor to libcloud, and am >> technically still> a committer although I don’t think I’ve been active at >> all since you >> stepped into the fold. I just wanted to start by saying thanks >> for your> hard work! >> >> I fully support your proposal. It will take a while to be ready >> to fully> exploit py3, we should get started. >> >> For me what went wrong in the current codebase is that the common >> interfaces weren’t common enough. Even within the simple use >> cases there> were often paper cuts (inconsistencies in exceptions raised). >> And the> ex_ approach was off-putting too. >> >> Ultimately I ended up only targeting AWS and it was a no >> brainer to use> botocore. >> >> An asyncio libcloud could be really compelling - but it should learn a> few >> tricks from boto/botocore. >> >> I think there should be a low level API that targets the actual API as> >> closely as possible. It should provide some commonality - >> primitives for> waiting, paginating, retrying, etc. Maybe some well known >> exceptions. It> should probably map to basic types rather than exposing XML. >> Fundamentally AWS APIs are inconsistent but these botocore >> utilities and> patterns make up for it mostly. >> >> High level abstractions should be built on top of this layer >> and only do> what is actually common between APIs - no iffy ex_ params. >> >> I wonder if we can make libcloud more attractive to providers >> wanting to> write a python sdk? >> >> One idea I had a long time ago was to see how well the botocore data >> driven model could extend to other providers. I don’t really think we> have >> the resources to do this. But the thinking was we could generate> requests, >> asyncio and twisted bindings from the same data and >> potentially share the data with an node cloud driver for example. >> >> Cheers >> John >> >>> On 8 Mar 2018, at 23:16, anthony shaw <anthonys...@apache.org> >>> wrote:> > >>> Hi All, >>> >>> The "cloud market" when Apache Libcloud was conceived in 2010 >>> is very> > different to how it looks today, some trends we are seeing >>> >>> 1- IaaS (our compute API) is one of many features in public clouds. >>> Amazon,> > Azure and GCP have 100's of individual services now [1]. Text-to- >>> speech,> > functions, automation, API gateways, it's impossible to keep up >>> 2- Private Clouds have seen a continued decline but are still >>> popular [2]> > 3- The advent of containers means it is now "easier" to >>> deploy an >>> application to multiple clouds >>> 4- The big 3 public clouds, Amazon, Azure and Google make up most >>> of the> > cloud market [3] if you compare Apache Libcloud downloads with >>> boto (the> > native AWS Python client) downloads, it's a massive delta. >>> Boto is >>> in the> > top 10 most popular PyPi packages >>> >>> In terms of users, I've pulled 3 snapshots of PyPi downloads, >>> January 2016,> > 2017 and 2018 [4] >>> Annual downloads of Apache-Libcloud have seen a slight increase, >>> but the> > 2016 - 2,778,687 >>> 2017 - 2,958,591 >>> >>> Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is >>> a huge> > drop. [4] >>> >>> *I would like to propose a drastic (depending on your perspective) >>> plan to> > take Apache Libcloud through to 2020* >>> >>> *1. Focus on specific use cases* >>> >>> Lack of consistency between implementations of the base class >>> means in> > practicality it's difficult to have abstracted applications. >>> >>> Apache Libcloud should (imo) come with a toolbox of utilities >>> to both> > demonstrate and validate cloud abstraction use cases, such as: >>> - Migrating storage objects from Cloud X to Cloud Y >>> - Splitting an application workload across multiple clouds >>> >>> *2. Improve performance by adopting asyncio* >>> >>> In almost all use cases, Libcloud would benefit from non-blocking >>> calls.> > Listing VM's requires multiple calls for the pages, uploading >>> storage> > objects can be done in multiple futures, deleting DNS records >>> would be> > better done in async. >>> >>> I'm suggesting we introduce a Python 3.5+ only API, move to >>> requests-futures or aiohttp for the base HTTP client. Yes, *I am >>> suggesting> > we drop Python 2 support in the future*. >>> >>> I've been researching how we could make this switch without breaking >>> 64% of> > our users.. >>> Pip now has a way to choose versions based on Python runtimes >>> https://hackernoon.com/phasing-out-python-runtimes-gracefully-956f112f33c4> >>> > We could have apache-libcloud 3+ for Python 3.5 users and then >>> maintain> > 2.3 patches for Python 2.7 and 3.4 users. >>> >>> I think we can come up with a way of continuing to maintain the >>> existing> > code base for 2.7 users but move forward with a new API for >>> async and> > Python 3.5+ users. >>> >>> The downloads [4] show that Python 2.7 is still the majority of >>> users but> > this is declining quickly and by 2020 the tables will have >>> turned. >>> We need> > to be ready for that. >>> >>> *3. Change our positioning on dependencies of 3rd party packages* >>> >>> We aren't seeing enough community contribution to keep up with >>> the rapid> > pace at which Microsoft, Amazon and Google are changing their >>> APIs. >>> Google> > have been contributing to their driver for years. We haven't seen >>> that from> > either Amazon or Microsoft. Alibaba have contributed to theirs, >>> many other> > cloud providers have contributed, but it seems to be after >>> APIs are> > changed, not in advance. >>> >>> >>> * 1. Docker is another example, that API changes almost every >>> month. The> > driver we have is unstable and doesn't support API versioning >>> correctly. *> > Please consider these options, >>> >>> I would like to hear from users how you are currently using Apache >>> Libcloud> > and how you are using it >>> >>> NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly> > >>> because they represent +80% of the cloud market [3]. This is a >>> community> > developed project, has no affiliation to those vendors and >>> these >>> are my> > opinions, not those of the project. >>> >>> [1] https://www.amazonaws.cn/en/products/ >>> [2] >>> http://www.computerweekly.com/news/450280991/IDC-research-predicts-gradual-decline-in-on-premise-hardware-spend-as-cloud-adoption-rises> >>> > [3] >>> https://www.crn.com.au/news/microsoft-ate-into-aws-market-share-in-q4-481240> >>> > [4] >>> https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7-vPIYPer2VFtSo_Q/edit?usp=sharing> >
