Hmm, I would actually go further and rebuild libcloud as a queue-oriented multithreaded multiprocessing asynchronous server.
Samuel Marks http://linkedin.com/in/samuelmarks On Fri, Mar 9, 2018 at 7:23 PM, John Carr <john.c...@unrouted.co.uk> wrote: > On 9 Mar 2018, at 01:18, Jed Smith <j...@jedsmith.org> wrote: > > > > A data-driven approach to generating bindings is really, really > > interesting. If that ends up being too bold for libcloud, I might help > > explore that separately. That’s a really interesting idea. > > -J > > > > Here’s a prototype I started working on in 2015. I didn’t get very far, > but if you check out the tests I did have one code base supporting sync and > async. I had a middleware-esque approach where the data described which > serialisers, request signers, etc to mash together. > > https://github.com/Jc2k/libcloudcore <https://github.com/Jc2k/libcloudcore > > > > John > > > > > On Thu, Mar 8, 2018, at 19:18, John Carr wrote: > >> Hi Anthony, > >> > >> I used to be an active contributor to libcloud, and am > >> technically still> a committer although I don’t think I’ve been active > at all since you > >> stepped into the fold. I just wanted to start by saying thanks > >> for your> hard work! > >> > >> I fully support your proposal. It will take a while to be ready > >> to fully> exploit py3, we should get started. > >> > >> For me what went wrong in the current codebase is that the common > >> interfaces weren’t common enough. Even within the simple use > >> cases there> were often paper cuts (inconsistencies in exceptions > raised). And the> ex_ approach was off-putting too. > >> > >> Ultimately I ended up only targeting AWS and it was a no > >> brainer to use> botocore. > >> > >> An asyncio libcloud could be really compelling - but it should learn a> > few tricks from boto/botocore. > >> > >> I think there should be a low level API that targets the actual API as> > closely as possible. It should provide some commonality - > >> primitives for> waiting, paginating, retrying, etc. Maybe some well > known > >> exceptions. It> should probably map to basic types rather than exposing > XML. > >> Fundamentally AWS APIs are inconsistent but these botocore > >> utilities and> patterns make up for it mostly. > >> > >> High level abstractions should be built on top of this layer > >> and only do> what is actually common between APIs - no iffy ex_ params. > >> > >> I wonder if we can make libcloud more attractive to providers > >> wanting to> write a python sdk? > >> > >> One idea I had a long time ago was to see how well the botocore data > >> driven model could extend to other providers. I don’t really think we> > have the resources to do this. But the thinking was we could generate> > requests, asyncio and twisted bindings from the same data and > >> potentially share the data with an node cloud driver for example. > >> > >> Cheers > >> John > >> > >>> On 8 Mar 2018, at 23:16, anthony shaw <anthonys...@apache.org> > >>> wrote:> > > >>> Hi All, > >>> > >>> The "cloud market" when Apache Libcloud was conceived in 2010 > >>> is very> > different to how it looks today, some trends we are seeing > >>> > >>> 1- IaaS (our compute API) is one of many features in public clouds. > >>> Amazon,> > Azure and GCP have 100's of individual services now [1]. > Text-to- > >>> speech,> > functions, automation, API gateways, it's impossible to > keep up > >>> 2- Private Clouds have seen a continued decline but are still > >>> popular [2]> > 3- The advent of containers means it is now "easier" to > deploy an > >>> application to multiple clouds > >>> 4- The big 3 public clouds, Amazon, Azure and Google make up most > >>> of the> > cloud market [3] if you compare Apache Libcloud downloads > with > >>> boto (the> > native AWS Python client) downloads, it's a massive > delta. Boto is > >>> in the> > top 10 most popular PyPi packages > >>> > >>> In terms of users, I've pulled 3 snapshots of PyPi downloads, > >>> January 2016,> > 2017 and 2018 [4] > >>> Annual downloads of Apache-Libcloud have seen a slight increase, > >>> but the> > 2016 - 2,778,687 > >>> 2017 - 2,958,591 > >>> > >>> Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is > >>> a huge> > drop. [4] > >>> > >>> *I would like to propose a drastic (depending on your perspective) > >>> plan to> > take Apache Libcloud through to 2020* > >>> > >>> *1. Focus on specific use cases* > >>> > >>> Lack of consistency between implementations of the base class > >>> means in> > practicality it's difficult to have abstracted > applications. > >>> > >>> Apache Libcloud should (imo) come with a toolbox of utilities > >>> to both> > demonstrate and validate cloud abstraction use cases, such > as: > >>> - Migrating storage objects from Cloud X to Cloud Y > >>> - Splitting an application workload across multiple clouds > >>> > >>> *2. Improve performance by adopting asyncio* > >>> > >>> In almost all use cases, Libcloud would benefit from non-blocking > >>> calls.> > Listing VM's requires multiple calls for the pages, uploading > >>> storage> > objects can be done in multiple futures, deleting DNS > records > >>> would be> > better done in async. > >>> > >>> I'm suggesting we introduce a Python 3.5+ only API, move to > >>> requests-futures or aiohttp for the base HTTP client. Yes, *I am > >>> suggesting> > we drop Python 2 support in the future*. > >>> > >>> I've been researching how we could make this switch without breaking > >>> 64% of> > our users.. > >>> Pip now has a way to choose versions based on Python runtimes > >>> https://hackernoon.com/phasing-out-python-runtimes- > gracefully-956f112f33c4> > We could have apache-libcloud 3+ for Python > 3.5 users and then > >>> maintain> > 2.3 patches for Python 2.7 and 3.4 users. > >>> > >>> I think we can come up with a way of continuing to maintain the > >>> existing> > code base for 2.7 users but move forward with a new API for > >>> async and> > Python 3.5+ users. > >>> > >>> The downloads [4] show that Python 2.7 is still the majority of > >>> users but> > this is declining quickly and by 2020 the tables will > have turned. > >>> We need> > to be ready for that. > >>> > >>> *3. Change our positioning on dependencies of 3rd party packages* > >>> > >>> We aren't seeing enough community contribution to keep up with > >>> the rapid> > pace at which Microsoft, Amazon and Google are changing > their APIs. > >>> Google> > have been contributing to their driver for years. We haven't > seen > >>> that from> > either Amazon or Microsoft. Alibaba have contributed to > theirs, > >>> many other> > cloud providers have contributed, but it seems to be > after APIs are> > changed, not in advance. > >>> > >>> > >>> * 1. Docker is another example, that API changes almost every > >>> month. The> > driver we have is unstable and doesn't support API > versioning > >>> correctly. *> > Please consider these options, > >>> > >>> I would like to hear from users how you are currently using Apache > >>> Libcloud> > and how you are using it > >>> > >>> NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly> > > because they represent +80% of the cloud market [3]. This is a > >>> community> > developed project, has no affiliation to those vendors > and these > >>> are my> > opinions, not those of the project. > >>> > >>> [1] https://www.amazonaws.cn/en/products/ > >>> [2] > >>> http://www.computerweekly.com/news/450280991/IDC-research- > predicts-gradual-decline-in-on-premise-hardware-spend-as- > cloud-adoption-rises> > [3] > >>> https://www.crn.com.au/news/microsoft-ate-into-aws-market- > share-in-q4-481240> > [4] > >>> https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7- > vPIYPer2VFtSo_Q/edit?usp=sharing> > > > >
