+ a million to John, and I’m actually in the same boat. I stepped away due to a conflict, though I technically remain on the PMC (it seems). Every time I’ve wanted to come back the current design stops me, and that’s not speaking negatively, just that I have a difficult time approaching modern libcloud. I work for a company much more supportive of open source now, and I’d love to put my volunteer hours into libcloud again. I agree with everything architectural discussed in this thread, and moving that direction would rouse me from hibernation. What would I add to the discussion is that the individual layers should strive for requests-level simplicity and thought. I agree the ex_ approach is offputting. I think Python, especially modern Python, is expressive enough to do this aspect well. A data-driven approach to generating bindings is really, really interesting. If that ends up being too bold for libcloud, I might help explore that separately. That’s a really interesting idea. -J
On Thu, Mar 8, 2018, at 19:18, John Carr wrote: > Hi Anthony, > > I used to be an active contributor to libcloud, and am > technically still> a committer although I don’t think I’ve been active at all > since you > stepped into the fold. I just wanted to start by saying thanks > for your> hard work! > > I fully support your proposal. It will take a while to be ready > to fully> exploit py3, we should get started. > > For me what went wrong in the current codebase is that the common > interfaces weren’t common enough. Even within the simple use > cases there> were often paper cuts (inconsistencies in exceptions raised). > And the> ex_ approach was off-putting too. > > Ultimately I ended up only targeting AWS and it was a no > brainer to use> botocore. > > An asyncio libcloud could be really compelling - but it should learn a> few > tricks from boto/botocore. > > I think there should be a low level API that targets the actual API as> > closely as possible. It should provide some commonality - > primitives for> waiting, paginating, retrying, etc. Maybe some well known > exceptions. It> should probably map to basic types rather than exposing XML. > Fundamentally AWS APIs are inconsistent but these botocore > utilities and> patterns make up for it mostly. > > High level abstractions should be built on top of this layer > and only do> what is actually common between APIs - no iffy ex_ params. > > I wonder if we can make libcloud more attractive to providers > wanting to> write a python sdk? > > One idea I had a long time ago was to see how well the botocore data > driven model could extend to other providers. I don’t really think we> have > the resources to do this. But the thinking was we could generate> requests, > asyncio and twisted bindings from the same data and > potentially share the data with an node cloud driver for example. > > Cheers > John > > > On 8 Mar 2018, at 23:16, anthony shaw <anthonys...@apache.org> > > wrote:> > > > Hi All, > > > > The "cloud market" when Apache Libcloud was conceived in 2010 > > is very> > different to how it looks today, some trends we are seeing > > > > 1- IaaS (our compute API) is one of many features in public clouds. > > Amazon,> > Azure and GCP have 100's of individual services now [1]. Text-to- > > speech,> > functions, automation, API gateways, it's impossible to keep up > > 2- Private Clouds have seen a continued decline but are still > > popular [2]> > 3- The advent of containers means it is now "easier" to > > deploy an > > application to multiple clouds > > 4- The big 3 public clouds, Amazon, Azure and Google make up most > > of the> > cloud market [3] if you compare Apache Libcloud downloads with > > boto (the> > native AWS Python client) downloads, it's a massive delta. > > Boto is > > in the> > top 10 most popular PyPi packages > > > > In terms of users, I've pulled 3 snapshots of PyPi downloads, > > January 2016,> > 2017 and 2018 [4] > > Annual downloads of Apache-Libcloud have seen a slight increase, > > but the> > 2016 - 2,778,687 > > 2017 - 2,958,591 > > > > Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is > > a huge> > drop. [4] > > > > *I would like to propose a drastic (depending on your perspective) > > plan to> > take Apache Libcloud through to 2020* > > > > *1. Focus on specific use cases* > > > > Lack of consistency between implementations of the base class > > means in> > practicality it's difficult to have abstracted applications. > > > > Apache Libcloud should (imo) come with a toolbox of utilities > > to both> > demonstrate and validate cloud abstraction use cases, such as: > > - Migrating storage objects from Cloud X to Cloud Y > > - Splitting an application workload across multiple clouds > > > > *2. Improve performance by adopting asyncio* > > > > In almost all use cases, Libcloud would benefit from non-blocking > > calls.> > Listing VM's requires multiple calls for the pages, uploading > > storage> > objects can be done in multiple futures, deleting DNS records > > would be> > better done in async. > > > > I'm suggesting we introduce a Python 3.5+ only API, move to > > requests-futures or aiohttp for the base HTTP client. Yes, *I am > > suggesting> > we drop Python 2 support in the future*. > > > > I've been researching how we could make this switch without breaking > > 64% of> > our users.. > > Pip now has a way to choose versions based on Python runtimes > > https://hackernoon.com/phasing-out-python-runtimes-gracefully-956f112f33c4> > > > We could have apache-libcloud 3+ for Python 3.5 users and then > > maintain> > 2.3 patches for Python 2.7 and 3.4 users. > > > > I think we can come up with a way of continuing to maintain the > > existing> > code base for 2.7 users but move forward with a new API for > > async and> > Python 3.5+ users. > > > > The downloads [4] show that Python 2.7 is still the majority of > > users but> > this is declining quickly and by 2020 the tables will have > > turned. > > We need> > to be ready for that. > > > > *3. Change our positioning on dependencies of 3rd party packages* > > > > We aren't seeing enough community contribution to keep up with > > the rapid> > pace at which Microsoft, Amazon and Google are changing their > > APIs. > > Google> > have been contributing to their driver for years. We haven't seen > > that from> > either Amazon or Microsoft. Alibaba have contributed to theirs, > > many other> > cloud providers have contributed, but it seems to be after > > APIs are> > changed, not in advance. > > > > > > * 1. Docker is another example, that API changes almost every > > month. The> > driver we have is unstable and doesn't support API > > versioning > > correctly. *> > Please consider these options, > > > > I would like to hear from users how you are currently using Apache > > Libcloud> > and how you are using it > > > > NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly> > > > because they represent +80% of the cloud market [3]. This is a > > community> > developed project, has no affiliation to those vendors and > > these > > are my> > opinions, not those of the project. > > > > [1] https://www.amazonaws.cn/en/products/ > > [2] > > http://www.computerweekly.com/news/450280991/IDC-research-predicts-gradual-decline-in-on-premise-hardware-spend-as-cloud-adoption-rises> > > > [3] > > https://www.crn.com.au/news/microsoft-ate-into-aws-market-share-in-q4-481240> > > > [4] > > https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7-vPIYPer2VFtSo_Q/edit?usp=sharing>
