The regenerated provider tables? I had a merge conflict (well, many merge conflicts) on the way and re-running the table generator is the safest way to resolve those ones.
They need running again, since the AWS regions are a dictionary, the ordering is random. On Tue, Jan 10, 2017 at 7:37 PM, Allard Hoeve <allardho...@gmail.com> wrote: > Hey Anthony, > > Wow, that is a lot of work :) Thanks for that. > > I started reviewing the branch, but it's 3k+, 3k- and I immediately ran > into the problem of "reviewing a patch that's too big". For example > <https://github.com/apache/libcloud/pull/923/files#diff-f9b7105b8fba7a7f1a0ddec6ef14c8feR413>: > why is that added in? Seems unrelated, but you must've done that for a > reason. > > So before the review devolves into a bunch of nitpicks, how do you suppose > we'd review most efficiently? > > Maybe you can go through the PR and comment yourself on why certain things > are as they are? > > Best, > > Allard > > > On Mon, Jan 9, 2017 at 5:40 AM anthony shaw <anthony.p.s...@gmail.com> > wrote: > >> This pull-request is FINALLY finished! It's taken me the best part of >> a year to complete it. >> >> https://github.com/apache/libcloud/pull/923 >> >> This last round of changes I had to completely rewrite the Storage API >> base classes. I've been testing using my GCP account and uploading and >> downloading files and it all looks good now. I've gone back and tested >> a bunch of random other drivers like GoDaddy and Dimension Data for >> which I have accounts for. >> >> I'm going to merge it and raise another PR this week with an >> integration suite module. This will have a driver and a FLASK web app >> with a tox definition to test the libcloud library from end to end. >> The way the httplib_ssl module is mocked out in our unit tests leaves >> a lot of room for mistakes. >> >> Things that visually don't make much sense but I don't have an account to >> test >> - The code for Azure Blob leases looked fragile. This really needs >> testing properly >> - Multipart uploads should work for storage APIs, S3 has a custom API >> that is now disabled >> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingRESTAPImpUpload.html >> - The Aliyun OSS driver had some really visible bugs in it, I doubt >> the existing driver works. We need integration testers for it. >> >> >> >> On Fri, Jan 6, 2017 at 8:05 PM, Samia, Michel <msa...@netsuite.com> wrote: >> > <html><bodyFor multipart upload you can use >> https://pypi.python.org/pypi/requests-toolbelt >> > >> > -----Original Message----- >> > From: anthony shaw [mailto:anthony.p.s...@gmail.com] >> > Sent: Friday, January 6, 2017 6:07 AM >> > To: dev <dev@libcloud.apache.org> >> > Subject: Re: [dev] [DISCUSS] Using requests instead of httplib >> > >> > Good news is I figured out a way of implementing the upload >> functionality using the requests package. >> > >> > >> https://github.com/apache/libcloud/pull/923/commits/5e04dbce554830eca3f9812272076a2fbdbe7cdc >> > >> > I've tested it against the Google Cloud Storage account, downloaded and >> uploaded a file using both the direct file_path option and the option >> passing a context manager (IOStream or ByteStream). >> > >> > The bad news is : >> > - The S3 multipart upload I've removed. I don't have time right now to >> implement this feature from scratch >> > - The unit tests are all coupled to the private methods, the call back >> system and a bunch of other bad coupling practices, so they are broken BUT >> it does actually work >> > >> > It's nearly there. >> > >> > Ant >> > >> > On Fri, Jan 6, 2017 at 7:28 AM, anthony shaw <anthony.p.s...@gmail.com> >> wrote: >> >> it's more of an existential question :-) >> >> >> >> The _upload_object method inside the libcloud.storage.base submodule >> >> makes a 'raw' call to the LibcloudConnection, which will send the top >> >> part of the HTTP request then some headers and leave the connection >> >> open (i.e. not read the response). >> >> >> >> Then, depending on the driver, the file and other things, it will >> >> callback one of the methods like _stream_data, which writes directly >> >> to the HTTP session using the `send()` method, which is only available >> >> in httplib. >> >> >> >> httplib is a very low level library, requests is very high level. You >> >> don't get access to the HTTP session directly in requests. >> >> >> >> That means that I would have to throw away the code we already have >> >> (which I am definitely in favour of in the long term since it is >> >> fragile) and replace it with requests' APIs for doing chunked uploads >> >> using file streams. >> >> >> >> It would probably take me another day or two to finish that >> implementation. >> >> >> >> I always preached that you should change 1 thing at a thing, in small >> >> amount, and keep testing. So far this has been more like pulling a >> >> thread on a sweater, I've touched every single file in the code base >> >> practically! >> >> The odds are, I will have missed something. So 2.0.0rc1 (if we do call >> >> it that), despite my best intentions will introduce a new bug just >> >> based on the number of things I have changed. >> >> >> >> On Fri, Jan 6, 2017 at 1:11 AM, Tom Melendez <t...@supertom.com> wrote: >> >>> Hi Anthony, >> >>> >> >>> Nice job getting this going! >> >>> >> >>> Would you mind elaborating on this point? >> >>> "The raw connection still uses httplib. I decided it was too risky to >> >>> swap that for requests' method of uploading files." >> >>> >> >>> Since you're going through the trouble, it would be ideal to go to >> >>> Requests completely. What's blocking us on the upload code >> >>> (Admittedly, I haven't studied the upload code)? Anything the >> community can do to help? >> >>> >> >>> Thanks, >> >>> >> >>> Tom >> >>> >> >>> >> >>> On Thu, Jan 5, 2017 at 3:43 AM, Chris DeRamus >> >>> <chris.dera...@gmail.com> >> >>> wrote: >> >>> >> >>>> For what it's worth my company (DivvyCloud) has been using the good >> >>>> work you've done now for almost six months. We had to make a few >> >>>> tweaks, but the core code contributed has worked flawlessly across >> >>>> AWS, Azure, OpenStack, Google, VMware and more. The only issue I >> >>>> believe that still stands which I've seen is an issue when >> >>>> LIBCLOUD_DEBUG is set to true. Logging doesn't appear to function >> >>>> properly, but that may have been addressed since your initial >> submission last year. >> >>>> >> >>>> Nice work on this and we sincerely appreciate the contribution. >> >>>> >> >>>> On Thu, Jan 5, 2017 at 12:21 AM, anthony shaw >> >>>> <anthony.p.s...@gmail.com> >> >>>> wrote: >> >>>> >> >>>> > That package had a dumb error in it, I've since fixed it and >> >>>> > against a live API (GoDaddy). I've tested the following scenarios >> >>>> > >> >>>> > - Applying a custom proxy via the environment variable >> >>>> > - Using libcloud.security to disable SSL verification >> >>>> > - Using libcloud.security to set a custom CA certificate >> >>>> > - Combining all of those scenarios >> >>>> > - Verification of custom headers applied by the driver using >> >>>> > Charles Proxy and inspecting the HTTP messages manually >> >>>> > - Decoding JSON messages - although this still uses the existing >> >>>> > methods, not the requests own json() decoder. >> >>>> > >> >>>> > IMO, this is ready to merge. I would like to test the raw >> >>>> > connections and file uploads if anyone has an account on one of >> those providers? >> >>>> > >> >>>> > On Thu, Jan 5, 2017 at 8:30 AM, Tomaz Muraus <to...@apache.org> >> wrote: >> >>>> > > Thanks for working on this again! >> >>>> > > >> >>>> > > Once we get a green light from people testing those changes, I >> >>>> > > propose >> >>>> to >> >>>> > > first roll out v2.0.0-rc1 and eventually after we are happy with >> >>>> > > the stability call it v2.0.0. >> >>>> > > >> >>>> > > On Wed, Jan 4, 2017 at 6:20 AM, anthony shaw >> >>>> > > <anthony.p.s...@gmail.com >> >>>> > >> >>>> > > wrote: >> >>>> > > >> >>>> > >> Hi, >> >>>> > >> >> >>>> > >> I tried doing this a year ago but put it in the 'too hard' >> bucket. >> >>>> > >> I've opened a PR (again) replacing the use of httplib with the >> >>>> > >> requests package. >> >>>> > >> >> >>>> > >> The consequences are: >> >>>> > >> - Connection.conn_classes is no longer a tuple, it is >> >>>> > >> Connection.conn_class. There is no separation between a HTTP >> >>>> > >> and HTTPS connection. I could have just hacked around this but >> >>>> > >> I updated all the tests instead >> >>>> > >> - Mock implementations no longer use the tuple as above >> >>>> > >> - We cannot support Python 3.2 officially anymore. Requests >> >>>> > >> does not support 3.2 >> >>>> > >> - The raw connection still uses httplib. I decided it was too >> >>>> > >> risky to swap that for requests' method of uploading files. >> >>>> > >> >> >>>> > >> https://github.com/apache/libcloud/pull/923#partial-pull-mergin >> >>>> > >> g >> >>>> > >> >> >>>> > >> Please remote fetch this branch and test it out on some working >> >>>> > >> code talking to real APIs. Mocks can only go so far. >> >>>> > >> >> >>>> > >> I've uploaded the package here - >> >>>> > >> http://people.apache.org/~anthonyshaw/libcloud/1.5.0.post0/ >> >>>> > >> >> >>>> > >> I would like to get this merged but would like some additional >> >>>> > >> nods before it gets merged into trunk. >> >>>> > >> >> >>>> > >> Ant >> >>>> > >> >> >>>> > >> >>>> >> > >> > >> > NOTICE: This email and any attachments may contain confidential and >> proprietary information of NetSuite Inc. and is for the sole use of the >> intended recipient for the stated purpose. Any improper use or distribution >> is prohibited. If you are not the intended recipient, please notify the >> sender; do not review, copy or distribute; and promptly delete or destroy >> all transmitted information. Please note that all communications and >> information transmitted through this email system may be monitored by >> NetSuite or its agents and that all incoming email is automatically scanned >> by a third party spam and filtering service >> > >> > </body></html> >>
