Hi Ignasi, I have raised a pull request for async put blob api in Jclouds git. For reference https://github.com/jclouds/jclouds/pull/1114
-- Thanks -- Kishore On 15/06/17, 10:53 AM, "Battula Kishore" <batt...@adobe.com.INVALID> wrote: >Hi Ignasi, >Thanks. I will cleanup and will raise a pull request directly to Jclouds in >couple of days. > >-- Thanks >-- Kishore > > > > > > > > > > >On 14/06/17, 3:19 PM, "Ignasi Barrera" <n...@apache.org> wrote: > >>I participated in both reviews and I'm happy with the status of the >>code in the PRs. I'd say you clean things up and open directly a PR to >>the jclouds repo. If there is anything to discuss further, we can do >>it there. As mentioned else-thread, the implementation should add >>async support for all HTTP drivers that support it, so I'd like to see >>the changes tot he OkHttp driver too in the PR. >> >>Keep up your fantastic job! >> >> >>I. >> >>On 13 June 2017 at 11:27, Battula Kishore <batt...@adobe.com.invalid> wrote: >>> Hi Ignasi, >>> >>> Just to do a quick walkthrough what I have already did towards the async >>> approach. >>> Here is the initial pull request for AsyncPOC >>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSpandanThakur%2Fjclouds%2Fpull%2F2&data=02%7C01%7C%7Cc47d52d09b5846a8a9e008d4b30aa82e%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636330305747272162&sdata=Z0JPWGfCL9m8JQpYGlXQnJNp2WHJeWPBJp4WCf4fHVs%3D&reserved=0 >>> This pull request has gone through couple of reviews from the community >>> members and I have made necessary changes. One last thing I was asked is to >>> implement AsyncBlobInterface instead of adding it to existing >>> BlobInterface. I have done it in this pull request. >>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSpandanThakur%2Fjclouds%2Fpull%2F4&data=02%7C01%7C%7Cc47d52d09b5846a8a9e008d4b30aa82e%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636330305747272162&sdata=tkq5epIrtaoLdILEV6A3u8q%2BiZpx0wSfenzZFMY%2BO4Y%3D&reserved=0 >>> >>> I am looking for some inputs on how to proceed from here to get it into a >>> state where it can be merged. Then after the I can follow the same steps >>> for other API's as well once I get familiar with the process for one API. >>> Looking forward for your inputs. >>> >>> --Thanks >>> --Kishore >>> >>> -----Original Message----- >>> From: Ignasi Barrera [mailto:n...@apache.org] >>> Sent: Monday, June 12, 2017 11:47 PM >>> To: dev@jclouds.apache.org >>> Subject: Re: Adding async support for Jclouds library >>> >>> Hi! >>> >>> I think that given the output from your benchmarks it is OK to proceed with >>> the async approach. The asnyc and OutputStream versions are not exclusive >>> and the benchmarks show no reason for not starting with implementation of >>> the async feature. >>> >>> For me, let's go with the next step and open a pull request with the PoC >>> code/continue the discussion/whatever plan you had in mind :) >>> >>> >>> I. >>> >>> On Jun 12, 2017 4:48 AM, "Battula Kishore" <batt...@adobe.com.invalid> >>> wrote: >>> >>> Hi Andrew, Ignasi >>> >>> Checking once again any update on this? >>> >>> -- Thanks >>> -- Kishore >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On 06/06/17, 12:10 PM, "Ignasi Barrera" <n...@apache.org> wrote: >>> >>>>Hi, >>>> >>>>I can pick it up, but I think we should have some more feedback from >>>>Andrew first, at least to see if he still has some concerns about the >>>>implementation. >>>>I do like the current async approach but it wouldn't be right to move >>>>forward without the OK of the team member that had concerns and has >>>>been more involved in the design discussion. >>>> >>>> >>>>I, >>>> >>>>On 6 June 2017 at 08:28, Felix Meschberger <fmesc...@adobe.com.invalid> >>> wrote: >>>>> Hi Jclouds dev >>>>> >>>>> Anyone else who could pick up instead of AndrewG ? >>>>> >>>>> It would be great to be able to have this support added to JClouds. >>>>> >>>>> Thanks >>>>> Felix >>>>> >>>>>> Am 06.06.2017 um 05:49 schrieb Andrew Phillips <andr...@apache.org>: >>>>>> >>>>>> Hi Kishore >>>>>> >>>>>> Andrew G. is travelling for most of June, as far as I understand, so >>> will likely be a bit slower to respond. Thanks for your patience! >>>>>> >>>>>> ap >>>>>> >>>>>> On 2017-06-05 23:42, Battula Kishore wrote: >>>>>>> Hi Andrew, >>>>>>> Any update on this? >>>>>>> -- Thanks >>>>>>> -- Kishore >>>>>>> On 29/05/17, 1:06 PM, "Battula Kishore" <batt...@adobe.com.INVALID> >>> wrote: >>>>>>>> Hi Andrew, >>>>>>>> Thanks andrew for the quick response. Here is the GitHub repo and >>> instructions on how to run the tests >>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fna01.safelinks&data=02%7C01%7C%7C6ad1a11f02da4249a9df08d4b1bf4312%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636328882406496498&sdata=VxmspqySHr7onsxPr75PVxGzQgfEVVJoITrOfy5BdHU%3D&reserved=0. >>> protection.outlook.com/?url=https%3A%2F%2Fgithub.com% >>> 2Fkishore25kumar%2Fs3proxy-async-test-setup&data=02%7C01%7C% >>> 7C8969ed65c6a749aebfa608d4a66583cc%7Cfa7b1b5a7b34438794aed2c178de >>> cee1%7C0%7C0%7C636316402308126519&sdata=0YESeCSlmChUOicpe2m9SYC1WaB7m% >>> 2B7hIB6o%2BTOp1Jo%3D&reserved=0. The READE.md also has the repo details of >>> s3proxy as well as jclouds implementation. Hope this helps? Let me know if >>> you need anything else? >>>>>>>> In the mean time you review these results if you can let me know >>>>>>>> the >>> design review process I can be prepared for that. >>>>>>>> -- Thanks >>>>>>>> -- Kishore >>>>>>>> On 26/05/17, 12:40 PM, "Andrew Gaul" <g...@apache.org> wrote: >>>>>>>>> Kishore, these are promising results! I reformatted the most >>> important >>>>>>>>> rows which show a 2x improvement in throughput and latency: >>>>>>>>> 10 10,000 Async Http Lib 209 282 48 >>>>>>>>> 10 10,000 OutputStream 392 542 25 >>>>>>>>> Can you share the implementation and include instructions on how >>>>>>>>> to replicate these tests? >>>>>>>>> On Wed, May 24, 2017 at 05:50:52AM +0000, Battula Kishore wrote: >>>>>>>>>> Hi, >>>>>>>>>> This is Kishore who is working on async poc using mail id( >>> kishore25ku...@gmail.com<mailto:kishore25ku...@gmail.com>). I work at adobe >>> and we wanted to implement async support for jclouds library and contribute >>> it back. >>>>>>>>>> From the last discussion I was asked to get the performance >>>>>>>>>> numbers >>> for the two approaches. >>>>>>>>>> Approach 1: Using Http Async Library Approach 2: Using >>>>>>>>>> Outputstream Test setup: >>>>>>>>>> 1. Both the s3 proxy server and test runner are running in >>> same Docker container in azure west-us region. >>>>>>>>>> 2. Azure storage account is also residing in same west-us >>> region. >>>>>>>>>> 3. A bucket is prepopulated with 100,000 files, each file of >>> 1 MB size before test start. >>>>>>>>>> 4. The test runner sends unique requests to s3proxy to >>> download files. >>>>>>>>>> Virtual Machine spec: CPU - 8 cores, Memory - 28 GB (Standard_D4 >>> Azure machine) >>>>>>>>>> S3proxy is running with 1 jetty worker thread in all the scenarios. >>> The payload size used is 1 MB file. Here are the performance numbers. >>>>>>>>>> Test Runner Threads >>>>>>>>>> Iteration Per thread >>>>>>>>>> Approach >>>>>>>>>> Avg response time (ms) >>>>>>>>>> 99%tile time (ms) >>>>>>>>>> Throughput >>>>>>>>>> (Requests / sec) >>>>>>>>>> 1 >>>>>>>>>> 10,000 >>>>>>>>>> Async Http Lib >>>>>>>>>> 45 >>>>>>>>>> 87 >>>>>>>>>> 22 >>>>>>>>>> 5 >>>>>>>>>> 10,000 >>>>>>>>>> Async Http Lib >>>>>>>>>> 107 >>>>>>>>>> 159 >>>>>>>>>> 47 >>>>>>>>>> 10 >>>>>>>>>> 10,000 >>>>>>>>>> Async Http Lib >>>>>>>>>> 209 >>>>>>>>>> 282 >>>>>>>>>> 48 >>>>>>>>>> 1 >>>>>>>>>> 10,000 >>>>>>>>>> OutputStream >>>>>>>>>> 41 >>>>>>>>>> 85 >>>>>>>>>> 24 >>>>>>>>>> 5 >>>>>>>>>> 10,000 >>>>>>>>>> OutputStream >>>>>>>>>> 190 >>>>>>>>>> 283 >>>>>>>>>> 26 >>>>>>>>>> 10 >>>>>>>>>> 10,000 >>>>>>>>>> OutputStream >>>>>>>>>> 392 >>>>>>>>>> 542 >>>>>>>>>> 25 >>>>>>>>>> Summary: Under load Http Async Library approach is providing >>>>>>>>>> more >>> throughput compared to Output stream approach. >>>>>>>>>> Both the approaches improve performance. The output stream >>>>>>>>>> approach >>> can be used along with Http Async library approach which is giving around >>> (3-5 ms) improvement in latency. >>>>>>>>>> Each approach is independent development. At this point I am >>>>>>>>>> keen >>> to take up Http Async Library development. >>>>>>>>>> -- Thanks >>>>>>>>>> -- Kishore >>>>>>>>> -- >>>>>>>>> Andrew Gaul >>>>>>>>> https://na01.safelinks.protection.outlook.com/?url= >>> http%3A%2F%2Fgaul.org%2F&data=02%7C01%7C%7C8690ee3e00bc4e3da0e608d4a406565e% >>> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636313794511467457&sdata=XQ% >>> 2FshVjdqC3KiVEuyH6%2FJvmDN5DHBmS0kIBx98V89KY%3D&reserved=0 >>>>>
