Hi Please find my answer below On Apr 22, 2014 10:49 AM, "Jasdeep Hundal" <[email protected]> wrote: > > Hey Sumit, > > I have a couple more questions that might help clarify the situation: > > 1. Are you running the stability test as a single long running Java process > (that just keeps cycling through the 10 uploads/gets/deletes)? >
Yes. But this process has threads. > 2. Are you always running the test in the same container, or are you > creating new containers for each test iteration? > No, I am doing roundrobin in 1000 containers > 3. If the answer to #2 is is that the test runs in a single container, how > many objects does that container currently have? > 0 in ideal case. But as I m facing 409 delete fail also... so there are some objects on each container in hundreds only. > It may also help to time each of the individual blobstore actions as you > run the test to see if any particular one is slowing down. > Even indivitual put and del time increase over the time. > Jasdeep > > > On Mon, Apr 21, 2014 at 6:21 PM, Sumit Gaur <[email protected]> wrote: > > > hi Shri, > > Please find answers below > > > > On Tue, Apr 22, 2014 at 9:23 AM, Shrinand Javadekar < > > [email protected] > > > wrote: > > Few more questions to try and understand this better: > > > > 1) On the Swift instance you are using, how many replicas do you have? > > > > 3 replica > > > > 2) Also, how are you using the curl command in the shell script? > > > > send below command in backgroud for 10 iterations and wait similiar to the > > 10 threads in jclouds. > > > > curl -X PUT -i -T 100k -H "X-Auth-Token: $OS_AUTH_TOKEN" > > http:// > > > > $PROXY_LOCAL_NET_IP:80/v1/AUTH_${KEYSTONE_ID}/zest1-${cn}/zest1-${k}-${i}-${j}.txt > > curl -X DELETE -i -H "X-Auth-Token: $OS_AUTH_TOKEN" http:// > > > > $PROXY_LOCAL_NET_IP:80/v1/AUTH_${KEYSTONE_ID}/zest1-${cn}/zest1-${k}-${i}-${j}.txt > > > > I > > think the shell script and jclouds-with-10-parallel-threads may not be > > doing the same amount of work. In 20 hours jclouds might be doing much > > more work than the shell script. If you let the shell script also go > > upto that point, it might see failures too. Do you know how many > > PUT-GET-DEL operations have been performed when you start seeing the > > 409 errors. > > > > Actually 409 errors are coming since the start of the test but TPS start > > degrading after 24-48 hours. > > On Apr 22, 2014 9:23 AM, "Shrinand Javadekar" <[email protected]> > > wrote: > > > > > Few more questions to try and understand this better: > > > > > > 1) On the Swift instance you are using, how many replicas do you have? > > > 2) Also, how are you using the curl command in the shell script? I > > > think the shell script and jclouds-with-10-parallel-threads may not be > > > doing the same amount of work. In 20 hours jclouds might be doing much > > > more work than the shell script. If you let the shell script also go > > > upto that point, it might see failures too. Do you know how many > > > PUT-GET-DEL operations have been performed when you start seeing the > > > 409 errors. > > > > > > -Shri > > > > > > > > > On Mon, Apr 21, 2014 at 4:55 PM, Sumit Gaur <[email protected]> > > wrote: > > > > FYI ..This is block of code ..... also I am using jclouds 1.7.1 > > (Stable > > > > branch) > > > > try { > > > > String key = "objkey" + UUID.randomUUID(); > > > > Blob blob = > > > > Example9.blobStore.blobBuilder(key).payload(Example9.file).build(); > > > > > > Example9.blobStore.putBlob(Example9.containerName+count, > > > > blob); > > > > > > Example9.blobStore.getBlob(Example9.containerName+count, > > > > key); > > > > > > > Example9.blobStore.removeBlob(Example9.containerName+count, > > > > key); > > > > } catch (Exception ace) { > > > > System.out.println("Request failed for objkey " + key > > + " > > > > " + ace); > > > > } > > > > > > > > > > > > > > > > On Tue, Apr 22, 2014 at 8:32 AM, Sumit Gaur <[email protected]> > > > wrote: > > > > > > > >> Hi Shri, > > > >> Thanks for paying attention to it, Please find my answers below:- > > > >> > > > >> > > > >> On Tue, Apr 22, 2014 at 2:31 AM, Shrinand Javadekar < > > > >> [email protected]> wrote: > > > >> > > > >>> Sumit, > > > >>> > > > >>> I realize that you had sent out a similar email sometime ago about > > > >>> performance degradation. I'm not sure if anyone has run these types > > of > > > >>> long running experiments with jclouds. So this may be a first. > > > >>> > > > >> Tried to debug it in last 2 weeks without success. Want to understand > > > more > > > >> how jclouds code handle this use case or any pointers that this is a > > > >> problematic use case would help > > > >> > > > >>> > > > >>> The 409 status is returned because of a conflict [1]. Are you sure > > you > > > >>> didn't have two or more threads trying to delete the same object? > > > >>> > > > >> No two threads share the same object key in my programme (String key = > > > >> "objkey" + UUID.randomUUID();). It is some kind of race between PUT > > and > > > >> DEL call . If I put say 10 ms sleep between call then there is no 409 > > > error. > > > >> > > > >> > > > >>> Also, I see that that 409 is returned by Swift if you try to delete a > > > >>> container that isn't empty[2]. Is that something your test code > > > >>> could've tried? > > > >>> > > > >> I am trying to delete objects .. not containers. > > > >> > > > >>> > > > >>> When you say there was a similar test you're trying with curl, are > > you > > > >>> using the curl command-line utility or the libcurl library? > > > >> > > > >> curl command in shell script with for loops. > > > >> > > > >> > > > >>> How are > > > >>> you specifying the number of threads to use and what object each > > > >>> thread should get/put/delete? > > > >>> > > > >> > > > >> It is a java test programme using ThreadPoolExecutor. Somthing > > similiar > > > as > > > >> here > > > >> > > > >> > > > > > http://www.javacodegeeks.com/2013/01/java-thread-pool-example-using-executors-and-threadpoolexecutor.html > > > >> > > > >> Object is a 5KB file. with key = "objkey" + UUID.randomUUID(); with > > > Pool > > > >> of 10 threads. > > > >> > > > >> > > > >> Hope this would give a good inside. Let me know if you get any problem > > > >> here. > > > >> > > > >> > > > >>> > > > >>> Thanks. > > > >>> -Shri > > > >>> > > > >>> [1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html > > > >>> [2] https://bugs.launchpad.net/horizon/+bug/1096084 > > > >>> > > > >>> On Sun, Apr 20, 2014 at 5:55 PM, Sumit Gaur <[email protected]> > > > wrote: > > > >>> > Hi > > > >>> > I using jclouds lib integrated with Openstack Swift+ keystone > > > >>> combinaiton. > > > >>> > Things are working fine except stability test. After 20-30 hours of > > > test > > > >>> > jclouds/SWIFT start degrading in TPS and keep going down over the > > > time. > > > >>> > > > > >>> > 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads. > > > >>> > 2) I am getting a lot of 409 and DEL failure for the as response > > too > > > >>> from > > > >>> > SWIFT. > > > >>> > 3) Direct similiar test from curl does not show much impact and TPS > > > >>> remain > > > >>> > constant. > > > >>> > > > > >>> > Can sombody help me wht is going wrong here ? > > > >>> > > > > >>> > Thanks > > > >>> > sumit > > > >>> > > > >> > > > >> > > > > >
