[openstack-dev] (no subject)
http://bloggsatt.se/wp-admin/css/afternews.php___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [SWIFT] Delete Object imediately after PUT
Hi All, Is it not a correct use case to Delete a object just after Putting it. I saw failures from SWIFT in such scenerio. if it is not a valid scenerio please let me know Why ? Thank sumit ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [SWIFT] Delete Object imediately after PUT
Hi Gil, below are the logs for imediate delete. I am getting 409 : conflict error. I am getting no error in case I delay DELETE a bit. Also these errors are coming for performance test (multiclient doing PUT-GET_GET cycle in loop) *Client Node* Request failed for objkey objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615 org.jclouds.http.HttpResponseException: command: DELETE http://www.example.com:80/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615HTTP/1.1 failed with response: HTTP/1.1 409 Conflict; content: [htmlh1Conflict/h1pThere was a conflict when trying to complete your request./p/html] *Proxy Node * Apr 23 12:00:27 proxy1 swift-proxy allow user with role admin as account admin (txn: txe45b336b0ab44aee8a698-0053572ccb) (client_ip: 10.41.54.27) *Storage Node* Apr 23 12:00:26 object1 container-server 10.41.54.9 - - [23/Apr/2014:03:00:26 +] PUT /vdb1/955/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615 201 - tx3ba78bc8924349d18f7af-0053572ccc PUT http://www.example.com/vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615; obj-server 17908 0.0006 Apr 23 12:00:26 object1 object-server 10.41.54.21 - - [23/Apr/2014:03:00:26 +] PUT /vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615 201 - PUT http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615; tx3ba78bc8924349d18f7af-0053572ccc proxy-server 23927 0.0113 Apr 23 12:00:26 object1 object-server 10.41.54.19 - - [23/Apr/2014:03:00:26 +] DELETE /vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615 409 - DELETE http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615; txe45b336b0ab44aee8a698-0053572ccb proxy-server 10071 0.0003 Thanks sumit On Thu, Apr 24, 2014 at 6:49 PM, Gil Vernik g...@il.ibm.com wrote: What failures you saw? Can you please provide logs? Thanks, Gil. From:Sumit Gaur sumitkg...@gmail.com To:OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org, Date:24/04/2014 11:34 AM Subject:[openstack-dev] [SWIFT] Delete Object imediately after PUT -- Hi All, Is it not a correct use case to Delete a object just after Putting it. I saw failures from SWIFT in such scenerio. if it is not a valid scenerio please let me know Why ? Thank sumit___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [SWIFT] Delete operation problem
Hi clay, Thanks for responding , first about setup , it is multi node env, and problem is with delete object . I checked proxy and storage nodes clock, they are in sync. Also storage node run only main daemons I.e. no auditor, and others. I traced the 409 call and it is coming from object server. I am generating random id for key so no reuse of same name. But after one put(t1) next is del(t1).is this could be a problem...I think they are sync calls. Regards Sumit On Apr 23, 2014 5:14 AM, Clay Gerrard clay.gerr...@gmail.com wrote: 409 on DELETE (object?) is a pretty specific error. That should mean that the timestamp assigned to the delete is earlier than the timestamp of the data file. Most likely mean that you're getting some time-drift on your proxies (but that assumes multi-node) or maybe that you're reusing names between threads and your object server's see PUT(ts1) PUT(ts3) DELETE(ts2) - but that'd be a pretty tight race... Should all be logged - try and find a DELETE that went 409 and trace the transaction id. On Mon, Apr 21, 2014 at 5:54 AM, taurus huang huanggeng.8...@gmail.com wrote: Please provide the log file： /var/log/swift/swift.log AND /var/log/keystone/keystone.log On Mon, Apr 21, 2014 at 11:55 AM, Sumit Gaur sumitkg...@gmail.com wrote: Hi I using jclouds lib integrated with Openstack Swift+ keystone combination. Things are working fine except stability test. After 20-30 hours of test jclouds/SWIFT start degrading in TPS and keep going down over the time. 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads. 2) I am getting a lot of 409 and DEL failure for the response too from SWIFT. Can sombody help me what is going wrong here ? Thanks sumit ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [SWIFT] Delete operation problem
Further to above details please find related logs ... *object server* 4:07:21:20 +] DELETE /vdb1/844/AUTH_215c458021024a8c87471157a7040644/suntmp7/objkeye385cf9c-ad96-423f-8c8d-94d001631f8b 409 - DELETE http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/suntmp7/objkeye385cf9c-ad96-423f-8c8d-94d001631f8b; txcb3b4b47278e483ca560e-005350d271 proxy-server 9619 0.0004 or [22/Apr/2014:08:30:07 +] DELETE /vdb1/238/AUTH_215c458021024a8c87471157a7040644/zoom43/objkeyf03550a3-2be9-4b59-b12b-df75378aea14 409 - DELETE http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom43/objkeyf03550a3-2be9-4b59-b12b-df75378aea14; tx0bc79d62fec445d28dfab-005356288f proxy-server 2142 0.0005 On Wed, Apr 23, 2014 at 8:20 AM, Sumit Gaur sumitkg...@gmail.com wrote: Hi clay, Thanks for responding , first about setup , it is multi node env, and problem is with delete object . I checked proxy and storage nodes clock, they are in sync. Also storage node run only main daemons I.e. no auditor, and others. I traced the 409 call and it is coming from object server. I am generating random id for key so no reuse of same name. But after one put(t1) next is del(t1).is this could be a problem...I think they are sync calls. Regards Sumit On Apr 23, 2014 5:14 AM, Clay Gerrard clay.gerr...@gmail.com wrote: 409 on DELETE (object?) is a pretty specific error. That should mean that the timestamp assigned to the delete is earlier than the timestamp of the data file. Most likely mean that you're getting some time-drift on your proxies (but that assumes multi-node) or maybe that you're reusing names between threads and your object server's see PUT(ts1) PUT(ts3) DELETE(ts2) - but that'd be a pretty tight race... Should all be logged - try and find a DELETE that went 409 and trace the transaction id. On Mon, Apr 21, 2014 at 5:54 AM, taurus huang huanggeng.8...@gmail.com wrote: Please provide the log file： /var/log/swift/swift.log AND /var/log/keystone/keystone.log On Mon, Apr 21, 2014 at 11:55 AM, Sumit Gaur sumitkg...@gmail.com wrote: Hi I using jclouds lib integrated with Openstack Swift+ keystone combination. Things are working fine except stability test. After 20-30 hours of test jclouds/SWIFT start degrading in TPS and keep going down over the time. 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads. 2) I am getting a lot of 409 and DEL failure for the response too from SWIFT. Can sombody help me what is going wrong here ? Thanks sumit ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [SWIFT] Delete operation problem
*FYI more concrete logs * *Client Node* Request failed for objkey objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615 org.jclouds.http.HttpResponseException: command: DELETE http://www.example.com:80/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615HTTP/1.1 failed with response: HTTP/1.1 409 Conflict; content: [htmlh1Conflict/h1pThere was a conflict when trying to complete your request./p/html] *Proxy Node * Apr 23 12:00:27 proxy1 swift-proxy allow user with role admin as account admin (txn: txe45b336b0ab44aee8a698-0053572ccb) (client_ip: 10.41.54.27) *Storage Node* Apr 23 12:00:26 object1* container-server* 10.41.54.9 - - [23/Apr/2014:03:00:26 +] PUT /vdb1/955/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615 201 - tx3ba78bc8924349d18f7af-0053572ccc PUT http://www.example.com/vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615; obj-server 17908 0.0006 Apr 23 12:00:26 object1 *object-server* 10.41.54.21 - - [23/Apr/2014:03:00:26 +] PUT /vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615 201 - *PUT * http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615; tx3ba78bc8924349d18f7af-0053572ccc proxy-server 23927 0.0113 Apr 23 12:00:26 object1 *object-server *10.41.54.19 - - [23/Apr/2014:03:00:26 +] DELETE /vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615 409 - *DELETE * http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615; txe45b336b0ab44aee8a698-0053572ccb proxy-server 10071 0.0003 On Wed, Apr 23, 2014 at 5:09 AM, Clay Gerrard clay.gerr...@gmail.comwrote: 409 on DELETE (object?) is a pretty specific error. That should mean that the timestamp assigned to the delete is earlier than the timestamp of the data file. Most likely mean that you're getting some time-drift on your proxies (but that assumes multi-node) or maybe that you're reusing names between threads and your object server's see PUT(ts1) PUT(ts3) DELETE(ts2) - but that'd be a pretty tight race... Should all be logged - try and find a DELETE that went 409 and trace the transaction id. On Mon, Apr 21, 2014 at 5:54 AM, taurus huang huanggeng.8...@gmail.comwrote: Please provide the log file： /var/log/swift/swift.log AND /var/log/keystone/keystone.log On Mon, Apr 21, 2014 at 11:55 AM, Sumit Gaur sumitkg...@gmail.comwrote: Hi I using jclouds lib integrated with Openstack Swift+ keystone combination. Things are working fine except stability test. After 20-30 hours of test jclouds/SWIFT start degrading in TPS and keep going down over the time. 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads. 2) I am getting a lot of 409 and DEL failure for the response too from SWIFT. Can sombody help me what is going wrong here ? Thanks sumit ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [SWIFT] Delete operation problem
Hi I using jclouds lib integrated with Openstack Swift+ keystone combination. Things are working fine except stability test. After 20-30 hours of test jclouds/SWIFT start degrading in TPS and keep going down over the time. 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads. 2) I am getting a lot of 409 and DEL failure for the response too from SWIFT. Can sombody help me what is going wrong here ? Thanks sumit ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [swift] Enterprise Deployment
Hi I understand that Swift need a lot of configuration and manual steps. Need to know if somebody knows about any existing opensource tool/process that can help in deploying multinode enterprise deployment of Openstack SWIFT. I tried chef and some existing cookbooks but using them need a lot of learning curve for ruby and chef , also I understand that dev stack also does not provide enterprise level scripts. Thanks sumit ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Swift] Request's Round trip time
Hi I want to see the Round Trip Time for swift request in proxy log. I am able to see RTT for failures and timeouts and no RTT for normal requests. Do I need to set and config param. Thanks sumit ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
