Re: [Openstack] Issue in debugging OpenStack Swift
HI Muhammad If you’ve got full debug working well, I would love to replicate your environment (and suspect I’m not the only one). If you have some time, it would be great if you could detail setup instructions and config options on this post. Thx Paul From: Openstack [mailto:openstack-bounces+paul.e.luse=intel@lists.launchpad.net] On Behalf Of Muhammad Kazim Sent: Tuesday, July 23, 2013 12:33 AM To: Gareth Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Issue in debugging OpenStack Swift Worked perfectly in the first attempt. Thanks a lot for the help. :) On Tue, Jul 23, 2013 at 10:41 AM, Gareth academicgar...@gmail.commailto:academicgar...@gmail.com wrote: the key reason it here: https://github.com/openstack/swift/blob/master/swift/common/utils.py#L1047 you have many ways to stop flush IO; I prefer add a line stdio_files = [] after line 1037 On Sun, Jul 21, 2013 at 1:58 AM, Muhammad Kazim kazima...@gmail.commailto:kazima...@gmail.com wrote: Hi all, I wanted to debug Swift source code. I have a Devstack setup running on my machine that i have configured to only run keystone, mysql and swift. I have used pdb to debug source code. I was able to debug python-swiftclient, middleware (common), and proxy-server code. However, when i try to debug Object Server of Swift, by issuing pdb.set_trace() call in diskwrite or any write function in /swift/obj/server.py (except constructor) and than issue a 'swift post' or 'swift upload' command from other terminal, the object server crashes. The screen of s-object shows an error in the line in which 'pdb.set_trace()' is called. Can someone please guide me how can i debug the Swift s-object code in /swift/obj/server.py. Thanks. Regards, Muhammad Kazim ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Gareth Cloud Computing, OpenStack, Fitness, Basketball OpenStack contributor Company: UnitedStackhttp://www.ustack.com My promise: if you find any spelling or grammar mistakes in my email from Mar 1 2013, notify me and I'll donate $1 or ￥1 to an open organization you specify. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Storage Server Redirection
So I'm not attached to 5xx as an error code, I was actually debating between the two before sending this message out so have no argument for 5xx - I'll gladly switch the plan to 3xx :) Thanks Paul -Original Message- From: Peter Portante [mailto:peter.a.porta...@gmail.com] Sent: Monday, June 03, 2013 3:41 AM To: Hua ZZ Zhang Cc: Luse, Paul E; Openstack; openstack@lists.launchpad.net Subject: Re: [Openstack] [Swift] Storage Server Redirection Hi Paul, I would like to echo Hua's and John's concern about 5xx code uses. Using 3xx codes should be fine. Can you give a sufficient argument to consider 5xx? Thanks, -peter On Mon, Jun 3, 2013 at 5:49 AM, Hua ZZ Zhang zhu...@cn.ibm.com wrote: 2) The basic idea is that an object server (via middleware or otherwise) will be given the ability to respond to a request to indicate ‘not me but I know who should handle this’. I’m thinking this makes more sense as a 5xx response with additional information (partition, nodes) about the route included in the response body (as opposed to a 3xx code) My concern is that 5xx response is a kind of error that occurs on server side. The server can't handle the client request due to internal error (500), not implemented (501), bad gateway(502), service unavailable(503), gateway timeout(504) or http version not supported(505). It doesn't make much sense to use it in your context of 'not me but I know who should handle this’. I'm also curious about how could this happen if the object URL is the unique identity. How does the server exactly know what client request is another object? Based on what kind of information? Edward Zhang(张华) Advisory Software Engineer Software Standards Open Source Software Emerging Technology Institute(ETI) IBM China Software Development Lab e-mail: zhu...@cn.ibm.com Luse, Paul E ---2013-06-01 上午 07:53:21---Luse, Paul E paul.e.l...@intel.com Luse, Paul E paul.e.l...@intel.com Sent by: Openstack openstack-bounces+zhuadl=cn.ibm@lists.launchpad.net 2013-06-01 上午 07:53 To openstack@lists.launchpad.net openstack@lists.launchpad.net, cc Subject [Openstack] [Swift] Storage Server Redirection I’m looking at tacking this item: https://blueprints.launchpad.net/swift/+spec/support-storage-server-re directs and wanted to get some feedback on the following observations/thoughts: 1) This is a capability that would be checked in independent of other blueprints that might use it (2 are mentioned in the link above) and unit test code would be the only way to initially exercise it; it essentially enables other activities at this point 2) The basic idea is that an object server (via middleware or otherwise) will be given the ability to respond to a request to indicate ‘not me but I know who should handle this’. I’m thinking this makes more sense as a 5xx response with additional information (partition, nodes) about the route included in the response body (as opposed to a 3xx code) 3) The proxy server will be modified to process the response accordingly but using the partition, nodes info from the response as opposed to object_ring.get_nodes() to determine which nodes to use 4) Protection will be required to avoid endless redirection loops 5) This applies only to GET operations Appreciate any thoughts/feedback., In addition to the two usages of this capability referenced in the blueprint I think there’s applicable to another Tiering blueprint which interests me as well. Thanks Paul ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Storage Server Redirection
Thanks for the response John, see below... Thx Paul -Original Message- From: John Dickinson [mailto:m...@not.mn] Sent: Sunday, June 02, 2013 11:29 PM To: Luse, Paul E Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [Swift] Storage Server Redirection On May 31, 2013, at 4:53 PM, Luse, Paul E paul.e.l...@intel.com wrote: I'm looking at tacking this item: https://blueprints.launchpad.net/swift/+spec/support-storage-server-redirects and wanted to get some feedback on the following observations/thoughts: 1) This is a capability that would be checked in independent of other blueprints that might use it (2 are mentioned in the link above) and unit test code would be the only way to initially exercise it; it essentially enables other activities at this point correct, IMO, but see below. 2) The basic idea is that an object server (via middleware or otherwise) will be given the ability to respond to a request to indicate 'not me but I know who should handle this'. I'm thinking this makes more sense as a 5xx response with additional information (partition, nodes) about the route included in the response body (as opposed to a 3xx code) There are already some specific checks around a 5xx response from object servers that relate to failure handling. I'd assume a 3xx response would be used for redirects, with any additional info given in headers. Can you share more about why a 5xx response is more appropriate? PL replied on another thread, I'm cool with 3xx 3) The proxy server will be modified to process the response accordingly but using the partition, nodes info from the response as opposed to object_ring.get_nodes() to determine which nodes to use yes 4) Protection will be required to avoid endless redirection loops Yes, but since this is handled by a single proxy worker on a per-request basis, protection should be fairly simple. PL agreed, just calling it out for completeness 5) This applies only to GET operations Supporting this on writes can allow for less of a replication storm later if the object server knows the correct location before proxy is updated (eg during a cluster upgrade or a ring deploy). PL I was just looking to simplify this but see your point. Handling writes as well would be slightly more complex, unless I'm missing something, as the proxy would have to deal with multiple redirect responses that potentially don't match. Is acceptable to approach it this was (GET only) and submit a 2nd blueprint to add PUT support or is that breaking it down too much? Appreciate any thoughts/feedback., In addition to the two usages of this capability referenced in the blueprint I think there's applicable to another Tiering blueprint which interests me as well. The fun part will be figuring out what to do when the cluster is not in an internally consistent state (eg cluster upgrades or ring deployment times). You want to redirect if the object server knows best, but you don't if the proxy server knows best. It may make sense (from a usefulness perspective) to implement https://blueprints.launchpad.net/swift/+spec/send-ring-version first. PL Or I could just merge them so neither contribution would be initially 'unused' when checked in - or another alternative would be that the proxy could include a new header to indicate that it is overriding any object redirection and for that request the obj server is to simply do as its told. Or I could do both... Thanks Paul ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Storage Server Redirection
Hi Peter, I'd ask that Edward may want to chime in on object variation as I may not understand that one correctly - again I'm proposing to just implement the redirect response blueprint and reference object variation as a potential user simply because John mentioned it in his original write-up of the redirect response blueprint. Edward? Wrt your second question, there are copies of the rings on object servers as well and they are used for lookup in various scenarios, replication for one. As an example take a small 6 node cluster where all servers are running all services, if an admin adds/deletes a device from the ring they do so on one of the nodes and then copes the .gz file to all the others. During the window when the first node detects the new ring file and .gz files are copied to all the other nodes and detected/brought into memory, any responses fielded by the proxy service on the other nodes will have older ring info than the first node updated. Another case could be a larger cluster where there's a proxy tier and a capacity tier (not all services running everywhere). The admin could update the ring on any node and copy the ring files on the nodes in any order - if any capacity node gets an update before any proxy node you can run into the same scenario. The ring versions blueprint was John's so he may have additions/corrections Thanks Paul -Original Message- From: Peter Portante [mailto:peter.a.porta...@gmail.com] Sent: Monday, June 03, 2013 3:50 AM To: Luse, Paul E Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [Swift] Storage Server Redirection Hi Paul, Can you explain more about the two use cases referenced? Object Variation seems like a nice idea, but can you clarify how the mechanism would work to get at the variations? Any examples? From the current descriptions, it seems like the LFS Patch could help here without adding anything to the object server. Regarding the ring updates, given that rings are used by the proxy servers, and as far as I can tell, the object servers don't know about the rings, how would object servers have more up-to-date knowledge to perform the redirect? Thanks, -peter On Fri, May 31, 2013 at 7:53 PM, Luse, Paul E paul.e.l...@intel.com wrote: I'm looking at tacking this item: https://blueprints.launchpad.net/swift/+spec/support-storage-server-re directs and wanted to get some feedback on the following observations/thoughts: 1) This is a capability that would be checked in independent of other blueprints that might use it (2 are mentioned in the link above) and unit test code would be the only way to initially exercise it; it essentially enables other activities at this point 2) The basic idea is that an object server (via middleware or otherwise) will be given the ability to respond to a request to indicate 'not me but I know who should handle this'. I'm thinking this makes more sense as a 5xx response with additional information (partition, nodes) about the route included in the response body (as opposed to a 3xx code) 3) The proxy server will be modified to process the response accordingly but using the partition, nodes info from the response as opposed to object_ring.get_nodes() to determine which nodes to use 4) Protection will be required to avoid endless redirection loops 5) This applies only to GET operations Appreciate any thoughts/feedback., In addition to the two usages of this capability referenced in the blueprint I think there's applicable to another Tiering blueprint which interests me as well. Thanks Paul ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] Storage Server Redirection
Hi Peter- On the first point, note that all 3 of the features in question were submitted by John as separate blueprints - I don't need to understand *all* of the possible usages of the re-direct capability to implement it, it's a generic 'utility' capability that either of these other two blueprints can make use of or even alter when the times comes, if needed, for their needs provided that alteration doesn't break anything of course. There's more than just the other two usages as well, I see use for this in a tiering capacity that hasn't been fully explored yet (separate topic yet again). On the second point, thanks for the tip as I have not yet read the LFS details but I will do so for sure. Thx Paul -Original Message- From: Peter Portante [mailto:peter.a.porta...@gmail.com] Sent: Monday, June 03, 2013 7:08 AM To: Luse, Paul E Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [Swift] Storage Server Redirection On Mon, Jun 3, 2013 at 8:41 AM, Luse, Paul E paul.e.l...@intel.com wrote: Hi Peter, I'd ask that Edward may want to chime in on object variation as I may not understand that one correctly - again I'm proposing to just implement the redirect response blueprint and reference object variation as a potential user simply because John mentioned it in his original write-up of the redirect response blueprint. Edward? But if there is not concrete need for the redirection, why implement it? Wrt your second question, there are copies of the rings on object servers as well and they are used for lookup in various scenarios, replication for one. As an example take a small 6 node cluster where all servers are running all services, if an admin adds/deletes a device from the ring they do so on one of the nodes and then copes the .gz file to all the others. During the window when the first node detects the new ring file and .gz files are copied to all the other nodes and detected/brought into memory, any responses fielded by the proxy service on the other nodes will have older ring info than the first node updated. Another case could be a larger cluster where there's a proxy tier and a capacity tier (not all services running everywhere). The admin could update the ring on any node and copy the ring files on the nodes in any order - if any capacity node gets an update before any proxy node you can run into the same scenario. The ring versions blueprint was John's so he ma y have additions/corrections So the copies of the RIng is not used by the server, but by the account reaper, container updater, container sync, object updater, and object replicator. Why would we want to get the object server involved here as well? It feels like that information could be localized in the proxy servers, but perhaps another discussion. Have you looked at the Pete Zaitcev's work on the LFS patch? It seems like keeping this level of information in the proxy server, specifically in a form of the object server controllers would be a good place for this kind of thing. Thanks Paul -Original Message- From: Peter Portante [mailto:peter.a.porta...@gmail.com] Sent: Monday, June 03, 2013 3:50 AM To: Luse, Paul E Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [Swift] Storage Server Redirection Hi Paul, Can you explain more about the two use cases referenced? Object Variation seems like a nice idea, but can you clarify how the mechanism would work to get at the variations? Any examples? From the current descriptions, it seems like the LFS Patch could help here without adding anything to the object server. Regarding the ring updates, given that rings are used by the proxy servers, and as far as I can tell, the object servers don't know about the rings, how would object servers have more up-to-date knowledge to perform the redirect? Thanks, -peter On Fri, May 31, 2013 at 7:53 PM, Luse, Paul E paul.e.l...@intel.com wrote: I'm looking at tacking this item: https://blueprints.launchpad.net/swift/+spec/support-storage-server-r e directs and wanted to get some feedback on the following observations/thoughts: 1) This is a capability that would be checked in independent of other blueprints that might use it (2 are mentioned in the link above) and unit test code would be the only way to initially exercise it; it essentially enables other activities at this point 2) The basic idea is that an object server (via middleware or otherwise) will be given the ability to respond to a request to indicate 'not me but I know who should handle this'. I'm thinking this makes more sense as a 5xx response with additional information (partition, nodes) about the route included in the response body (as opposed to a 3xx code) 3) The proxy server will be modified to process the response accordingly but using the partition, nodes info from the response as opposed to object_ring.get_nodes
[Openstack] [Swift] Storage Server Redirection
I'm looking at tacking this item: https://blueprints.launchpad.net/swift/+spec/support-storage-server-redirects and wanted to get some feedback on the following observations/thoughts: 1) This is a capability that would be checked in independent of other blueprints that might use it (2 are mentioned in the link above) and unit test code would be the only way to initially exercise it; it essentially enables other activities at this point 2) The basic idea is that an object server (via middleware or otherwise) will be given the ability to respond to a request to indicate 'not me but I know who should handle this'. I'm thinking this makes more sense as a 5xx response with additional information (partition, nodes) about the route included in the response body (as opposed to a 3xx code) 3) The proxy server will be modified to process the response accordingly but using the partition, nodes info from the response as opposed to object_ring.get_nodes() to determine which nodes to use 4) Protection will be required to avoid endless redirection loops 5) This applies only to GET operations Appreciate any thoughts/feedback., In addition to the two usages of this capability referenced in the blueprint I think there's applicable to another Tiering blueprint which interests me as well. Thanks Paul ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
