Re: [openstack-dev] [neutron] Supporting retries in neutronclient
Carl, I haven't been able to try this yet as it requires us to run a pretty big scale test. But to try to summarize the current feeling on this thread... the retry logic is being put into the neutronclient already (via https://review.openstack.org/#/c/71464/), it's just that it's not "automatic" and is being left up to the invoker to decide when to use retry. The idea of doing the retries automatically isn't the way to go because it is dangerous for non-idempotent operations. So... I think we leave the proposed change as is and will potentially need to enhance users as we see fit. The invoker in our failure case is nova trying to get network info, so this seems like a good first one to try out. Thoughts? Thanks, Paul Quoting Carl Baldwin : Paul, I'm curious. Have you been able to update to a client using requests? Has it solved your problem? Carl On Thu, May 29, 2014 at 11:15 AM, Paul Ward wrote: Yes, we're still on a code level that uses httplib2. I noticed that as well, but wasn't sure if that would really help here as it seems like an ssl thing itself. But... who knows?? I'm not sure how consistently we can recreate this, but if we can, I'll try using that patch to use requests and see if that helps. "Armando M." wrote on 05/29/2014 11:52:34 AM: From: "Armando M." To: "OpenStack Development Mailing List (not for usage questions)" , Date: 05/29/2014 11:58 AM Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient Hi Paul, Just out of curiosity, I am assuming you are using the client that still relies on httplib2. Patch [1] replaced httplib2 with requests, but I believe that a new client that incorporates this change has not yet been published. I wonder if the failures you are referring to manifest themselves with the former http library rather than the latter. Could you clarify? Thanks, Armando [1] - https://review.openstack.org/#/c/89879/ On 29 May 2014 17:25, Paul Ward wrote: > Well, for my specific error, it was an intermittent ssl handshake error > before the request was ever sent to the > neutron-server. In our case, we saw that 4 out of 5 resize operations > worked, the fifth failed with this ssl > handshake error in neutronclient. > > I certainly think a GET is safe to retry, and I agree with your > statement > that PUTs and DELETEs probably > are as well. This still leaves a change in nova needing to be made to > actually a) specify a conf option and > b) pass it to neutronclient where appropriate. > > > Aaron Rosen wrote on 05/28/2014 07:38:56 PM: > >> From: Aaron Rosen > > >> To: "OpenStack Development Mailing List (not for usage questions)" >> , >> Date: 05/28/2014 07:44 PM > >> Subject: Re: [openstack-dev] [neutron] Supporting retries in >> neutronclient >> >> Hi, >> >> I'm curious if other openstack clients implement this type of retry >> thing. I think retrying on GET/DELETES/PUT's should probably be okay. >> >> What types of errors do you see in the neutron-server when it fails >> to respond? I think it would be better to move the retry logic into >> the server around the failures rather than the client (or better yet >> if we fixed the server :)). Most of the times I've seen this type of >> failure is due to deadlock errors caused between (sqlalchemy and >> eventlet *i think*) which cause the client to eventually timeout. >> >> Best, >> >> Aaron >> > >> On Wed, May 28, 2014 at 11:51 AM, Paul Ward wrote: >> Would it be feasible to make the retry logic only apply to read-only >> operations? This would still require a nova change to specify the >> number of retries, but it'd also prevent invokers from shooting >> themselves in the foot if they call for a write operation. >> >> >> >> Aaron Rosen wrote on 05/27/2014 09:40:00 PM: >> >> > From: Aaron Rosen >> >> > To: "OpenStack Development Mailing List (not for usage questions)" >> > , >> > Date: 05/27/2014 09:44 PM >> >> > Subject: Re: [openstack-dev] [neutron] Supporting retries in >> > neutronclient >> > >> > Hi, >> >> > >> > Is it possible to detect when the ssl handshaking error occurs on >> > the client side (and only retry for that)? If so I think we should >> > do that rather than retrying multiple times. The danger here is >> > mostly for POST operations (as Eugene pointed out) where it's >> > possible for the response to not make it back to the client and for >> > the operation to actually succeed. >> > >>
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
Paul, I'm curious. Have you been able to update to a client using requests? Has it solved your problem? Carl On Thu, May 29, 2014 at 11:15 AM, Paul Ward wrote: > Yes, we're still on a code level that uses httplib2. I noticed that as > well, but wasn't sure if that would really > help here as it seems like an ssl thing itself. But... who knows?? I'm not > sure how consistently we can > recreate this, but if we can, I'll try using that patch to use requests and > see if that helps. > > > > "Armando M." wrote on 05/29/2014 11:52:34 AM: > >> From: "Armando M." > > >> To: "OpenStack Development Mailing List (not for usage questions)" >> , >> Date: 05/29/2014 11:58 AM > >> Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient >> >> Hi Paul, >> >> Just out of curiosity, I am assuming you are using the client that >> still relies on httplib2. Patch [1] replaced httplib2 with requests, >> but I believe that a new client that incorporates this change has not >> yet been published. I wonder if the failures you are referring to >> manifest themselves with the former http library rather than the >> latter. Could you clarify? >> >> Thanks, >> Armando >> >> [1] - https://review.openstack.org/#/c/89879/ >> >> On 29 May 2014 17:25, Paul Ward wrote: >> > Well, for my specific error, it was an intermittent ssl handshake error >> > before the request was ever sent to the >> > neutron-server. In our case, we saw that 4 out of 5 resize operations >> > worked, the fifth failed with this ssl >> > handshake error in neutronclient. >> > >> > I certainly think a GET is safe to retry, and I agree with your >> > statement >> > that PUTs and DELETEs probably >> > are as well. This still leaves a change in nova needing to be made to >> > actually a) specify a conf option and >> > b) pass it to neutronclient where appropriate. >> > >> > >> > Aaron Rosen wrote on 05/28/2014 07:38:56 PM: >> > >> >> From: Aaron Rosen >> > >> > >> >> To: "OpenStack Development Mailing List (not for usage questions)" >> >> , >> >> Date: 05/28/2014 07:44 PM >> > >> >> Subject: Re: [openstack-dev] [neutron] Supporting retries in >> >> neutronclient >> >> >> >> Hi, >> >> >> >> I'm curious if other openstack clients implement this type of retry >> >> thing. I think retrying on GET/DELETES/PUT's should probably be okay. >> >> >> >> What types of errors do you see in the neutron-server when it fails >> >> to respond? I think it would be better to move the retry logic into >> >> the server around the failures rather than the client (or better yet >> >> if we fixed the server :)). Most of the times I've seen this type of >> >> failure is due to deadlock errors caused between (sqlalchemy and >> >> eventlet *i think*) which cause the client to eventually timeout. >> >> >> >> Best, >> >> >> >> Aaron >> >> >> > >> >> On Wed, May 28, 2014 at 11:51 AM, Paul Ward wrote: >> >> Would it be feasible to make the retry logic only apply to read-only >> >> operations? This would still require a nova change to specify the >> >> number of retries, but it'd also prevent invokers from shooting >> >> themselves in the foot if they call for a write operation. >> >> >> >> >> >> >> >> Aaron Rosen wrote on 05/27/2014 09:40:00 PM: >> >> >> >> > From: Aaron Rosen >> >> >> >> > To: "OpenStack Development Mailing List (not for usage questions)" >> >> > , >> >> > Date: 05/27/2014 09:44 PM >> >> >> >> > Subject: Re: [openstack-dev] [neutron] Supporting retries in >> >> > neutronclient >> >> > >> >> > Hi, >> >> >> >> > >> >> > Is it possible to detect when the ssl handshaking error occurs on >> >> > the client side (and only retry for that)? If so I think we should >> >> > do that rather than retrying multiple times. The danger here is >> >> > mostly for POST operations (as Eugene pointed out) where it's >> >> > possible for the response to not make it back to the client and for
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
+1. After reading through this thread, I think that a blind --retries N could be harmful and unwise given the current API definition. Users that need a retry for an SSL error are going to get in to the habit of adding --retries N to all their calls and they'll end up in trouble because they really should be taking action on the particular error that occurs, not just retrying on any error. Carl On Tue, May 27, 2014 at 8:40 PM, Aaron Rosen wrote: > Hi, > > Is it possible to detect when the ssl handshaking error occurs on the client > side (and only retry for that)? If so I think we should do that rather than > retrying multiple times. The danger here is mostly for POST operations (as > Eugene pointed out) where it's possible for the response to not make it back > to the client and for the operation to actually succeed. > > Having this retry logic nested in the client also prevents things like nova > from handling these types of failures individually since this retry logic is > happening inside of the client. I think it would be better not to have this > internal mechanism in the client and instead make the user of the client > implement retry so they are aware of failures. > > Aaron > > > On Tue, May 27, 2014 at 10:48 AM, Paul Ward wrote: >> >> Currently, neutronclient is hardcoded to only try a request once in >> retry_request by virtue of the fact that it uses self.retries as the retry >> count, and that's initialized to 0 and never changed. We've seen an issue >> where we get an ssl handshaking error intermittently (seems like more of an >> ssl bug) and a retry would probably have worked. Yet, since neutronclient >> only tries once and gives up, it fails the entire operation. Here is the >> code in question: >> >> >> https://github.com/openstack/python-neutronclient/blob/master/neutronclient/v2_0/client.py#L1296 >> >> Does anybody know if there's some explicit reason we don't currently allow >> configuring the number of retries? If not, I'm inclined to propose a change >> for just that. >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
Httplib2 does very little directly with ssl other than a wrap_socket call. Unless requests has special ssl error handling and retry logic, it will be exposed to the same set of underlying errors from the ssl library so a retry that at least catches socket and ssl errors is a good idea. On May 29, 2014 1:17 PM, "Paul Ward" wrote: > Yes, we're still on a code level that uses httplib2. I noticed that as > well, but wasn't sure if that would really > help here as it seems like an ssl thing itself. But... who knows?? I'm > not sure how consistently we can > recreate this, but if we can, I'll try using that patch to use requests > and see if that helps. > > > > "Armando M." wrote on 05/29/2014 11:52:34 AM: > > > From: "Armando M." > > To: "OpenStack Development Mailing List (not for usage questions)" > > , > > Date: 05/29/2014 11:58 AM > > Subject: Re: [openstack-dev] [neutron] Supporting retries in > neutronclient > > > > Hi Paul, > > > > Just out of curiosity, I am assuming you are using the client that > > still relies on httplib2. Patch [1] replaced httplib2 with requests, > > but I believe that a new client that incorporates this change has not > > yet been published. I wonder if the failures you are referring to > > manifest themselves with the former http library rather than the > > latter. Could you clarify? > > > > Thanks, > > Armando > > > > [1] - https://review.openstack.org/#/c/89879/ > > > > On 29 May 2014 17:25, Paul Ward wrote: > > > Well, for my specific error, it was an intermittent ssl handshake error > > > before the request was ever sent to the > > > neutron-server. In our case, we saw that 4 out of 5 resize operations > > > worked, the fifth failed with this ssl > > > handshake error in neutronclient. > > > > > > I certainly think a GET is safe to retry, and I agree with your > statement > > > that PUTs and DELETEs probably > > > are as well. This still leaves a change in nova needing to be made to > > > actually a) specify a conf option and > > > b) pass it to neutronclient where appropriate. > > > > > > > > > Aaron Rosen wrote on 05/28/2014 07:38:56 PM: > > > > > >> From: Aaron Rosen > > > > > > > > >> To: "OpenStack Development Mailing List (not for usage questions)" > > >> , > > >> Date: 05/28/2014 07:44 PM > > > > > >> Subject: Re: [openstack-dev] [neutron] Supporting retries in > neutronclient > > >> > > >> Hi, > > >> > > >> I'm curious if other openstack clients implement this type of retry > > >> thing. I think retrying on GET/DELETES/PUT's should probably be okay. > > >> > > >> What types of errors do you see in the neutron-server when it fails > > >> to respond? I think it would be better to move the retry logic into > > >> the server around the failures rather than the client (or better yet > > >> if we fixed the server :)). Most of the times I've seen this type of > > >> failure is due to deadlock errors caused between (sqlalchemy and > > >> eventlet *i think*) which cause the client to eventually timeout. > > >> > > >> Best, > > >> > > >> Aaron > > >> > > > > > >> On Wed, May 28, 2014 at 11:51 AM, Paul Ward > wrote: > > >> Would it be feasible to make the retry logic only apply to read-only > > >> operations? This would still require a nova change to specify the > > >> number of retries, but it'd also prevent invokers from shooting > > >> themselves in the foot if they call for a write operation. > > >> > > >> > > >> > > >> Aaron Rosen wrote on 05/27/2014 09:40:00 PM: > > >> > > >> > From: Aaron Rosen > > >> > > >> > To: "OpenStack Development Mailing List (not for usage questions)" > > >> > , > > >> > Date: 05/27/2014 09:44 PM > > >> > > >> > Subject: Re: [openstack-dev] [neutron] Supporting retries in > > >> > neutronclient > > >> > > > >> > Hi, > > >> > > >> > > > >> > Is it possible to detect when the ssl handshaking error occurs on > > >> > the client side (and only retry for that)? If so I think we should > > >> > do that rather than
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
mishandling of SSL was the very reason why I brought that change forward; so I wouldn't rule it out completely ;) A. On 29 May 2014 19:15, Paul Ward wrote: > Yes, we're still on a code level that uses httplib2. I noticed that as > well, but wasn't sure if that would really > help here as it seems like an ssl thing itself. But... who knows?? I'm not > sure how consistently we can > recreate this, but if we can, I'll try using that patch to use requests and > see if that helps. > > > > "Armando M." wrote on 05/29/2014 11:52:34 AM: > >> From: "Armando M." > > >> To: "OpenStack Development Mailing List (not for usage questions)" >> , >> Date: 05/29/2014 11:58 AM > >> Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient >> >> Hi Paul, >> >> Just out of curiosity, I am assuming you are using the client that >> still relies on httplib2. Patch [1] replaced httplib2 with requests, >> but I believe that a new client that incorporates this change has not >> yet been published. I wonder if the failures you are referring to >> manifest themselves with the former http library rather than the >> latter. Could you clarify? >> >> Thanks, >> Armando >> >> [1] - https://review.openstack.org/#/c/89879/ >> >> On 29 May 2014 17:25, Paul Ward wrote: >> > Well, for my specific error, it was an intermittent ssl handshake error >> > before the request was ever sent to the >> > neutron-server. In our case, we saw that 4 out of 5 resize operations >> > worked, the fifth failed with this ssl >> > handshake error in neutronclient. >> > >> > I certainly think a GET is safe to retry, and I agree with your >> > statement >> > that PUTs and DELETEs probably >> > are as well. This still leaves a change in nova needing to be made to >> > actually a) specify a conf option and >> > b) pass it to neutronclient where appropriate. >> > >> > >> > Aaron Rosen wrote on 05/28/2014 07:38:56 PM: >> > >> >> From: Aaron Rosen >> > >> > >> >> To: "OpenStack Development Mailing List (not for usage questions)" >> >> , >> >> Date: 05/28/2014 07:44 PM >> > >> >> Subject: Re: [openstack-dev] [neutron] Supporting retries in >> >> neutronclient >> >> >> >> Hi, >> >> >> >> I'm curious if other openstack clients implement this type of retry >> >> thing. I think retrying on GET/DELETES/PUT's should probably be okay. >> >> >> >> What types of errors do you see in the neutron-server when it fails >> >> to respond? I think it would be better to move the retry logic into >> >> the server around the failures rather than the client (or better yet >> >> if we fixed the server :)). Most of the times I've seen this type of >> >> failure is due to deadlock errors caused between (sqlalchemy and >> >> eventlet *i think*) which cause the client to eventually timeout. >> >> >> >> Best, >> >> >> >> Aaron >> >> >> > >> >> On Wed, May 28, 2014 at 11:51 AM, Paul Ward wrote: >> >> Would it be feasible to make the retry logic only apply to read-only >> >> operations? This would still require a nova change to specify the >> >> number of retries, but it'd also prevent invokers from shooting >> >> themselves in the foot if they call for a write operation. >> >> >> >> >> >> >> >> Aaron Rosen wrote on 05/27/2014 09:40:00 PM: >> >> >> >> > From: Aaron Rosen >> >> >> >> > To: "OpenStack Development Mailing List (not for usage questions)" >> >> > , >> >> > Date: 05/27/2014 09:44 PM >> >> >> >> > Subject: Re: [openstack-dev] [neutron] Supporting retries in >> >> > neutronclient >> >> > >> >> > Hi, >> >> >> >> > >> >> > Is it possible to detect when the ssl handshaking error occurs on >> >> > the client side (and only retry for that)? If so I think we should >> >> > do that rather than retrying multiple times. The danger here is >> >> > mostly for POST operations (as Eugene pointed out) where it's >> >> > possible for the response to not make it back to the client and for >> >
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
Yes, we're still on a code level that uses httplib2. I noticed that as well, but wasn't sure if that would really help here as it seems like an ssl thing itself. But... who knows?? I'm not sure how consistently we can recreate this, but if we can, I'll try using that patch to use requests and see if that helps. "Armando M." wrote on 05/29/2014 11:52:34 AM: > From: "Armando M." > To: "OpenStack Development Mailing List (not for usage questions)" > , > Date: 05/29/2014 11:58 AM > Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient > > Hi Paul, > > Just out of curiosity, I am assuming you are using the client that > still relies on httplib2. Patch [1] replaced httplib2 with requests, > but I believe that a new client that incorporates this change has not > yet been published. I wonder if the failures you are referring to > manifest themselves with the former http library rather than the > latter. Could you clarify? > > Thanks, > Armando > > [1] - https://review.openstack.org/#/c/89879/ > > On 29 May 2014 17:25, Paul Ward wrote: > > Well, for my specific error, it was an intermittent ssl handshake error > > before the request was ever sent to the > > neutron-server. In our case, we saw that 4 out of 5 resize operations > > worked, the fifth failed with this ssl > > handshake error in neutronclient. > > > > I certainly think a GET is safe to retry, and I agree with your statement > > that PUTs and DELETEs probably > > are as well. This still leaves a change in nova needing to be made to > > actually a) specify a conf option and > > b) pass it to neutronclient where appropriate. > > > > > > Aaron Rosen wrote on 05/28/2014 07:38:56 PM: > > > >> From: Aaron Rosen > > > > > >> To: "OpenStack Development Mailing List (not for usage questions)" > >> , > >> Date: 05/28/2014 07:44 PM > > > >> Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient > >> > >> Hi, > >> > >> I'm curious if other openstack clients implement this type of retry > >> thing. I think retrying on GET/DELETES/PUT's should probably be okay. > >> > >> What types of errors do you see in the neutron-server when it fails > >> to respond? I think it would be better to move the retry logic into > >> the server around the failures rather than the client (or better yet > >> if we fixed the server :)). Most of the times I've seen this type of > >> failure is due to deadlock errors caused between (sqlalchemy and > >> eventlet *i think*) which cause the client to eventually timeout. > >> > >> Best, > >> > >> Aaron > >> > > > >> On Wed, May 28, 2014 at 11:51 AM, Paul Ward wrote: > >> Would it be feasible to make the retry logic only apply to read-only > >> operations? This would still require a nova change to specify the > >> number of retries, but it'd also prevent invokers from shooting > >> themselves in the foot if they call for a write operation. > >> > >> > >> > >> Aaron Rosen wrote on 05/27/2014 09:40:00 PM: > >> > >> > From: Aaron Rosen > >> > >> > To: "OpenStack Development Mailing List (not for usage questions)" > >> > , > >> > Date: 05/27/2014 09:44 PM > >> > >> > Subject: Re: [openstack-dev] [neutron] Supporting retries in > >> > neutronclient > >> > > >> > Hi, > >> > >> > > >> > Is it possible to detect when the ssl handshaking error occurs on > >> > the client side (and only retry for that)? If so I think we should > >> > do that rather than retrying multiple times. The danger here is > >> > mostly for POST operations (as Eugene pointed out) where it's > >> > possible for the response to not make it back to the client and for > >> > the operation to actually succeed. > >> > > >> > Having this retry logic nested in the client also prevents things > >> > like nova from handling these types of failures individually since > >> > this retry logic is happening inside of the client. I think it would > >> > be better not to have this internal mechanism in the client and > >> > instead make the user of the client implement retry so they are > >> > aware of failures. > >> > > >> > Aaron > >> >
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
Hi Paul, Just out of curiosity, I am assuming you are using the client that still relies on httplib2. Patch [1] replaced httplib2 with requests, but I believe that a new client that incorporates this change has not yet been published. I wonder if the failures you are referring to manifest themselves with the former http library rather than the latter. Could you clarify? Thanks, Armando [1] - https://review.openstack.org/#/c/89879/ On 29 May 2014 17:25, Paul Ward wrote: > Well, for my specific error, it was an intermittent ssl handshake error > before the request was ever sent to the > neutron-server. In our case, we saw that 4 out of 5 resize operations > worked, the fifth failed with this ssl > handshake error in neutronclient. > > I certainly think a GET is safe to retry, and I agree with your statement > that PUTs and DELETEs probably > are as well. This still leaves a change in nova needing to be made to > actually a) specify a conf option and > b) pass it to neutronclient where appropriate. > > > Aaron Rosen wrote on 05/28/2014 07:38:56 PM: > >> From: Aaron Rosen > > >> To: "OpenStack Development Mailing List (not for usage questions)" >> , >> Date: 05/28/2014 07:44 PM > >> Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient >> >> Hi, >> >> I'm curious if other openstack clients implement this type of retry >> thing. I think retrying on GET/DELETES/PUT's should probably be okay. >> >> What types of errors do you see in the neutron-server when it fails >> to respond? I think it would be better to move the retry logic into >> the server around the failures rather than the client (or better yet >> if we fixed the server :)). Most of the times I've seen this type of >> failure is due to deadlock errors caused between (sqlalchemy and >> eventlet *i think*) which cause the client to eventually timeout. >> >> Best, >> >> Aaron >> > >> On Wed, May 28, 2014 at 11:51 AM, Paul Ward wrote: >> Would it be feasible to make the retry logic only apply to read-only >> operations? This would still require a nova change to specify the >> number of retries, but it'd also prevent invokers from shooting >> themselves in the foot if they call for a write operation. >> >> >> >> Aaron Rosen wrote on 05/27/2014 09:40:00 PM: >> >> > From: Aaron Rosen >> >> > To: "OpenStack Development Mailing List (not for usage questions)" >> > , >> > Date: 05/27/2014 09:44 PM >> >> > Subject: Re: [openstack-dev] [neutron] Supporting retries in >> > neutronclient >> > >> > Hi, >> >> > >> > Is it possible to detect when the ssl handshaking error occurs on >> > the client side (and only retry for that)? If so I think we should >> > do that rather than retrying multiple times. The danger here is >> > mostly for POST operations (as Eugene pointed out) where it's >> > possible for the response to not make it back to the client and for >> > the operation to actually succeed. >> > >> > Having this retry logic nested in the client also prevents things >> > like nova from handling these types of failures individually since >> > this retry logic is happening inside of the client. I think it would >> > be better not to have this internal mechanism in the client and >> > instead make the user of the client implement retry so they are >> > aware of failures. >> > >> > Aaron >> > >> >> > On Tue, May 27, 2014 at 10:48 AM, Paul Ward wrote: >> > Currently, neutronclient is hardcoded to only try a request once in >> > retry_request by virtue of the fact that it uses self.retries as the >> > retry count, and that's initialized to 0 and never changed. We've >> > seen an issue where we get an ssl handshaking error intermittently >> > (seems like more of an ssl bug) and a retry would probably have >> > worked. Yet, since neutronclient only tries once and gives up, it >> > fails the entire operation. Here is the code in question: >> > >> > https://github.com/openstack/python-neutronclient/blob/master/ >> > neutronclient/v2_0/client.py#L1296 >> > >> > Does anybody know if there's some explicit reason we don't currently >> > allow configuring the number of retries? If not, I'm inclined to >> > propose a change for just that. >> > >> > _
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
Well, for my specific error, it was an intermittent ssl handshake error before the request was ever sent to the neutron-server. In our case, we saw that 4 out of 5 resize operations worked, the fifth failed with this ssl handshake error in neutronclient. I certainly think a GET is safe to retry, and I agree with your statement that PUTs and DELETEs probably are as well. This still leaves a change in nova needing to be made to actually a) specify a conf option and b) pass it to neutronclient where appropriate. Aaron Rosen wrote on 05/28/2014 07:38:56 PM: > From: Aaron Rosen > To: "OpenStack Development Mailing List (not for usage questions)" > , > Date: 05/28/2014 07:44 PM > Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient > > Hi, > > I'm curious if other openstack clients implement this type of retry > thing. I think retrying on GET/DELETES/PUT's should probably be okay. > > What types of errors do you see in the neutron-server when it fails > to respond? I think it would be better to move the retry logic into > the server around the failures rather than the client (or better yet > if we fixed the server :)). Most of the times I've seen this type of > failure is due to deadlock errors caused between (sqlalchemy and > eventlet *i think*) which cause the client to eventually timeout. > > Best, > > Aaron > > On Wed, May 28, 2014 at 11:51 AM, Paul Ward wrote: > Would it be feasible to make the retry logic only apply to read-only > operations? This would still require a nova change to specify the > number of retries, but it'd also prevent invokers from shooting > themselves in the foot if they call for a write operation. > > > > Aaron Rosen wrote on 05/27/2014 09:40:00 PM: > > > From: Aaron Rosen > > > To: "OpenStack Development Mailing List (not for usage questions)" > > , > > Date: 05/27/2014 09:44 PM > > > Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient > > > > Hi, > > > > > Is it possible to detect when the ssl handshaking error occurs on > > the client side (and only retry for that)? If so I think we should > > do that rather than retrying multiple times. The danger here is > > mostly for POST operations (as Eugene pointed out) where it's > > possible for the response to not make it back to the client and for > > the operation to actually succeed. > > > > Having this retry logic nested in the client also prevents things > > like nova from handling these types of failures individually since > > this retry logic is happening inside of the client. I think it would > > be better not to have this internal mechanism in the client and > > instead make the user of the client implement retry so they are > > aware of failures. > > > > Aaron > > > > > On Tue, May 27, 2014 at 10:48 AM, Paul Ward wrote: > > Currently, neutronclient is hardcoded to only try a request once in > > retry_request by virtue of the fact that it uses self.retries as the > > retry count, and that's initialized to 0 and never changed. We've > > seen an issue where we get an ssl handshaking error intermittently > > (seems like more of an ssl bug) and a retry would probably have > > worked. Yet, since neutronclient only tries once and gives up, it > > fails the entire operation. Here is the code in question: > > > > https://github.com/openstack/python-neutronclient/blob/master/ > > neutronclient/v2_0/client.py#L1296 > > > > Does anybody know if there's some explicit reason we don't currently > > allow configuring the number of retries? If not, I'm inclined to > > propose a change for just that. > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
Hi, I'm curious if other openstack clients implement this type of retry thing. I think retrying on GET/DELETES/PUT's should probably be okay. What types of errors do you see in the neutron-server when it fails to respond? I think it would be better to move the retry logic into the server around the failures rather than the client (or better yet if we fixed the server :)). Most of the times I've seen this type of failure is due to deadlock errors caused between (sqlalchemy and eventlet *i think*) which cause the client to eventually timeout. Best, Aaron On Wed, May 28, 2014 at 11:51 AM, Paul Ward wrote: > Would it be feasible to make the retry logic only apply to read-only > operations? This would still require a nova change to specify the number > of retries, but it'd also prevent invokers from shooting themselves in the > foot if they call for a write operation. > > > > Aaron Rosen wrote on 05/27/2014 09:40:00 PM: > > > From: Aaron Rosen > > > To: "OpenStack Development Mailing List (not for usage questions)" > > , > > Date: 05/27/2014 09:44 PM > > > Subject: Re: [openstack-dev] [neutron] Supporting retries in > neutronclient > > > > Hi, > > > > > Is it possible to detect when the ssl handshaking error occurs on > > the client side (and only retry for that)? If so I think we should > > do that rather than retrying multiple times. The danger here is > > mostly for POST operations (as Eugene pointed out) where it's > > possible for the response to not make it back to the client and for > > the operation to actually succeed. > > > > Having this retry logic nested in the client also prevents things > > like nova from handling these types of failures individually since > > this retry logic is happening inside of the client. I think it would > > be better not to have this internal mechanism in the client and > > instead make the user of the client implement retry so they are > > aware of failures. > > > > Aaron > > > > > On Tue, May 27, 2014 at 10:48 AM, Paul Ward wrote: > > Currently, neutronclient is hardcoded to only try a request once in > > retry_request by virtue of the fact that it uses self.retries as the > > retry count, and that's initialized to 0 and never changed. We've > > seen an issue where we get an ssl handshaking error intermittently > > (seems like more of an ssl bug) and a retry would probably have > > worked. Yet, since neutronclient only tries once and gives up, it > > fails the entire operation. Here is the code in question: > > > > https://github.com/openstack/python-neutronclient/blob/master/ > > neutronclient/v2_0/client.py#L1296 > > > > Does anybody know if there's some explicit reason we don't currently > > allow configuring the number of retries? If not, I'm inclined to > > propose a change for just that. > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
Would it be feasible to make the retry logic only apply to read-only operations? This would still require a nova change to specify the number of retries, but it'd also prevent invokers from shooting themselves in the foot if they call for a write operation. Aaron Rosen wrote on 05/27/2014 09:40:00 PM: > From: Aaron Rosen > To: "OpenStack Development Mailing List (not for usage questions)" > , > Date: 05/27/2014 09:44 PM > Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient > > Hi, > > Is it possible to detect when the ssl handshaking error occurs on > the client side (and only retry for that)? If so I think we should > do that rather than retrying multiple times. The danger here is > mostly for POST operations (as Eugene pointed out) where it's > possible for the response to not make it back to the client and for > the operation to actually succeed. > > Having this retry logic nested in the client also prevents things > like nova from handling these types of failures individually since > this retry logic is happening inside of the client. I think it would > be better not to have this internal mechanism in the client and > instead make the user of the client implement retry so they are > aware of failures. > > Aaron > > On Tue, May 27, 2014 at 10:48 AM, Paul Ward wrote: > Currently, neutronclient is hardcoded to only try a request once in > retry_request by virtue of the fact that it uses self.retries as the > retry count, and that's initialized to 0 and never changed. We've > seen an issue where we get an ssl handshaking error intermittently > (seems like more of an ssl bug) and a retry would probably have > worked. Yet, since neutronclient only tries once and gives up, it > fails the entire operation. Here is the code in question: > > https://github.com/openstack/python-neutronclient/blob/master/ > neutronclient/v2_0/client.py#L1296 > > Does anybody know if there's some explicit reason we don't currently > allow configuring the number of retries? If not, I'm inclined to > propose a change for just that. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
On Tue, May 27, 2014 at 8:13 PM, Mike Spreitzer wrote: > Joe Gordon wrote on 05/27/2014 07:31:16 PM: > > > From: Joe Gordon > > To: "OpenStack Development Mailing List (not for usage questions)" > > , > > Date: 05/27/2014 07:32 PM > > Subject: Re: [openstack-dev] [neutron] Supporting retries in > neutronclient > > > > > > > On Tue, May 27, 2014 at 1:51 PM, Eugene Nikanorov < > enikano...@mirantis.com > > > wrote: > > In fact, nova should be careful about changing number of retries for > > neutron client. > > It's known that under significant load (people test serial VM > > creation) neutron client may timeout on POST operation which does > > port creation; retrying this again leads to multiple fixed IPs > > assigned to a VM > > > > This sounds like a bug in nova, nova should be able to handle > > intermittent service timeouts without doing the wrong thing. > > It's just one of many bugs that are unavoidable consequences of APIs that > do not support idempotent usage. > ++ > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
Joe Gordon wrote on 05/27/2014 07:31:16 PM: > From: Joe Gordon > To: "OpenStack Development Mailing List (not for usage questions)" > , > Date: 05/27/2014 07:32 PM > Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient > > > On Tue, May 27, 2014 at 1:51 PM, Eugene Nikanorov > wrote: > In fact, nova should be careful about changing number of retries for > neutron client. > It's known that under significant load (people test serial VM > creation) neutron client may timeout on POST operation which does > port creation; retrying this again leads to multiple fixed IPs > assigned to a VM > > This sounds like a bug in nova, nova should be able to handle > intermittent service timeouts without doing the wrong thing. It's just one of many bugs that are unavoidable consequences of APIs that do not support idempotent usage. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
Hi, Is it possible to detect when the ssl handshaking error occurs on the client side (and only retry for that)? If so I think we should do that rather than retrying multiple times. The danger here is mostly for POST operations (as Eugene pointed out) where it's possible for the response to not make it back to the client and for the operation to actually succeed. Having this retry logic nested in the client also prevents things like nova from handling these types of failures individually since this retry logic is happening inside of the client. I think it would be better not to have this internal mechanism in the client and instead make the user of the client implement retry so they are aware of failures. Aaron On Tue, May 27, 2014 at 10:48 AM, Paul Ward wrote: > Currently, neutronclient is hardcoded to only try a request once in > retry_request by virtue of the fact that it uses self.retries as the retry > count, and that's initialized to 0 and never changed. We've seen an issue > where we get an ssl handshaking error intermittently (seems like more of an > ssl bug) and a retry would probably have worked. Yet, since neutronclient > only tries once and gives up, it fails the entire operation. Here is the > code in question: > > > https://github.com/openstack/python-neutronclient/blob/master/neutronclient/v2_0/client.py#L1296 > > Does anybody know if there's some explicit reason we don't currently allow > configuring the number of retries? If not, I'm inclined to propose a > change for just that. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
On Tue, May 27, 2014 at 1:51 PM, Eugene Nikanorov wrote: > In fact, nova should be careful about changing number of retries for > neutron client. > It's known that under significant load (people test serial VM creation) > neutron client may timeout on POST operation which does port creation; > retrying this again leads to multiple fixed IPs assigned to a VM > This sounds like a bug in nova, nova should be able to handle intermittent service timeouts without doing the wrong thing. > > Thanks, > Eugene. > > > On Wed, May 28, 2014 at 12:09 AM, Kyle Mestery > wrote: > >> I'm not aware of any such change at the moment, no. >> >> On Tue, May 27, 2014 at 3:06 PM, Paul Ward wrote: >> > Great! Do you know if there's any corresponding nova changes to support >> > this as a conf option that gets passed in to this new parm? >> > >> > >> > >> > Kyle Mestery wrote on 05/27/2014 01:56:12 >> PM: >> > >> >> From: Kyle Mestery >> >> To: "OpenStack Development Mailing List (not for usage questions)" >> >> , >> >> Date: 05/27/2014 02:00 PM >> >> Subject: Re: [openstack-dev] [neutron] Supporting retries in >> neutronclient >> > >> > >> >> >> >> On Tue, May 27, 2014 at 12:48 PM, Paul Ward wrote: >> >> > Currently, neutronclient is hardcoded to only try a request once in >> >> > retry_request by virtue of the fact that it uses self.retries as the >> >> > retry >> >> > count, and that's initialized to 0 and never changed. We've seen an >> >> > issue >> >> > where we get an ssl handshaking error intermittently (seems like >> more of >> >> > an >> >> > ssl bug) and a retry would probably have worked. Yet, since >> >> > neutronclient >> >> > only tries once and gives up, it fails the entire operation. Here is >> >> > the >> >> > code in question: >> >> > >> >> > https://github.com/openstack/python-neutronclient/blob/master/ >> >> neutronclient/v2_0/client.py#L1296 >> >> > >> >> > Does anybody know if there's some explicit reason we don't currently >> >> > allow >> >> > configuring the number of retries? If not, I'm inclined to propose a >> >> > change >> >> > for just that. >> >> > >> >> There is a review to address this in place now [1]. I've given a -1 >> >> due to a trivial reason which I hope Jakub will address soon so we can >> >> land this patch in the client code. >> >> >> >> Thanks, >> >> Kyle >> >> >> >> [1] https://review.openstack.org/#/c/90104/ >> >> >> >> > >> >> > ___ >> >> > OpenStack-dev mailing list >> >> > OpenStack-dev@lists.openstack.org >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > >> >> >> >> ___ >> >> OpenStack-dev mailing list >> >> OpenStack-dev@lists.openstack.org >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> > >> > >> > ___ >> > OpenStack-dev mailing list >> > OpenStack-dev@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
That is great information, thanks Eugene. Eugene Nikanorov wrote on 05/27/2014 03:51:36 PM: > From: Eugene Nikanorov > To: "OpenStack Development Mailing List (not for usage questions)" > , > Date: 05/27/2014 03:56 PM > Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient > > In fact, nova should be careful about changing number of retries for > neutron client. > It's known that under significant load (people test serial VM > creation) neutron client may timeout on POST operation which does > port creation; retrying this again leads to multiple fixed IPs > assigned to a VM > > Thanks, > Eugene. > > On Wed, May 28, 2014 at 12:09 AM, Kyle Mestery > wrote: > I'm not aware of any such change at the moment, no. > > On Tue, May 27, 2014 at 3:06 PM, Paul Ward wrote: > > Great! Do you know if there's any corresponding nova changes to support > > this as a conf option that gets passed in to this new parm? > > > > > > > > Kyle Mestery wrote on 05/27/2014 01:56:12 PM: > > > >> From: Kyle Mestery > >> To: "OpenStack Development Mailing List (not for usage questions)" > >> , > >> Date: 05/27/2014 02:00 PM > >> Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient > > > > > >> > >> On Tue, May 27, 2014 at 12:48 PM, Paul Ward wrote: > >> > Currently, neutronclient is hardcoded to only try a request once in > >> > retry_request by virtue of the fact that it uses self.retries as the > >> > retry > >> > count, and that's initialized to 0 and never changed. We've seen an > >> > issue > >> > where we get an ssl handshaking error intermittently (seems like more of > >> > an > >> > ssl bug) and a retry would probably have worked. Yet, since > >> > neutronclient > >> > only tries once and gives up, it fails the entire operation. Here is > >> > the > >> > code in question: > >> > > >> > https://github.com/openstack/python-neutronclient/blob/master/ > >> neutronclient/v2_0/client.py#L1296 > >> > > >> > Does anybody know if there's some explicit reason we don't currently > >> > allow > >> > configuring the number of retries? If not, I'm inclined to propose a > >> > change > >> > for just that. > >> > > >> There is a review to address this in place now [1]. I've given a -1 > >> due to a trivial reason which I hope Jakub will address soon so we can > >> land this patch in the client code. > >> > >> Thanks, > >> Kyle > >> > >> [1] https://review.openstack.org/#/c/90104/ > >> > >> > > >> > ___ > >> > OpenStack-dev mailing list > >> > OpenStack-dev@lists.openstack.org > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > >> > >> ___ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
In fact, nova should be careful about changing number of retries for neutron client. It's known that under significant load (people test serial VM creation) neutron client may timeout on POST operation which does port creation; retrying this again leads to multiple fixed IPs assigned to a VM Thanks, Eugene. On Wed, May 28, 2014 at 12:09 AM, Kyle Mestery wrote: > I'm not aware of any such change at the moment, no. > > On Tue, May 27, 2014 at 3:06 PM, Paul Ward wrote: > > Great! Do you know if there's any corresponding nova changes to support > > this as a conf option that gets passed in to this new parm? > > > > > > > > Kyle Mestery wrote on 05/27/2014 01:56:12 > PM: > > > >> From: Kyle Mestery > >> To: "OpenStack Development Mailing List (not for usage questions)" > >> , > >> Date: 05/27/2014 02:00 PM > >> Subject: Re: [openstack-dev] [neutron] Supporting retries in > neutronclient > > > > > >> > >> On Tue, May 27, 2014 at 12:48 PM, Paul Ward wrote: > >> > Currently, neutronclient is hardcoded to only try a request once in > >> > retry_request by virtue of the fact that it uses self.retries as the > >> > retry > >> > count, and that's initialized to 0 and never changed. We've seen an > >> > issue > >> > where we get an ssl handshaking error intermittently (seems like more > of > >> > an > >> > ssl bug) and a retry would probably have worked. Yet, since > >> > neutronclient > >> > only tries once and gives up, it fails the entire operation. Here is > >> > the > >> > code in question: > >> > > >> > https://github.com/openstack/python-neutronclient/blob/master/ > >> neutronclient/v2_0/client.py#L1296 > >> > > >> > Does anybody know if there's some explicit reason we don't currently > >> > allow > >> > configuring the number of retries? If not, I'm inclined to propose a > >> > change > >> > for just that. > >> > > >> There is a review to address this in place now [1]. I've given a -1 > >> due to a trivial reason which I hope Jakub will address soon so we can > >> land this patch in the client code. > >> > >> Thanks, > >> Kyle > >> > >> [1] https://review.openstack.org/#/c/90104/ > >> > >> > > >> > ___ > >> > OpenStack-dev mailing list > >> > OpenStack-dev@lists.openstack.org > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > >> > >> ___ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
Great! Do you know if there's any corresponding nova changes to support this as a conf option that gets passed in to this new parm? Kyle Mestery wrote on 05/27/2014 01:56:12 PM: > From: Kyle Mestery > To: "OpenStack Development Mailing List (not for usage questions)" > , > Date: 05/27/2014 02:00 PM > Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient > > On Tue, May 27, 2014 at 12:48 PM, Paul Ward wrote: > > Currently, neutronclient is hardcoded to only try a request once in > > retry_request by virtue of the fact that it uses self.retries as the retry > > count, and that's initialized to 0 and never changed. We've seen an issue > > where we get an ssl handshaking error intermittently (seems like more of an > > ssl bug) and a retry would probably have worked. Yet, since neutronclient > > only tries once and gives up, it fails the entire operation. Here is the > > code in question: > > > > https://github.com/openstack/python-neutronclient/blob/master/ > neutronclient/v2_0/client.py#L1296 > > > > Does anybody know if there's some explicit reason we don't currently allow > > configuring the number of retries? If not, I'm inclined to propose a change > > for just that. > > > There is a review to address this in place now [1]. I've given a -1 > due to a trivial reason which I hope Jakub will address soon so we can > land this patch in the client code. > > Thanks, > Kyle > > [1] https://review.openstack.org/#/c/90104/ > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
I'm not aware of any such change at the moment, no. On Tue, May 27, 2014 at 3:06 PM, Paul Ward wrote: > Great! Do you know if there's any corresponding nova changes to support > this as a conf option that gets passed in to this new parm? > > > > Kyle Mestery wrote on 05/27/2014 01:56:12 PM: > >> From: Kyle Mestery >> To: "OpenStack Development Mailing List (not for usage questions)" >> , >> Date: 05/27/2014 02:00 PM >> Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient > > >> >> On Tue, May 27, 2014 at 12:48 PM, Paul Ward wrote: >> > Currently, neutronclient is hardcoded to only try a request once in >> > retry_request by virtue of the fact that it uses self.retries as the >> > retry >> > count, and that's initialized to 0 and never changed. We've seen an >> > issue >> > where we get an ssl handshaking error intermittently (seems like more of >> > an >> > ssl bug) and a retry would probably have worked. Yet, since >> > neutronclient >> > only tries once and gives up, it fails the entire operation. Here is >> > the >> > code in question: >> > >> > https://github.com/openstack/python-neutronclient/blob/master/ >> neutronclient/v2_0/client.py#L1296 >> > >> > Does anybody know if there's some explicit reason we don't currently >> > allow >> > configuring the number of retries? If not, I'm inclined to propose a >> > change >> > for just that. >> > >> There is a review to address this in place now [1]. I've given a -1 >> due to a trivial reason which I hope Jakub will address soon so we can >> land this patch in the client code. >> >> Thanks, >> Kyle >> >> [1] https://review.openstack.org/#/c/90104/ >> >> > >> > ___ >> > OpenStack-dev mailing list >> > OpenStack-dev@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Supporting retries in neutronclient
On Tue, May 27, 2014 at 12:48 PM, Paul Ward wrote: > Currently, neutronclient is hardcoded to only try a request once in > retry_request by virtue of the fact that it uses self.retries as the retry > count, and that's initialized to 0 and never changed. We've seen an issue > where we get an ssl handshaking error intermittently (seems like more of an > ssl bug) and a retry would probably have worked. Yet, since neutronclient > only tries once and gives up, it fails the entire operation. Here is the > code in question: > > https://github.com/openstack/python-neutronclient/blob/master/neutronclient/v2_0/client.py#L1296 > > Does anybody know if there's some explicit reason we don't currently allow > configuring the number of retries? If not, I'm inclined to propose a change > for just that. > There is a review to address this in place now [1]. I've given a -1 due to a trivial reason which I hope Jakub will address soon so we can land this patch in the client code. Thanks, Kyle [1] https://review.openstack.org/#/c/90104/ > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] Supporting retries in neutronclient
Currently, neutronclient is hardcoded to only try a request once in retry_request by virtue of the fact that it uses self.retries as the retry count, and that's initialized to 0 and never changed. We've seen an issue where we get an ssl handshaking error intermittently (seems like more of an ssl bug) and a retry would probably have worked. Yet, since neutronclient only tries once and gives up, it fails the entire operation. Here is the code in question: https://github.com/openstack/python-neutronclient/blob/master/neutronclient/v2_0/client.py#L1296 Does anybody know if there's some explicit reason we don't currently allow configuring the number of retries? If not, I'm inclined to propose a change for just that.___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
