Hi Paul, Just out of curiosity, I am assuming you are using the client that still relies on httplib2. Patch [1] replaced httplib2 with requests, but I believe that a new client that incorporates this change has not yet been published. I wonder if the failures you are referring to manifest themselves with the former http library rather than the latter. Could you clarify?
Thanks, Armando [1] - https://review.openstack.org/#/c/89879/ On 29 May 2014 17:25, Paul Ward <[email protected]> wrote: > Well, for my specific error, it was an intermittent ssl handshake error > before the request was ever sent to the > neutron-server. In our case, we saw that 4 out of 5 resize operations > worked, the fifth failed with this ssl > handshake error in neutronclient. > > I certainly think a GET is safe to retry, and I agree with your statement > that PUTs and DELETEs probably > are as well. This still leaves a change in nova needing to be made to > actually a) specify a conf option and > b) pass it to neutronclient where appropriate. > > > Aaron Rosen <[email protected]> wrote on 05/28/2014 07:38:56 PM: > >> From: Aaron Rosen <[email protected]> > > >> To: "OpenStack Development Mailing List (not for usage questions)" >> <[email protected]>, >> Date: 05/28/2014 07:44 PM > >> Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient >> >> Hi, >> >> I'm curious if other openstack clients implement this type of retry >> thing. I think retrying on GET/DELETES/PUT's should probably be okay. >> >> What types of errors do you see in the neutron-server when it fails >> to respond? I think it would be better to move the retry logic into >> the server around the failures rather than the client (or better yet >> if we fixed the server :)). Most of the times I've seen this type of >> failure is due to deadlock errors caused between (sqlalchemy and >> eventlet *i think*) which cause the client to eventually timeout. >> >> Best, >> >> Aaron >> > >> On Wed, May 28, 2014 at 11:51 AM, Paul Ward <[email protected]> wrote: >> Would it be feasible to make the retry logic only apply to read-only >> operations? This would still require a nova change to specify the >> number of retries, but it'd also prevent invokers from shooting >> themselves in the foot if they call for a write operation. >> >> >> >> Aaron Rosen <[email protected]> wrote on 05/27/2014 09:40:00 PM: >> >> > From: Aaron Rosen <[email protected]> >> >> > To: "OpenStack Development Mailing List (not for usage questions)" >> > <[email protected]>, >> > Date: 05/27/2014 09:44 PM >> >> > Subject: Re: [openstack-dev] [neutron] Supporting retries in >> > neutronclient >> > >> > Hi, >> >> > >> > Is it possible to detect when the ssl handshaking error occurs on >> > the client side (and only retry for that)? If so I think we should >> > do that rather than retrying multiple times. The danger here is >> > mostly for POST operations (as Eugene pointed out) where it's >> > possible for the response to not make it back to the client and for >> > the operation to actually succeed. >> > >> > Having this retry logic nested in the client also prevents things >> > like nova from handling these types of failures individually since >> > this retry logic is happening inside of the client. I think it would >> > be better not to have this internal mechanism in the client and >> > instead make the user of the client implement retry so they are >> > aware of failures. >> > >> > Aaron >> > >> >> > On Tue, May 27, 2014 at 10:48 AM, Paul Ward <[email protected]> wrote: >> > Currently, neutronclient is hardcoded to only try a request once in >> > retry_request by virtue of the fact that it uses self.retries as the >> > retry count, and that's initialized to 0 and never changed. We've >> > seen an issue where we get an ssl handshaking error intermittently >> > (seems like more of an ssl bug) and a retry would probably have >> > worked. Yet, since neutronclient only tries once and gives up, it >> > fails the entire operation. Here is the code in question: >> > >> > https://github.com/openstack/python-neutronclient/blob/master/ >> > neutronclient/v2_0/client.py#L1296 >> > >> > Does anybody know if there's some explicit reason we don't currently >> > allow configuring the number of retries? If not, I'm inclined to >> > propose a change for just that. >> > >> > _______________________________________________ >> > OpenStack-dev mailing list >> > [email protected] >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > _______________________________________________ >> > OpenStack-dev mailing list >> > [email protected] >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> _______________________________________________ >> OpenStack-dev mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> _______________________________________________ >> OpenStack-dev mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
