Hi Michael, Thanks for the feedback. Responses to your questions (and Josh's follow-up question on retry backoff times) are inline below.
On Sat, Feb 11, 2017 at 1:57 PM, 'Michael Rose' via grpc.io < [email protected]> wrote: > A few questions: > > 1) Under this design, is it possible to add a load balancing constraints > for retried/hedged requests? Especially during hedging, I'd like to be able > to try a different server since the original server might be garbage > collecting or have otherwise collected a queue of requests such that a > retry/hedge to this server will not be very useful. Or, perhaps the key I'm > looking up lives on a specific subset of storage servers and therefore > should be balanced to that specific subset. While that's the domain of a LB > policy, what information will hedging/retries provide to the LB policy? > > We are not supporting explicit load balancing constraints for retries. The retry attempt or hedged RPC will be re-resolved through the load-balancer, so it's up to the service owner to ensure that this has a low-likelihood of issuing the request to the same backend. This is part of a decision to keep the retry design as simple as possible while satisfying the majority of use cases. If your load-balancing policy has a high likelihood of sending requests to the same server each time, hedging (and to some extent retries) will be less useful regardless. There will be metadata attached to the call indicating that it's a retry, but it won't include information about which servers the previous requests went to. > 2) "Clients cannot override retry policy set by the service config." -- is > this intended for inside Google? How about gRPC users outside of Google > which don't use the DNS mechanism to push configuration? It seems like > having a client override for retry/hedging policy is pragmatic. > > In general, we don't want to support client specification of retry policies. The necessary information about what methods are safe to retry or hedge, the potential for increased load, etc., are really decisions that should be left to the service owner. The retry policy will definitely be a part of the service config. While there are still some security-related discussions about the exact delivery mechanism for the service config and retry policies, I think your concern here should be part of the service config design discussion rather than something specific to retry support. > 3) Retry backoff time -- if I'm reading it right, it will always retry in > random(0, current_backoff) milliseconds. What's your feeling on this vs. a > retry w/ configurable jitter parameter (e.x. linear 1000ms increase w/ 10% > jitter). Is it OK if there's no minimum backoff? > > You are reading the backoff time correctly. There are a number of ways of doing this, (see https://www.awsarchitectureblog.com/2015/03/backoff.html) but choosing between random(0, current_backoff) is done intentionally and should generally give the best results. We do not want a configurable "jitter" parameter. Empirically, the retries should have more varied backoff time, and we also do not want to let service owners specify very low values for jitter (e.g., 1% or even 0), as this would cluster all retries tightly together and further contribute to server overloading. Best, Eric Gribkoff Regards, > Michael > > On Friday, February 10, 2017 at 5:31:01 PM UTC-7, [email protected] > wrote: >> >> I've created a gRFC describing the design and implementation plan for >> gRPC Retries. >> >> Take a look at the gRPC on Github >> <https://github.com/grpc/proposal/pull/12>. >> > > *CONFIDENTIALITY NOTICE: This email message, and any documents, files or > previous e-mail messages attached to it is for the sole use of the intended > recipient(s) and may contain confidential and privileged information. Any > unauthorized review, use, disclosure or distribution is prohibited. If you > are not the intended recipient, please contact the sender by reply email > and destroy all copies of the original message.* > > -- > You received this message because you are subscribed to the Google Groups " > grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/grpc-io. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/grpc-io/62809dba-3349-4a60-9aa9-ccc044d27f53%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/62809dba-3349-4a60-9aa9-ccc044d27f53%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CALUXJ7hL9Y%2BZo8iyPC2RjtFgSQbHcEmdDQcA4BwHp2hkpiEMhQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
