Hi Michael, To address your comments, we will be making a small change to the load balancing policy with respect to hedging RPCs. The change will support passing the local lb_policy a list of previously used addresses. The list will essentially be, "if possible, don't choose one of these addresses." For most cases this will solve your concern about the relation between affinity routing and hedging.
These changes will only occur in the local lb_policy. We do not want to send any extra data over the wire due to performance concerns. gRPC support for affinity routing is ongoing, but this change to the existing policy will make it easier to have hedging and affinity routing work together in the future. On Sun, Feb 12, 2017 at 7:26 PM, <elementat...@gmail.com> wrote: > > We are not supporting explicit load balancing constraints for retries. > The retry attempt or hedged RPC will be re-resolved through the > load-balancer, so it's up to the service owner to ensure that this has a > low-likelihood of issuing the request to the same backend. > > That seems fairly difficult for any service with request-dependent routing > semantics. Lets use a DFS as an example: many DFSes maintain N replicas of > a given file block. In the case where you send a hedged request for a > block, your likelihood is 1/N of requerying the same DFS node which might > well have a slow disk. At least for us using HDFS, N=3 most of the time; a > therefore 33% chance of requerying the same node. Even assuming a smart > load balancing service which intelligently removes poorly performing > storage nodes from service, it still seems desirable to ensure hedged > requests go to a different node. Not having a story for more informed load > balancing seems like it makes a lot of use cases more difficult than they > need to be. > > Regards, > Michael > > On Sunday, February 12, 2017 at 7:24:59 PM UTC-7, Eric Gribkoff wrote: >> >> Hi Michael, >> >> Thanks for the feedback. Responses to your questions (and Josh's >> follow-up question on retry backoff times) are inline below. >> >> On Sat, Feb 11, 2017 at 1:57 PM, 'Michael Rose' via grpc.io < >> grp...@googlegroups.com> wrote: >> >>> A few questions: >>> >>> 1) Under this design, is it possible to add a load balancing constraints >>> for retried/hedged requests? Especially during hedging, I'd like to be able >>> to try a different server since the original server might be garbage >>> collecting or have otherwise collected a queue of requests such that a >>> retry/hedge to this server will not be very useful. Or, perhaps the key I'm >>> looking up lives on a specific subset of storage servers and therefore >>> should be balanced to that specific subset. While that's the domain of a LB >>> policy, what information will hedging/retries provide to the LB policy? >>> >>> >> We are not supporting explicit load balancing constraints for retries. >> The retry attempt or hedged RPC will be re-resolved through the >> load-balancer, so it's up to the service owner to ensure that this has a >> low-likelihood of issuing the request to the same backend. This is part of >> a decision to keep the retry design as simple as possible while satisfying >> the majority of use cases. If your load-balancing policy has a high >> likelihood of sending requests to the same server each time, hedging (and >> to some extent retries) will be less useful regardless. There will be >> metadata attached to the call indicating that it's a retry, but it won't >> include information about which servers the previous requests went to. >> >> >> >>> 2) "Clients cannot override retry policy set by the service config." -- >>> is this intended for inside Google? How about gRPC users outside of Google >>> which don't use the DNS mechanism to push configuration? It seems like >>> having a client override for retry/hedging policy is pragmatic. >>> >>> >> In general, we don't want to support client specification of retry >> policies. The necessary information about what methods are safe to retry or >> hedge, the potential for increased load, etc., are really decisions that >> should be left to the service owner. The retry policy will definitely be a >> part of the service config. While there are still some security-related >> discussions about the exact delivery mechanism for the service config and >> retry policies, I think your concern here should be part of the service >> config design discussion rather than something specific to retry support. >> >> >>> 3) Retry backoff time -- if I'm reading it right, it will always retry >>> in random(0, current_backoff) milliseconds. What's your feeling on this vs. >>> a retry w/ configurable jitter parameter (e.x. linear 1000ms increase w/ >>> 10% jitter). Is it OK if there's no minimum backoff? >>> >>> >> You are reading the backoff time correctly. There are a number of ways of >> doing this, (see https://www.awsarchitectureblog.com/2015/03/backoff.html) >> but choosing between random(0, current_backoff) is done intentionally and >> should generally give the best results. We do not want a configurable >> "jitter" parameter. Empirically, the retries should have more varied >> backoff time, and we also do not want to let service owners specify very >> low values for jitter (e.g., 1% or even 0), as this would cluster all >> retries tightly together and further contribute to server overloading. >> >> Best, >> >> Eric Gribkoff >> >> >> Regards, >>> Michael >>> >>> On Friday, February 10, 2017 at 5:31:01 PM UTC-7, ncte...@google.com >>> wrote: >>>> >>>> I've created a gRFC describing the design and implementation plan for >>>> gRPC Retries. >>>> >>>> Take a look at the gRPC on Github >>>> <https://github.com/grpc/proposal/pull/12>. >>>> >>> >>> *CONFIDENTIALITY NOTICE: This email message, and any documents, files or >>> previous e-mail messages attached to it is for the sole use of the intended >>> recipient(s) and may contain confidential and privileged information. Any >>> unauthorized review, use, disclosure or distribution is prohibited. If you >>> are not the intended recipient, please contact the sender by reply email >>> and destroy all copies of the original message.* >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "grpc.io" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to grpc-io+u...@googlegroups.com. >>> To post to this group, send email to grp...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/grpc-io. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/grpc-io/62809dba-3349-4a60-9aa9-ccc044d27f53%40googlegroups.com >>> <https://groups.google.com/d/msgid/grpc-io/62809dba-3349-4a60-9aa9-ccc044d27f53%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups " > grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to grpc-io+unsubscr...@googlegroups.com. > To post to this group, send email to grpc-io@googlegroups.com. > Visit this group at https://groups.google.com/group/grpc-io. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/grpc-io/ce59f63d-1dee-46ff-a3eb-c813d15fc2dc%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/ce59f63d-1dee-46ff-a3eb-c813d15fc2dc%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To post to this group, send email to grpc-io@googlegroups.com. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAPYwnkj6bMiTvgJ5OXVRfvb5rMNy93zZrM7zTPDQ55L0sr_vxA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
