For some more color, we (internally) have made outages worse by retrying on status codes we shouldn't, sometimes through multiple layers of services resulting in essentially DDoSing our own services. For instance if you retry 3 times at each client, and your service passes through N layers, then you have 3^N retries. A service I worked on ended up 4 layers deep with misconfigured retry behavior that resulted in 81 retries per top-level request. That was fun, attempting to slough off ~82x our normal traffic. :)
Also as Mark said, it may not always be correct to retry: not all RPCs are idempotent and may have state implications, so this really should be a case-by-case (and a code-by-code) decision. No sense in retrying something that isn't transient. On Wed, Mar 20, 2019, 08:06 'Mark D. Roth' via grpc.io < [email protected]> wrote: > In general, unless an application is explicitly designed to allow an RPC > to be retried, it's not safe to do so. As a result, we wanted service > owners to make an explicit choice about which ones they deem safe to retry, > rather than accidentally configuring retries in a case where it's not safe. > > On Tue, Mar 19, 2019 at 7:05 PM alun via grpc.io <[email protected]> > wrote: > >> I have a query about: >> >> When gRPC receives a non-OK response status from a server, this status is >>> checked against the set of retryable status codes in retryableStatusCodes >>> to determine if a retry attempt should be made. >> >> >> I was wondering why it wasn't chosen to have a set of fatalStatusCodes, >> to determine if a retry attempt should not be made ? >> >> - Especially with respect to Postel's law. >> >> thanks, >> >> A. >> >> >> On Friday, February 10, 2017 at 4:31:01 PM UTC-8, [email protected] >> wrote: >>> >>> I've created a gRFC describing the design and implementation plan for >>> gRPC Retries. >>> >>> Take a look at the gRPC on Github >>> <https://github.com/grpc/proposal/pull/12>. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "grpc.io" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/grpc-io. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/grpc-io/b78c2861-49ea-4fe3-a0dd-70e5ed199432%40googlegroups.com >> <https://groups.google.com/d/msgid/grpc-io/b78c2861-49ea-4fe3-a0dd-70e5ed199432%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > Mark D. Roth <[email protected]> > Software Engineer > Google, Inc. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "grpc.io" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/grpc-io/zzHIICbwTZE/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/grpc-io. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/CAJgPXp7_9vhjoJEy%2Bb-t%2B70ooZwbZ8FWZte2wiL93M1LAAN6hg%40mail.gmail.com > <https://groups.google.com/d/msgid/grpc-io/CAJgPXp7_9vhjoJEy%2Bb-t%2B70ooZwbZ8FWZte2wiL93M1LAAN6hg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAOMdnbKUhY3TQ8J6CgfieNiW9JW9nTZxAXXe1O%2BDSJBoh6TZsw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
