Re: [grpc-io] gRPC A6: Retries

[liuwenzhe_neoye]

before you provide retry feat, I guess we can use interceptor to do the 
same thing, like: 
https://github.com/grpc-ecosystem/go-grpc-middleware/blob/master/retry/retry.go 
does?

在 2019年2月27日星期三 UTC+8下午11:20:10，Mark D. Roth写道：
>
> Unfortunately, we still can't provide any ETA on this.  We will definitely 
> get back to this at some point, but we've got other higher priority work 
> that will be occupying our attention for the next few quarters, so the 
> soonest we might be able to get back to this would be toward the end of the 
> year.
>
> FWIW, I am personally committed to getting this done at some point, 
> because I've already sunk about a year's worth of time into it, and I don't 
> want that to have been for nothing. :)
>
> On Tue, Feb 26, 2019 at 3:42 PM > wrote:
>
>> Hi, I just wanted to bump this up - any updates on shipping retry support 
>>  for public use? Thanks.
>>
>> On Monday, March 13, 2017 at 1:41:39 PM UTC-7, Mark D. Roth wrote:
>>>
>>> After much discussion with the DNS and security folks, we've decided on 
>>> a way to address the potential security issue of allowing an attacker to 
>>> inject a service config with a large number of retries or hedged requests.  
>>> We will do this by imposing an upper bound on the max number of retries or 
>>> hedged requests that are configurable via the service config.  That upper 
>>> bound will be 5 by default, but applications will be able to explicitly 
>>> override it if needed via a channel argument.
>>>
>>> This approach not only limits the damage that can be caused by a 
>>> malicious attacker but also damage that can be caused by a simple typo.
>>>
>>> Noah, can you please add a section about this to the design doc?  Thanks!
>>>
>>> On Fri, Feb 10, 2017 at 4:31 PM, ncteisen via grpc.io <
>>> grp...@googlegroups.com> wrote:
>>>
 I've created a gRFC describing the design and implementation plan for 
 gRPC Retries.

 Take a look at the gRPC on Github 
 .

 -- 
 You received this message because you are subscribed to the Google 
 Groups "grpc.io" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to grpc-io+u...@googlegroups.com.
 To post to this group, send email to grp...@googlegroups.com.
 Visit this group at https://groups.google.com/group/grpc-io.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/grpc-io/30e29cbc-439c-46c4-b54f-6e97637a0735%40googlegroups.com
  
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>>
>>> -- 
>>> Mark D. Roth 
>>> Software Engineer
>>> Google, Inc.
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to grpc-io+u...@googlegroups.com .
>> To post to this group, send email to grp...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/grpc-io.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/grpc-io/d4134c00-fea6-4e23-a136-5ae730c261e6%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> -- 
> Mark D. Roth >
> Software Engineer
> Google, Inc.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/ac930e31-8f6c-4437-b880-47c0694661e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] gRPC A6: Retries

['Mark D. Roth' via grpc.io]

Unfortunately, we still can't provide any ETA on this.  We will definitely
get back to this at some point, but we've got other higher priority work
that will be occupying our attention for the next few quarters, so the
soonest we might be able to get back to this would be toward the end of the
year.

FWIW, I am personally committed to getting this done at some point, because
I've already sunk about a year's worth of time into it, and I don't want
that to have been for nothing. :)

On Tue, Feb 26, 2019 at 3:42 PM  wrote:

> Hi, I just wanted to bump this up - any updates on shipping retry support
>  for public use? Thanks.
>
> On Monday, March 13, 2017 at 1:41:39 PM UTC-7, Mark D. Roth wrote:
>>
>> After much discussion with the DNS and security folks, we've decided on a
>> way to address the potential security issue of allowing an attacker to
>> inject a service config with a large number of retries or hedged requests.
>> We will do this by imposing an upper bound on the max number of retries or
>> hedged requests that are configurable via the service config.  That upper
>> bound will be 5 by default, but applications will be able to explicitly
>> override it if needed via a channel argument.
>>
>> This approach not only limits the damage that can be caused by a
>> malicious attacker but also damage that can be caused by a simple typo.
>>
>> Noah, can you please add a section about this to the design doc?  Thanks!
>>
>> On Fri, Feb 10, 2017 at 4:31 PM, ncteisen via grpc.io <
>> grp...@googlegroups.com> wrote:
>>
>>> I've created a gRFC describing the design and implementation plan for
>>> gRPC Retries.
>>>
>>> Take a look at the gRPC on Github
>>> .
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "grpc.io" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to grpc-io+u...@googlegroups.com.
>>> To post to this group, send email to grp...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/grpc-io.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/grpc-io/30e29cbc-439c-46c4-b54f-6e97637a0735%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Mark D. Roth 
>> Software Engineer
>> Google, Inc.
>>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/d4134c00-fea6-4e23-a136-5ae730c261e6%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Mark D. Roth 
Software Engineer
Google, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAJgPXp5Q%2B-Bm4DsxYNMxKzO9kP4B6_hTAZ%2BQzuPM2dGvCGMzAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] gRPC A6: Retries

[oleg]

Hi, I just wanted to bump this up - any updates on shipping retry support 
 for public use? Thanks.

On Monday, March 13, 2017 at 1:41:39 PM UTC-7, Mark D. Roth wrote:
>
> After much discussion with the DNS and security folks, we've decided on a 
> way to address the potential security issue of allowing an attacker to 
> inject a service config with a large number of retries or hedged requests.  
> We will do this by imposing an upper bound on the max number of retries or 
> hedged requests that are configurable via the service config.  That upper 
> bound will be 5 by default, but applications will be able to explicitly 
> override it if needed via a channel argument.
>
> This approach not only limits the damage that can be caused by a malicious 
> attacker but also damage that can be caused by a simple typo.
>
> Noah, can you please add a section about this to the design doc?  Thanks!
>
> On Fri, Feb 10, 2017 at 4:31 PM, ncteisen via grpc.io <
> grp...@googlegroups.com > wrote:
>
>> I've created a gRFC describing the design and implementation plan for 
>> gRPC Retries.
>>
>> Take a look at the gRPC on Github 
>> .
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to grpc-io+u...@googlegroups.com .
>> To post to this group, send email to grp...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/grpc-io.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/grpc-io/30e29cbc-439c-46c4-b54f-6e97637a0735%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Mark D. Roth >
> Software Engineer
> Google, Inc.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/d4134c00-fea6-4e23-a136-5ae730c261e6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] gRPC A6: Retries

['Mark D. Roth' via grpc.io]

After much discussion with the DNS and security folks, we've decided on a
way to address the potential security issue of allowing an attacker to
inject a service config with a large number of retries or hedged requests.
We will do this by imposing an upper bound on the max number of retries or
hedged requests that are configurable via the service config.  That upper
bound will be 5 by default, but applications will be able to explicitly
override it if needed via a channel argument.

This approach not only limits the damage that can be caused by a malicious
attacker but also damage that can be caused by a simple typo.

Noah, can you please add a section about this to the design doc?  Thanks!

On Fri, Feb 10, 2017 at 4:31 PM, ncteisen via grpc.io <
grpc-io@googlegroups.com> wrote:

> I've created a gRFC describing the design and implementation plan for gRPC
> Retries.
>
> Take a look at the gRPC on Github
> .
>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/grpc-io/30e29cbc-439c-46c4-b54f-6e97637a0735%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Mark D. Roth 
Software Engineer
Google, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAJgPXp6Hws5dbbajfrjXb_RgwM4ZqhB9V4C5VsVL2r5wS1ygUQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] gRPC A6: Retries

['Eric Anderson' via grpc.io]

I see that retries add the x-grpc-retry-pushback-ms
and x-grpc-retry-attempts metadata keys. Is there a reason to prefix them
with the x-, even though the rest of the grpc keys just use the grpc-
prefix? I didn't see any discussion on that.

I also saw this in the spec:

> The value for this field will be a human-readable integer.


I'm not sure that really contributes anything. "three", "0x3", "03", and
"三" are all human-readable. I'd assume something akin to "base 10-encoded,
positive integer, without unnecessary leading zeros" is probably intended.
I'm also assuming not to send the metadata on the initial request, but that
would be another assumption.

I'd also note x-grpc-retry-pushback-ms doesn't define the format, although
the mention of the special-case -1 does imply some about it.

On Fri, Feb 10, 2017 at 4:31 PM, ncteisen via grpc.io <
grpc-io@googlegroups.com> wrote:

> I've created a gRFC describing the design and implementation plan for gRPC
> Retries.
>
> Take a look at the gRPC on Github
> .
>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/grpc-io/30e29cbc-439c-46c4-b54f-6e97637a0735%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CA%2B4M1oNjc_XmPPy2X0B9TQ-BfLn5M_w2PyJiGn%2BSYwV2pECSJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [grpc-io] gRPC A6: Retries

['Eric Gribkoff' via grpc.io]

On Thu, Mar 2, 2017 at 9:03 AM, 'Eric Anderson' via grpc.io <
grpc-io@googlegroups.com> wrote:

> On Thu, Mar 2, 2017 at 8:38 AM, Mark D. Roth  wrote:
>
>> On Thu, Mar 2, 2017 at 8:24 AM, Eric Gribkoff 
>> wrote:
>>
>>> On Thu, Mar 2, 2017 at 8:15 AM, Mark D. Roth  wrote:

 I agree that we don't need to say anything about whether or not the
 server delays sending Response-Headers until a message is sent.  However, I
 think we should say that if the server is going to immediately signal
 failure without sending any messages, it should send Trailers-Only instead
 of Response-Headers followed by Trailers.


>>>
>>> This is in the retry gRFC doc now (https://github.com/ncteisen/p
>>> roposal/blob/ad060be281c45c262e71a56e5777d26616dad69f/A6.md#
>>> when-retries-are-valid).
>>>
>>
> The language is still confusing:
>
>> The client receives a non-error response from the server. Because of the
>> gRPC wire specification, this will always be a Response-Headers frame
>> containing the initial metadata.
>
>
> What does "non-error response" mean there? I would have expected that
> means receiving a Status in some way (which is part of Response), as
> otherwise how is "error" decided. But the next part shows that isn't the
> case since Status isn't in Response-Headers.
>
>
The second sentence is defining what non-error response means: a
Response-Headers frame. The only alternative (an "error" response) is
Trailers-Only. I can chose a name other than "non-error response" to make
this clear.


> The wire spec *almost* says it: "Trailers-Only is permitted for calls
>>> that produce an immediate error" (https://github.com/grpc/grpc/
>>> blob/master/doc/PROTOCOL-HTTP2.md). Do you want this changed in the
>>> wire spec itself or is the inclusion in the gRFC for retries sufficient?
>>>
>>
>> I think it would be good to also change the wire spec doc.  We should do
>> something like changing "is permitted" to "SHOULD be used".  We may even
>> want to specifically mention that this is important for retry functionality
>> to work right.
>>
>
> Changing to 'should' sounds fine. Although maybe there should be a note
> that clients can't decide if something is an 'immediate error' so there
> must not be any validation for it client-side.
>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/grpc-io/CA%2B4M1oON-6sgSW%3DLLJZLABLm_RFCFgNb%
> 2Bki6%2BbwJuxMMPXMxUA%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CALUXJ7j%3DbnPCNgNZw5m444r3eaaVYCYSuY%2BwaM95oQhK-xAdvA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] gRPC A6: Retries

['Eric Gribkoff' via grpc.io]

On Thu, Mar 2, 2017 at 8:15 AM, Mark D. Roth  wrote:

> On Thu, Mar 2, 2017 at 8:09 AM, Eric Gribkoff 
> wrote:
>
>> I've update the gRFC document to include the latest discussions here.
>>
>> On Thu, Mar 2, 2017 at 7:20 AM, Mark D. Roth  wrote:
>>
>>> On Wed, Mar 1, 2017 at 2:47 PM, 'Eric Gribkoff' via grpc.io <
>>> grpc-io@googlegroups.com> wrote:
>>>
 I think the terminology here gets confusing between initial/trailing
 metadata, gRPC rule names, and HTTP/2 frame types. Our retry design doc was
 indeed underspecified in regards to dealing with initial metadata, and will
 be updated. I go over all of the considerations in detail below.

 For clarity, I will use all caps for the names of HTTP/2 frame types,
 e.g., HEADERS frame, and use the capitalized gRPC rule names from the
 specification
 
 .

 The gRPC specification ensures that a status (containing a gRPC status
 code) is only sent in Trailers, which is contained in an HTTP/2 HEADERS
 frame. The only way that the gRPC status code can be contained in the first
 HTTP/2 frame received is if the server sends a Trailers-Only response.

 Otherwise, the gRPC spec mandates that the first frame sent be the
 Response-Headers (again, sent in an HTTP/2 HEADERS frame). Response-Headers
 includes (optional) Custom-Metadata, which is usually what we are talking
 about when we say "initial metadata".

 Regardless of whether the Response-Headers includes anything in its
 Custom-Metadata, if the gRPC client library notifies the client application
 layer of what metadata is (or is not) included, we now have to view the RPC
 as committed, aka no longer retryable. This is the only option, as a later
 retry attempt could receive different Custom-Metadata, contradicting what
 we've already told the client application layer.

 We cannot include gRPC status codes in the Response-Headers along with
 "initial metadata". It's perfectly valid according to the spec for a server
 to send metadata along a stream in its Response-Headers, wait for one hour,
 then (without having sent any messages), close the stream with a retryable
 error.

 However, the proposal that a server include the gRPC status code (if
 known) in the initial response is still sound. Concretely, this means: if a
 gRPC server has not yet sent Response-Headers and receives an error
 response, it should send a Trailers-Only response containing the gRPC
 status code. This would allow retry attempts on the client-side to proceed,
 if applicable. This is going to be superior to sending Response-Headers
 immediately followed by Trailers, which would cause the RPC to become
 committed on the client side (if the Response-Header metadata is made
 available to the client application layer) and stop retry attempts.

 We still can encounter the case where a server intentionally sends
 Response-Headers to open a stream, then eventually closes the stream with
 an error without ever sending any messages. Such cases would not be
 retryable, but I think it's fair to argue that if the server *has* to send
 metadata in advance of sending any responses, that metadata is actually a
 response, and should be treated as such (i.e., their metadata just ensured
 the RPC will be committed on the client-side).

 Rather than either explicitly disallowing such behavior by modifying
 some specification (this behavior is currently entirely unspecified, so
 while specification is worthwhile, it should be separate from the retry
 policy design currently under discussion), we can just change the default
 server behavior of C++, and Go if necessary, to match Java. In Java
 servers, the Response-Headers are delayed until some response message is
 sent. If the server application returns an error status before sending a
 message, then Trailers-Only is sent instead of Response-Headers.

 We can also leave it up to the gRPC client library implementation to
 decide when an RPC is committed based on received Response-Headers. If and
 while the client library can guarantee that the presence (or absence) of
 initial metadata is not visible to the client application layer, the RPC
 can be considered uncommitted. This is an implementation detail that should
 very rarely be necessary if the above change is made to default server
 behavior, but it would not violate anything in the retry spec or semantics.

>>>
>>> I think that leaving this unspecified will lead to interoperability
>>> problems in the future.  I would rather have the spec be explicit about
>>> this, so that all future client and server implementations can interoperate

Re: [grpc-io] gRPC A6: Retries

['Mark D. Roth' via grpc.io]

On Thu, Mar 2, 2017 at 8:09 AM, Eric Gribkoff 
wrote:

> I've update the gRFC document to include the latest discussions here.
>
> On Thu, Mar 2, 2017 at 7:20 AM, Mark D. Roth  wrote:
>
>> On Wed, Mar 1, 2017 at 2:47 PM, 'Eric Gribkoff' via grpc.io <
>> grpc-io@googlegroups.com> wrote:
>>
>>> I think the terminology here gets confusing between initial/trailing
>>> metadata, gRPC rule names, and HTTP/2 frame types. Our retry design doc was
>>> indeed underspecified in regards to dealing with initial metadata, and will
>>> be updated. I go over all of the considerations in detail below.
>>>
>>> For clarity, I will use all caps for the names of HTTP/2 frame types,
>>> e.g., HEADERS frame, and use the capitalized gRPC rule names from the
>>> specification
>>> 
>>> .
>>>
>>> The gRPC specification ensures that a status (containing a gRPC status
>>> code) is only sent in Trailers, which is contained in an HTTP/2 HEADERS
>>> frame. The only way that the gRPC status code can be contained in the first
>>> HTTP/2 frame received is if the server sends a Trailers-Only response.
>>>
>>> Otherwise, the gRPC spec mandates that the first frame sent be the
>>> Response-Headers (again, sent in an HTTP/2 HEADERS frame). Response-Headers
>>> includes (optional) Custom-Metadata, which is usually what we are talking
>>> about when we say "initial metadata".
>>>
>>> Regardless of whether the Response-Headers includes anything in its
>>> Custom-Metadata, if the gRPC client library notifies the client application
>>> layer of what metadata is (or is not) included, we now have to view the RPC
>>> as committed, aka no longer retryable. This is the only option, as a later
>>> retry attempt could receive different Custom-Metadata, contradicting what
>>> we've already told the client application layer.
>>>
>>> We cannot include gRPC status codes in the Response-Headers along with
>>> "initial metadata". It's perfectly valid according to the spec for a server
>>> to send metadata along a stream in its Response-Headers, wait for one hour,
>>> then (without having sent any messages), close the stream with a retryable
>>> error.
>>>
>>> However, the proposal that a server include the gRPC status code (if
>>> known) in the initial response is still sound. Concretely, this means: if a
>>> gRPC server has not yet sent Response-Headers and receives an error
>>> response, it should send a Trailers-Only response containing the gRPC
>>> status code. This would allow retry attempts on the client-side to proceed,
>>> if applicable. This is going to be superior to sending Response-Headers
>>> immediately followed by Trailers, which would cause the RPC to become
>>> committed on the client side (if the Response-Header metadata is made
>>> available to the client application layer) and stop retry attempts.
>>>
>>> We still can encounter the case where a server intentionally sends
>>> Response-Headers to open a stream, then eventually closes the stream with
>>> an error without ever sending any messages. Such cases would not be
>>> retryable, but I think it's fair to argue that if the server *has* to send
>>> metadata in advance of sending any responses, that metadata is actually a
>>> response, and should be treated as such (i.e., their metadata just ensured
>>> the RPC will be committed on the client-side).
>>>
>>> Rather than either explicitly disallowing such behavior by modifying
>>> some specification (this behavior is currently entirely unspecified, so
>>> while specification is worthwhile, it should be separate from the retry
>>> policy design currently under discussion), we can just change the default
>>> server behavior of C++, and Go if necessary, to match Java. In Java
>>> servers, the Response-Headers are delayed until some response message is
>>> sent. If the server application returns an error status before sending a
>>> message, then Trailers-Only is sent instead of Response-Headers.
>>>
>>> We can also leave it up to the gRPC client library implementation to
>>> decide when an RPC is committed based on received Response-Headers. If and
>>> while the client library can guarantee that the presence (or absence) of
>>> initial metadata is not visible to the client application layer, the RPC
>>> can be considered uncommitted. This is an implementation detail that should
>>> very rarely be necessary if the above change is made to default server
>>> behavior, but it would not violate anything in the retry spec or semantics.
>>>
>>
>> I think that leaving this unspecified will lead to interoperability
>> problems in the future.  I would rather have the spec be explicit about
>> this, so that all future client and server implementations can interoperate
>> cleanly.
>>
>>
>
> It's fair to say in the retry design that we must count an RPC as
> committed as soon the Response-Headers arrive, and the doc now 

Re: [grpc-io] gRPC A6: Retries

['Eric Gribkoff' via grpc.io]

I think the terminology here gets confusing between initial/trailing
metadata, gRPC rule names, and HTTP/2 frame types. Our retry design doc was
indeed underspecified in regards to dealing with initial metadata, and will
be updated. I go over all of the considerations in detail below.

For clarity, I will use all caps for the names of HTTP/2 frame types, e.g.,
HEADERS frame, and use the capitalized gRPC rule names from the
specification

.

The gRPC specification ensures that a status (containing a gRPC status
code) is only sent in Trailers, which is contained in an HTTP/2 HEADERS
frame. The only way that the gRPC status code can be contained in the first
HTTP/2 frame received is if the server sends a Trailers-Only response.

Otherwise, the gRPC spec mandates that the first frame sent be the
Response-Headers (again, sent in an HTTP/2 HEADERS frame). Response-Headers
includes (optional) Custom-Metadata, which is usually what we are talking
about when we say "initial metadata".

Regardless of whether the Response-Headers includes anything in its
Custom-Metadata, if the gRPC client library notifies the client application
layer of what metadata is (or is not) included, we now have to view the RPC
as committed, aka no longer retryable. This is the only option, as a later
retry attempt could receive different Custom-Metadata, contradicting what
we've already told the client application layer.

We cannot include gRPC status codes in the Response-Headers along with
"initial metadata". It's perfectly valid according to the spec for a server
to send metadata along a stream in its Response-Headers, wait for one hour,
then (without having sent any messages), close the stream with a retryable
error.

However, the proposal that a server include the gRPC status code (if known)
in the initial response is still sound. Concretely, this means: if a gRPC
server has not yet sent Response-Headers and receives an error response, it
should send a Trailers-Only response containing the gRPC status code. This
would allow retry attempts on the client-side to proceed, if applicable.
This is going to be superior to sending Response-Headers immediately
followed by Trailers, which would cause the RPC to become committed on the
client side (if the Response-Header metadata is made available to the
client application layer) and stop retry attempts.

We still can encounter the case where a server intentionally sends
Response-Headers to open a stream, then eventually closes the stream with
an error without ever sending any messages. Such cases would not be
retryable, but I think it's fair to argue that if the server *has* to send
metadata in advance of sending any responses, that metadata is actually a
response, and should be treated as such (i.e., their metadata just ensured
the RPC will be committed on the client-side).

Rather than either explicitly disallowing such behavior by modifying some
specification (this behavior is currently entirely unspecified, so while
specification is worthwhile, it should be separate from the retry policy
design currently under discussion), we can just change the default server
behavior of C++, and Go if necessary, to match Java. In Java servers, the
Response-Headers are delayed until some response message is sent. If the
server application returns an error status before sending a message, then
Trailers-Only is sent instead of Response-Headers.

We can also leave it up to the gRPC client library implementation to decide
when an RPC is committed based on received Response-Headers. If and while
the client library can guarantee that the presence (or absence) of initial
metadata is not visible to the client application layer, the RPC can be
considered uncommitted. This is an implementation detail that should very
rarely be necessary if the above change is made to default server behavior,
but it would not violate anything in the retry spec or semantics.

Eric

On Wed, Mar 1, 2017 at 11:32 AM, 'Eric Anderson' via grpc.io <
grpc-io@googlegroups.com> wrote:

> On Wed, Mar 1, 2017 at 10:51 AM, 'Mark D. Roth' via grpc.io <
> grpc-io@googlegroups.com> wrote:
>
>> On Wed, Mar 1, 2017 at 10:20 AM, 'Eric Anderson' via grpc.io <
>> grpc-io@googlegroups.com> wrote:
>>
>>> What? That does not seem to be a proper understanding of the text, or
>>> the text is wrongly worded. Why would the RPC be "committed as soon as it
>>> receives the initial metadata"? That isn't in the text... In your example
>>> it seems it would be committed at "the trailing metadata that includes a
>>> status" as long as that status was OK, as per the "an explicit OK status"
>>> in the text.
>>>
>>
>> The language in the above quote is probably not as specific as it should
>> be, at least with respect to the wire protocol.  The intent here is that
>> the RPC should be considered committed when it receives either initial
>> metadata or a payload message.
>>

Re: [grpc-io] gRPC A6: Retries

['Mark D. Roth' via grpc.io]

On Tue, Feb 28, 2017 at 7:37 AM,  wrote:

> Right, let me chip in and continue the discussion from:
> https://github.com/grpc/proposal/pull/12#issuecomment-283063869 here. My
> comments are based on experience building a gRPC-Go interceptor for retries
> and using it in production at Improbable. It's important to note that we're
> pretty heavy users of gRPC (using it across 3 languages: Go, Java and C++),
> as we have quite a few people around (myself included) who are familiar
> with gRPC/Stubby from their prior jobs.
>
> Now, to recap the points:
>
>
> Thank you for the comments. We are trying to keep high-level discussion on
> the email thread (see here) but my responses to your points (b) and (c) are
> below.
>
> > > b) retry logic would benefit a lot from knowing whether the method is
> idempotent or not. II understand that this is supposed to be handled by
> "service configs", but realistically they're really hard to use. Few people
> would put their retry logic in DNS TXT entries, and even fewer people
> operate the gRPC LB protocol. Can we consider adding a .proto option
> (annotation) to Method definitions?
>
> > The re are two concerns here. One is that saying a method is idempotent
> really just means "retry on status codes x y and z". If we pick a canonical
> set of idempotent status codes, we are forcing every service owner to obey
> these semantics if they want to use retries. However, the gRPC status code
> mechanism is very flexible (even allowing services to return UNAVAILABLE
> arbitrarily) so we'd prefer to force service owners to consider the
> semantics of their application and pick a concrete set of status codes
> rather than just flipping an "idempotent" switch. The second concern is
> around the ease of use of the service config. The intent is for the service
> config to be a universally useful mechanism, and we want to avoid just
> baking everything into the proto. Concerns about the delivery mechanism for
> service config shouldn't invalidate its use for encoding retry policy, and
> may be something we have to tackle separately.
>
> I appreciate the push for the *service config, *and given my past SRE
> experience, I totally appreciate it. However, in the open source world
> simple solutions seem to get most traction. I would be hesitant to tie a
> very very important feature (such as retries) to the *service config*
>  adoption.
>
> I understand your concerns around the flexibility of service codes, and I
> wasn't advocating being prescriptive about it. However, I do think that
> having an option inside the .proto is a very valid approach. As a user of
> gRPC for internal and external purposes there are three cases how I use
> .protos as interfaces:
>  * have internal teams use each-other's services, in which case they code
> against code-generated interfaces that come out of .proto files
>  * to our end users provide a set of "published" .proto files and guides
> of how to generate and use them in the language of their choice through
> gRPC code generation (the true power of gRPC).
>  * to our end users provide rich client APIs for any language, in which
> case all bets are off and any thing can be implemented
>
> As such, for both external and internal services, the .proto *is* the
> canonical contract, controlled by the team building the service. Thus
> having something like the following is satisfying the most common use case:
>
> rpc RemoveTag(TagRemoveRequest) returns (google.protobuf.Empty) {
> option (grpc.extensions) = {
>   retriable_codes: ["UNAVAILABLE", "RESOURCE_EXHAUSTED"]
> };
>   };
>

There are a couple of reasons why we don't want to support configuring this
via the .proto file.

First, we don't want to require the use of protobuf in order to use gRPC.
We want people to be able to use gRPC with whatever serialization mechanism
they want (e.g., thrift).  We could potentially extend the serializer
abstraction that we currently use to provide a way to feed in this kind of
configuration information, but that would make the interface to the
serializer much more complex, which is undesirable.  And in the case of the
C-core gRPC implementation, it would require reimplementing the serializer
part in each wrapped language, rather than doing it just once in C-core.

Also, changes to the .proto file would require recompilation, which is
undesirable.  We want things like retry parameters to be something that
service owners can change at run-time without requiring action (such as
recompiling) from all of their clients.

(In a few cases, we have considered providing a way to configure something
through both the service config and the .proto file, but we've always wound
up deciding against it, because it would add a lot of complexity.  Not only
would we have to support two different code-paths for this, but we would
also need to define complex conflict resolution logic to handle the case
where we get different configuration from the two 

Re: [grpc-io] gRPC A6: Retries

['Mark D. Roth' via grpc.io]

While talking with Craig on Friday, we realized that we need to make the
wire protocol a bit stricter in order to implement retries.

Currently, the spec allows status to be sent either as part of initial
metadata or trailing metadata.  However, as per the When Retries are Valid

section
of the gRFC, an RFC becomes committed when "the client receives a non-error
response (either an explicit OK status or any response message) from the
server".  This means that in a case where the server sends a retryable
status, if the status is not included in the initial metadata, the client
will consider the RPC committed as soon as it receives the initial
metadata, even if the only thing sent after that is the trailing metadata
that includes the status.  Thus, we need to require that whenever the
server sends status without sending any messages, the server should include
the status in the initial metadata (and then close the stream without
bothering to send trailing metadata) instead of sending both initial
metadata and then trailing metadata.

Noah, can you please add a note about this to the gRFC?

Based on a previously encounted interop problem (see
https://github.com/markdroth/grpc/pull/3, which was included in
https://github.com/grpc/grpc/pull/7201), I believe that grpc-go already
does the right thing here (although Saila and Menghan should confirm
that).  However, since that previously encountered problem did not show up
with Java or C++, I suspect that those stacks do not do the right thing
here.

Craig has confirmed that C-core needs to be fixed in this regard, and I've
filed https://github.com/grpc/grpc/issues/9883 for that change.

Eric and Penn, can you confirm that Java will need to be changed?  I'm
hoping that this isn't too invasive of a change, but please let us know if
you foresee any problems.

Please let me know if anyone has any questions or problems with any of
this.  Thanks!

On Fri, Feb 10, 2017 at 4:31 PM, ncteisen via grpc.io <
grpc-io@googlegroups.com> wrote:

> I've created a gRFC describing the design and implementation plan for gRPC
> Retries.
>
> Take a look at the gRPC on Github
> .
>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/grpc-io/30e29cbc-439c-46c4-b54f-6e97637a0735%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Mark D. Roth 
Software Engineer
Google, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAJgPXp5CVHxpxYQ2DMmn-xqb9H6JGt%3Diye8mvjNxKswMXk3RUQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.