Thanks for reporting this! I think I see the problem, and it should be fixed by https://github.com/grpc/grpc/pull/29144.
I don't see a trivial way to write a test for it, since the current behavior is not broken, just slightly sub-optimal. But feel free to try out the patch and let me know if it fixes the problem. On Thu, Mar 17, 2022 at 1:09 PM František Bořánek <[email protected]> wrote: > It's not exactly the same version. This one is gRPC 1.43.2, but behaviour > do not vary form 1.44.0. > I attached also initial start up, in case you want to compare. > Reconnection of xds contains file goaway.from.xds.cpp.grpc.log. > The lifetime of channels are until shutdown of the application. > > > > 17. 3. 2022 v 16:52, Mark D. Roth <[email protected]>: > > That's very interesting. Can you run with the following environment > variables and send me the log? > > GRPC_VERBOSITY=DEBUG GRPC_TRACE=xds_client,xds_resolver > > On Wed, Mar 16, 2022 at 10:53 PM František Bořánek <[email protected]> > wrote: > >> C++, the latest 1.44.0 release. >> >> 16. 3. 2022 v 21:32, Mark D. Roth <[email protected]>: >> >> >> It is valid behavior, but I agree that it's a little sub-optimal for the >> client to send two messages when it reestablishes the stream. What >> language are you using gRPC in? >> >> On Wed, Mar 16, 2022 at 1:19 PM František Bořánek <[email protected]> >> wrote: >> >>> It makes sense. >>> >>> However, in this case, the initial connection was already made and this >>> tcpdump samples recovering of ADS stream. And as you write: >>> >>> > Note that once your application receives all 7 resources from the xDS >>> server, if the ADS stream is broken and the client reestablishes it at that >>> point, it should immediately subscribe to all 7 resources in the initial >>> message on the new stream. >>> >>> It behaves as I described. First 1 resource and then 7 resources. Note >>> that the messages have version_info. Brand new channels don't have >>> version_info. >>> >>> But as you pointed, the channels can be created anytime, thus it's valid >>> bahaviur. >>> >>> Thanks for replying. >>> >>> F. >>> >>> 16. 3. 2022 v 19:49, Mark D. Roth <[email protected]>: >>> >>> The gRPC client does not know a priori what set of resource names are >>> available on the xDS server, and even if it did, it would not request all >>> of them proactively, because it may not actually need all of them. >>> Instead, each time a gRPC channel is created with an "xds:" target URI, >>> that tells gRPC which LDS resource to request, and it requests the >>> resources as they are needed. Because the application can create new >>> channels at any time, the set of resources requested on the ADS stream can >>> change at any time. >>> >>> If your application immediately creates 7 channels when it starts up, >>> then the pattern you're seeing is expected. When the first channel >>> (presumably for "xds:server-1") starts connecting, it will immediately >>> subscribe to the corresponding LDS resource ("server-1"). While it's >>> waiting for that message to be sent, it notices that the other 6 channels >>> have started up, so it realizes that it has to subscribe to the other 6 >>> resources. By the time the initial message is sent and it's able to send a >>> second message, it knows about all 7 subscriptions, so the second message >>> contains all 7 of them. >>> >>> Note that once your application receives all 7 resources from the xDS >>> server, if the ADS stream is broken and the client reestablishes it at that >>> point, it should immediately subscribe to all 7 resources in the initial >>> message on the new stream. That is because it already knows about all of >>> the subscriptions at the point at which it sends the initial message on the >>> new stream. >>> >>> This communication is fully legal and should work fine with any >>> xDS-compliant server. As described in this section of the xDS spec >>> <https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#ack-nack-and-resource-type-instance-version>, >>> the intent of the version field in the DiscoveryRequest is to provide the >>> server with a way to know when it sends a version that the client considers >>> invalid. But if the client sends a second DiscoveryRequest with the same >>> version and a different list of resource names, that indicates that it is >>> changing what resources it is subscribing to, and the server cannot ignore >>> that just because the version is the same as one it has previously seen. >>> >>> I hope this information is helpful. >>> >>> On Mon, Mar 14, 2022 at 5:41 AM František Bořánek <[email protected]> >>> wrote: >>> >>>> Hi, >>>> I don't know if such behaviour is a bug. >>>> >>>> I have server communicates with 7 services configured via XDS and after >>>> the xds-client got GO AWAY packet from XDS server 6 channels cannot connect >>>> due to error and only one is ok. I did a tcpdump to see a chat between our >>>> implementation of XDS and C++gRPC and cannot deside if C++gRPC behave >>>> correct. >>>> >>>> It relates to >>>> https://github.com/grpc/proposal/blob/master/A27-xds-global-load-balancing.md#initial-request-on-the-ads-stream >>>> > The first request on the ADS stream will include a Node message >>>> that identifies the client. As mentioned above, most of the fields for the >>>> Node message will be read from the bootstrap file. >>>> >>>> I would expect that after reconnect the xds client sends to ADS stream >>>> 4 requests (LDS, CDS, EDS, RDS) however it sends 5 requests (LDS, LDS, CDS, >>>> EDS, RDS). On our side of xds server we are confused by these 2 LDS >>>> requests. >>>> >>>> 1. >>>> (PROTOBUF) envoy.api.v2.DiscoveryRequest >>>> { >>>> version_info: "ba0255a2-07b6-45ed-83b4-dad0cbf96b1f" >>>> node: { >>>> cluster: "default-cluster" >>>> user_agent_name: "gRPC C-core linux" >>>> user_agent_version: "C-core 22.0.0" >>>> client_features: "envoy.lb.does_not_support_overprovisioning" >>>> build_version: "gRPC C-core linux 22.0.0" >>>> } >>>> resource_names: "server-1" >>>> type_url: "type.googleapis.com/envoy.api.v2.Listener" >>>> } >>>> >>>> 2. >>>> (PROTOBUF) envoy.api.v2.DiscoveryRequest >>>> { >>>> version_info: "ba0255a2-07b6-45ed-83b4-dad0cbf96b1f" >>>> resource_names: "server-1" >>>> resource_names: "server-2" >>>> resource_names: "server-3" >>>> resource_names: "server-4" >>>> resource_names: "server-5" >>>> resource_names: "server-6" >>>> resource_names: "server-7" >>>> type_url: "type.googleapis.com/envoy.api.v2.Listener" >>>> } >>>> >>>> - Shouldn't it be only one LDS request? >>>> - Shouldn't it the initial request contains all configured resources? >>>> >>>> Root of our problem is that we responds only on first request since it >>>> have the same version. I know how to fix it on our end, but it is behaviour >>>> of gRPC xds-client correct? >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "grpc.io" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/grpc-io/e2a5befd-5943-4b40-afc4-4864938b9b88n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/grpc-io/e2a5befd-5943-4b40-afc4-4864938b9b88n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Mark D. Roth <[email protected]> >>> Software Engineer >>> Google, Inc. >>> >>> >>> >> >> -- >> Mark D. Roth <[email protected]> >> Software Engineer >> Google, Inc. >> >> > > -- > Mark D. Roth <[email protected]> > Software Engineer > Google, Inc. > > > -- Mark D. Roth <[email protected]> Software Engineer Google, Inc. -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAJgPXp5Wi5VjHYkB1%3DywXaNt5gy%3DsYYAsVgfb7AZNTFwGDVwUg%40mail.gmail.com.
