Thanks for pointer to the discussion. The MaxConnectionAge feature won't work for us as we do want to preserve very long lived connections. In future, if Custom Resolver API in C+ becomes public, then that would be the way to go. For now, we are going to implement a lookaside load balancer that proxies the DNS lookup and switch to grpclb scheme in the C++ client.
On Fri, Jan 18, 2019 at 2:45 PM Mark D. Roth <[email protected]> wrote: > The issue of forcing DNS to periodically re-resolve is being discussed in > https://github.com/grpc/grpc/issues/12295, but there's no consensus yet > that we actually want to implement that. That having been said, that issue > does discuss a work-around that people have used successfully, which is to > use the MaxConnectionAge feature on the server side to periodically force > clients to reconnect. Whenever that happens, the client will re-resolve. > > I think that in your situation, I would use the above workaround with the > existing DNS resolver and the existing round_robin LB policy. I don't > think you need to write any custom plugins. > > On Thu, Jan 17, 2019 at 11:30 PM Ram Kumar Rengaswamy <[email protected]> > wrote: > >> Hmm ... It's unfortunate that there is no way to force the C++ client to >> periodically refresh it's list of IP addresses. That's a show stopper as >> our backends scale up elastically and there is no way for gRPC client to >> become aware of them. >> >> Q1: If we implement our own lookaside LB, could we configure the client >> to consult this LB for a fresh set of IP addresses periodically ? >> Q2: Can the lookaside LB be within the client process ? >> >> >> On Thu, Jan 17, 2019, 11:00 PM apolcyn via grpc.io < >> [email protected] wrote: >> >>> I can add to a couple of questions. >>> Re: > "Do gRPC clients honor DNS TTL ?" >>> >>> gRPC clients don't look at TTL's at all. In C++, a gRPC client channel >>> will request it's DNS resolver to re-resolve when it determines that it has >>> reached "transient failure" state. The details of when exactly it reaches >>> that state depends on the load balancing policy in use. In "round robin", >>> it would be roughly when all individual connections in the list reach >>> "transient failure", i.e. if the connections all break. Effectively, if >>> backends are moving around and things break, then the default client will >>> re-resolve. But if you want the DNS resolution to be up to date for >>> different reasons, there's no polling built in to the default DNS resolver. >>> >>> This could be done with a custom resolver, but in C++ the resolver API >>> isn't currently public. I understand that making that API public is >>> something in progress though. >>> >>> Re: > "Q2: Is it possible for gRPC to resolve DNS via TCP instead of UDP >>> ?" >>> >>> If the DNS server sends back a large response (c-ares considers a large >>> response greater than 512 bytes), or if the response has a "truncated" bit >>> set, then it will re-send its query over TCP. I can confirm this with >>> "ares", I believe this is also the case with "native" (in C++ there's two >>> DNS resolvers: "ares" (c-ares) and "native" (getaddrinfo); "native" is the >>> default one right now, but "ares" should be the default in the upcoming >>> 1.19 release) >>> >>> >>> >>> >>> >>> >>> On Thursday, January 17, 2019 at 6:23:37 PM UTC-8, Carl Mastrangelo >>> wrote: >>>> >>>> I know you asked for C++, but At least for Java we do not honor TTL. >>>> (because the JVM doesn't surface it to us). If you implement your own >>>> NameResolver (not as hard as it sounds!) you can honor these TTLs. >>>> >>>> I believe C++ uses the cares resolver which IIRC can resort to doing >>>> TCP lookups if the response size is too large. Alas, I cannot answer with >>>> any more detail. >>>> >>>> gRPC has the option to do health checks, but what I think you actually >>>> want are keep-alives. This is configurable on the channel and the server. >>>> If you can add more detail about the problem you are trying to avoid, I can >>>> give a better answer. >>>> >>>> As for if DNS is a really bad idea: Not really. It has issues, but >>>> none of them are particularly damning. For example, when you add a new >>>> server most clients won't find out about it until they poll again. gRPC is >>>> designed around a push based name resolution model, with clients being told >>>> what servers they can talk to. DNS is adapted onto this model, by >>>> periodically spawning a thread and notifying the client via the >>>> push-interface. >>>> >>>> The DNS support is pretty good in gRPC, to the point that implementing >>>> a custom DNS resolver is likely to cause more issues (what happens if the A >>>> lookups succeed, but the AAAA fail?, what happens if there are lots of >>>> addresses for a single endpoint?, etc.) >>>> >>>> One last thing to consider: the loadbalancer in gRPC is independent of >>>> the name resolver. You could continue to use DNS (and do SRV lookups and >>>> such) and pass that info into your own custom client-side LB. This is what >>>> gRPCLB does, but you could customize your own copy to not depend on a >>>> gRPCLB server. There's lots of options here. >>>> >>>> >>>> >>>> On Wednesday, January 16, 2019 at 5:01:33 PM UTC-8, Ram Kumar >>>> Rengaswamy wrote: >>>>> >>>>> Hello ... We are looking to setup client-side loadbalancing in GRPC >>>>> (C++). >>>>> Our current plan roughly is the following: >>>>> 1. Use consul for service discovery, health checks etc. >>>>> 2. Expose the IP addresses behind a service to GRPC client via Consul >>>>> DNS Interface <https://www.consul.io/docs/agent/dns.html> >>>>> 3. Configure the client to use simple round_robin loadbalancing (All >>>>> our servers have the same capacity and therefore we don't need any >>>>> sophisticated load balancing) >>>>> >>>>> Before we embark on this path, it would be great if someone with gRPC >>>>> production experience could answer a few questions. >>>>> Q1: We plan to use a low DNS TTL (say 30s) to force the clients to >>>>> have the most up to date service discovery information. Do gRPC clients >>>>> honor DNS TTL ? >>>>> Q2: Is it possible for gRPC to resolve DNS via TCP instead of UDP ? We >>>>> could have a couple of hundred backends for a service. >>>>> Q3: Does gRPC do its own health checks and mark unhealthy connections? >>>>> >>>>> Also from experience, do folks think that this is a really bad idea >>>>> and we should really use grpclb policy and implement a look-aside >>>>> loadbalancer instead ? >>>>> >>>>> Thanks, >>>>> -Ram >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "grpc.io" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/grpc-io. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/grpc-io/f8a69f99-5c08-4f33-8299-3f4922aed084%40googlegroups.com >>> <https://groups.google.com/d/msgid/grpc-io/f8a69f99-5c08-4f33-8299-3f4922aed084%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "grpc.io" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/grpc-io. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/grpc-io/CA%2Bv4T%2BJ-n8Hy4hROEV%2B2p5ohSmkToPCjYhLbvcGbMg1JT6i-Rw%40mail.gmail.com >> <https://groups.google.com/d/msgid/grpc-io/CA%2Bv4T%2BJ-n8Hy4hROEV%2B2p5ohSmkToPCjYhLbvcGbMg1JT6i-Rw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > Mark D. Roth <[email protected]> > Software Engineer > Google, Inc. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CA%2Bv4T%2BJNq2XB6keGRMusjadYb-WqVG17DsBBtByX2Phqs08vYg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
