Hmm ... It's unfortunate that there is no way to force the C++ client to periodically refresh it's list of IP addresses. That's a show stopper as our backends scale up elastically and there is no way for gRPC client to become aware of them.
Q1: If we implement our own lookaside LB, could we configure the client to consult this LB for a fresh set of IP addresses periodically ? Q2: Can the lookaside LB be within the client process ? On Thu, Jan 17, 2019, 11:00 PM apolcyn via grpc.io <[email protected] wrote: > I can add to a couple of questions. > Re: > "Do gRPC clients honor DNS TTL ?" > > gRPC clients don't look at TTL's at all. In C++, a gRPC client channel > will request it's DNS resolver to re-resolve when it determines that it has > reached "transient failure" state. The details of when exactly it reaches > that state depends on the load balancing policy in use. In "round robin", > it would be roughly when all individual connections in the list reach > "transient failure", i.e. if the connections all break. Effectively, if > backends are moving around and things break, then the default client will > re-resolve. But if you want the DNS resolution to be up to date for > different reasons, there's no polling built in to the default DNS resolver. > > This could be done with a custom resolver, but in C++ the resolver API > isn't currently public. I understand that making that API public is > something in progress though. > > Re: > "Q2: Is it possible for gRPC to resolve DNS via TCP instead of UDP ?" > > If the DNS server sends back a large response (c-ares considers a large > response greater than 512 bytes), or if the response has a "truncated" bit > set, then it will re-send its query over TCP. I can confirm this with > "ares", I believe this is also the case with "native" (in C++ there's two > DNS resolvers: "ares" (c-ares) and "native" (getaddrinfo); "native" is the > default one right now, but "ares" should be the default in the upcoming > 1.19 release) > > > > > > > On Thursday, January 17, 2019 at 6:23:37 PM UTC-8, Carl Mastrangelo wrote: >> >> I know you asked for C++, but At least for Java we do not honor TTL. >> (because the JVM doesn't surface it to us). If you implement your own >> NameResolver (not as hard as it sounds!) you can honor these TTLs. >> >> I believe C++ uses the cares resolver which IIRC can resort to doing TCP >> lookups if the response size is too large. Alas, I cannot answer with any >> more detail. >> >> gRPC has the option to do health checks, but what I think you actually >> want are keep-alives. This is configurable on the channel and the server. >> If you can add more detail about the problem you are trying to avoid, I can >> give a better answer. >> >> As for if DNS is a really bad idea: Not really. It has issues, but none >> of them are particularly damning. For example, when you add a new server >> most clients won't find out about it until they poll again. gRPC is >> designed around a push based name resolution model, with clients being told >> what servers they can talk to. DNS is adapted onto this model, by >> periodically spawning a thread and notifying the client via the >> push-interface. >> >> The DNS support is pretty good in gRPC, to the point that implementing a >> custom DNS resolver is likely to cause more issues (what happens if the A >> lookups succeed, but the AAAA fail?, what happens if there are lots of >> addresses for a single endpoint?, etc.) >> >> One last thing to consider: the loadbalancer in gRPC is independent of >> the name resolver. You could continue to use DNS (and do SRV lookups and >> such) and pass that info into your own custom client-side LB. This is what >> gRPCLB does, but you could customize your own copy to not depend on a >> gRPCLB server. There's lots of options here. >> >> >> >> On Wednesday, January 16, 2019 at 5:01:33 PM UTC-8, Ram Kumar Rengaswamy >> wrote: >>> >>> Hello ... We are looking to setup client-side loadbalancing in GRPC >>> (C++). >>> Our current plan roughly is the following: >>> 1. Use consul for service discovery, health checks etc. >>> 2. Expose the IP addresses behind a service to GRPC client via Consul >>> DNS Interface <https://www.consul.io/docs/agent/dns.html> >>> 3. Configure the client to use simple round_robin loadbalancing (All our >>> servers have the same capacity and therefore we don't need any >>> sophisticated load balancing) >>> >>> Before we embark on this path, it would be great if someone with gRPC >>> production experience could answer a few questions. >>> Q1: We plan to use a low DNS TTL (say 30s) to force the clients to have >>> the most up to date service discovery information. Do gRPC clients honor >>> DNS TTL ? >>> Q2: Is it possible for gRPC to resolve DNS via TCP instead of UDP ? We >>> could have a couple of hundred backends for a service. >>> Q3: Does gRPC do its own health checks and mark unhealthy connections? >>> >>> Also from experience, do folks think that this is a really bad idea and >>> we should really use grpclb policy and implement a look-aside >>> loadbalancer instead ? >>> >>> Thanks, >>> -Ram >>> >> -- > You received this message because you are subscribed to the Google Groups " > grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/grpc-io. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/f8a69f99-5c08-4f33-8299-3f4922aed084%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/f8a69f99-5c08-4f33-8299-3f4922aed084%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CA%2Bv4T%2BJ-n8Hy4hROEV%2B2p5ohSmkToPCjYhLbvcGbMg1JT6i-Rw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
