>From your description it looks like you are destroying all the old servers and bringing up a completely new set of servers with new IPs. The gRPC client is still seeing old IPs in the cached DNS, none of which are available. It will try to connect to these unavailable IPs until the deadline is reached.
On Thursday, January 24, 2019 at 10:30:02 AM UTC-8, [email protected] wrote: > > Sure, I understand that part. But what I didn't understand was why I > continued to get DEADLINE_EXCEEDED errors after the UNAVAILABLE errors? I > would have though that since the old instances were terminated, I wouldn't > be able to even maintain a connection to the old instances, so > theoretically I would keep trying to reconnect and eventually open up a > connection to the new instance. > > On Wednesday, January 23, 2019 at 11:29:35 AM UTC-8, Srini Polavarapu > wrote: >> >> That's right. When a subchannel goes down, the channel re-resolves DNS in >> round-robin LB. Depending on your DNS TTL, the OS may still be returning >> cached DNS entry which might still contain the server IP that went down. >> Regardless, depending on DNS updates will not fully meet your LB >> requirements. This is because gRPC client does not periodically re-resolve >> DNS. This means when new backends are added, gRPC client will not know >> about those. See this <https://github.com/grpc/grpc/issues/14492> and >> this <https://groups.google.com/forum/#!topic/grpc-io/2a5zsRNBXLI>. >> >> On Sunday, January 20, 2019 at 9:27:49 PM UTC-8, [email protected] >> wrote: >>> >>> I'm trying to set up a python gRPC simple client and server, where the >>> client uses round robin load balancing against a single DNS record, where I >>> have multiple servers (instances) in the DNS record. >>> >>> In the beginning, I'm able to connect and issue queries fine, but when I >>> try a re-deploy of my servers, I get some weird behavior that I was hoping >>> would be resolved automatically by the client library. In my re-deploy, I >>> first bring up new servers, set the DNS record to the IPs of the new >>> servers, and then destroy the old servers. Everything seems to work until I >>> destroy the old servers, at which point I get a couple of >>> UNAVAILABLE_ERRORs followed by DEADLINE_EXCEEDED until I kill the client. >>> From what I understand, when the sub-channels go down (i.e. the server >>> instances are killed), the channel should re-resolve the DNS record and >>> attempt to re-connect to the new instances. Am I interpreting this >>> incorrectly? Is there some channel and/or server option I need to set in >>> order for this to work? >>> >>> Sample client below: >>> >>> channel = grpc.insecure_channel("localhost:10000", >>> options=(("grpc.lb_policy_name", "round_robin"),)) >>> fut = grpc.channel_ready_future(channel) >>> fut.result() >>> print("done waiting") >>> stub = test_pb2_grpc.TestStub(channel) >>> while True: >>> try: >>> print(stub.Test(request, timeout=5)) >>> except grpc.RpcError as e: >>> print("{} {}".format(time.time(), e)) >>> time.sleep(0.5) >>> >>> -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/29ca432a-8b67-44a6-a000-2d844ad4de21%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
