At least in Java, Go, and C, the DNS resolver refreshes when an underlying transport (the TCP connection) is closed. The transport can be closed for a variety of reasons. Most likely in your scenario, the traffic volume has an influence on whether connections are being closed on the AWS ELB side, which triggers DNS re-resolution in the high-volume case.
One way to force periodic DNS re-resolution from the client side is to set the maximum connection age. In C it's done via the channel options GRPC_ARG_MAX_CONNECTION_AGE_MS and GRPC_ARG_MAX_CONNECTION_AGE_GRACE_MS documented here: https://grpc.github.io/grpc/core/group__grpc__arg__keys.html#gabd3a16f46ad2cb5f06064bb607df7b5b https://grpc.github.io/grpc/core/group__grpc__arg__keys.html#gaf4574abe94c339c6f21163bca6e7b6b7 There are equivalents for other languages. This will cause a bit of connection churn, so you probably don't want to set the maximum age too low. Another option would be to implement regular connection closing on the server side, on your AWS ELB configuration -- not sure if AWS ELBs provide that as an option. Hope this helps, Antoine. On Tue, Apr 18, 2023 at 7:23 PM Mark Robinson <[email protected]> wrote: > Hi, > > I have a problem where I have two services communicating but won't update > the endpoint connection information when DNS updates. > > Very briefly, the architecture is > > http -> [AWS LB] -> [Service A] -> (grpc) [AWS ELB] -> [Service B] > > When I change the DNS information for service B, with the intent of > sending it through a physically different ELB, service A won't change to > point to the new IP address at low traffic volume. > > If I increase the traffic volume, it will switch fairly quickly. On the > order of minutes. However, if I keep the traffic volume low (<2 RPS/pod), > it'll stick on the old connection for hours if not forever. The only > solution I've found is to restart all of service A, but that's not a great > solution. > > Does anyone know what might be going on here? > > Mark > > -- > You received this message because you are subscribed to the Google Groups " > grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/1042f275-b01f-49d3-b39c-4edcfac34a84n%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/1042f275-b01f-49d3-b39c-4edcfac34a84n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CADrS3jK8__0RBvzrZgG%2BG6DKA%3D3sHyvytsj_9qTLQmhaYoPhjQ%40mail.gmail.com.
