Hello grpc community, We are trying to figure out load balancing across weighted endpoints behind a primary domain from client-side. We use grpc-java 1.3
The context: - We have a single long running client which establishes a GRPC channel to a cluster of GRPC Services behind a single L7 Application-level Load Balancer which load balances round-robin style - Our client is high traffic streaming calls to the services behind each load balancer - A single load balancer and cluster can be considered a single single unit. We want to spin up more units, and map each load balancer behind a primary domain name, which the client would then call. - We have configured a weighted routing policy for this primary DNS, with a weight assigned to each load balancer Example: 1. We have Loadbalancer A, Loadbalancer B, and DNS P 2. We associate A and B behind P, each with a weight of .5 and .5 respectively 3. If we query DNS P 10,000 times, then it resolves to LB A ~5000 times, and LB B ~5000 times The issue comes up though when actually sending requests through the client, which establishes a long running grpc channel with the primary DNS. Found that the behavior is not what we expected: - The amount of traffic actually routing to each cluster is not proportional to the weights we assigned to them. - The client establishes a connection to a single cluster, then it doesn't change and all requests are sent to that single cluster rather than being balanced across multiple clusters accordingly This seems like it already has a solution if we configure MAX_CONNECTION_AGE on the server side, but for certain reasons we cannot modify this property from the server side, and so want to investigate if there are possible solutions solely from a client side. - https://github.com/grpc/proposal/blob/master/A9-server-side-conn-mgt.md Some things we have found or are planning to look into: 1. defaultLoadBalancingPolicy("round_robin") - https://www.javadoc.io/doc/io.grpc/grpc-all/1.30.1/io/grpc/ForwardingChannelBuilder.html#defaultLoadBalancingPolicy-java.lang.String- - This initially seemed promising, but unsure of its feasibility since our DNS will only resolve to a single LoadBalancer as described above, rather than provide the entire address list of all units 2. Weighted round robin - https://github.com/grpc/proposal/pull/343/files - Our weighted routing policy is configured externally to the grpc service, server or client side, so this doesn't seem exactly usable for our scenario, although its naming sounds very similar to what we are trying to do. - This seems like a newer change, and it is infeasible for us to upgrade to a newer version of grpc-java for now 3. Recreating the channel on some interval - This can be ruled out since it goes against grpc best practices and creating channels so often is expensive Any ideas or suggestions would be greatly appreciated, thanks -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/579e38a6-6227-4034-8c72-0266106b4ff8n%40googlegroups.com.
