Turns out this *was* a problem with the google-cloud-endpoint/NGINX container being terminated before the gRPC server container in the same pod had a chance to close it's connections gracefully.
If anyone is remotely interested in knowing more, there's conversation I had with myself over here: https://groups.google.com/forum/#!topic/google-cloud-endpoints/FyfdvD6xS1Q On Thursday, 25 January 2018 15:47:04 UTC+1, MNS wrote: > > I have a unary streaming service from a Python server (1.8.4) to C# client > (1.8.3). > > When I signal the Python to shutdown (TERM15) the shutdown method in the > code listing below is called, the intention of which is to terminate the > rRPCs gracefully, and shut the server down. > > This works when I'm running a server on localhost, an RpcException with > the expected status is raised: `Status(StatusCode=Cancelled, > Detail="Completed")` > > However, when running the server on GKE and terminating the pod I receive: > `Grpc.Core.RpcException: Status(StatusCode=Unknown, Detail="Stream > removed")` in the client. The python server is behind an haproxy ingress > controller and google-cloud-endpoint proxy, but in my understanding neither > of these components should affect the connection. > > Can anybody think what could be causing the different statuses in the > RpcException on the client? > > Thanks, Mark > > def MyHandler(self, request, context): > while not stop_event.isSet(): > try: > yield update_q.get(True, 0.1) > except queue.Empty: > continue > > > context.set_code(grpc.StatusCode.CANCELLED) > context.set_details("Completed") > > > > > def shutdown(subscriber_service: StreamsService, > executor: futures.ThreadPoolExecutor, > server: grpc.Server, exit_code): > > > logger.info("Stopping stream handlers") > for stop_event in subscriber_service.stop_events: > stop_event.set() > > > logger.info("Stopping executor") > executor.shutdown() > > logger.info("Stopping server") > ev: threading.Event = server.stop(grace=10) # allows RPCs to > terminate gracefully > > > logger.info("Waiting for server to stop gracefully") > ev.wait() > > > logger.info("Stopping process with exit code {}".format(exit_code)) > sys.exit(exit_code) > > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/20966ebb-bde8-469d-8fc5-c8c02606f3e1%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
