AFAIK, the transparent retries is not fully implemented. The feature you want probably is "wait_for_ready <https://grpc.io/grpc/python/grpc.html#grpc.UnaryUnaryMultiCallable>". In case of TRANSIENT_FAILURE (server not available temporarily), it will automatically wait for the channel become READY again without failing. Read more <https://github.com/grpc/proposal/blob/master/L42-python-metadata-flags.md> about wait for ready.
Lidi Zheng On Mon, Apr 15, 2019 at 11:41 AM Manatsawin Hanmongkolchai < [email protected]> wrote: > Thank you for your answer Lidi and Sidhartha, > > I've updated the code in GitHub repository to gracefully stop server. > Unfortunately, the same problem still occurs. > Previously I also have tested the same issue using Python client and a > Go server (stopping by manually ctrl+c the server) which also has the > same problem. > > In proposal A6 section "Transparent Retries" mentions that in case of > RPC failure which RPC never leaves the client, it will be retried > automatically. > I've confirmed using Wireshark that the RPC never got sent out in the > wire, and there is only 2 outgoing calls (no retry). > I'm not sure whether the proposal is implemented or I have to set a > settings somewhere? > > On Tue, Apr 16, 2019 at 1:17 AM Lidi Zheng <[email protected]> wrote: > > > > gRPC Python server can be closed by "server.stop()". > > > > Underlying, the gRPC Python server has a background polling thread. The > deallocation of the server object is not directly terminating the polling > thread since there is no way to interrupt thread in Python. So, during the > server deallocation, it will set a destruction flag, and if the polling > thread sees the flag it will terminate itself. It's possible that there are > some lag before the server completely shuts down. To eliminate this issue, > we recommend you close the server explicitly by "server.stop()" which will > block until the server is completely shut down. > > > > Lidi Zheng > > > > On Mon, Apr 15, 2019 at 11:09 AM Sidhartha Thota <[email protected]> > wrote: > >> > >> This is known issue afaik. May be your issue is related to this one: > https://github.com/grpc/grpc/issues/9767 > >> > >> If the connection is down the session will not be re-useful but channel > is always useful. Here session is individual RPC/Streaming RPC call. So, > when session is in progress and connection is down then session cannot be > restored. Channel is different and maintains connection with other end > (here server) and retries to connect periodically so it takes time to > reconnect to to server. > >> > >> > >> > >> On Mon, Apr 15, 2019 at 8:49 AM <[email protected]> wrote: > >>> > >>> Steps to reproduce > >>> > >>> 1. Start server > >>> 2. Send a client RPC to server > >>> 3. Restart server > >>> 4. Using the same client, send another RPC. The call will fail > >>> 5. Send another RPC, this call will success > >>> > >>> Also I found that if the server is leave stopped for a long time > before starting up again, the call in step 5 will return "channel is in > state TRANSIENT_FAILURE" as well. > >>> > >>> Example code: https://github.com/whs/grpc-repro > >>> (Install from requirements.txt then run main.py) > >>> > >>> Expected result > >>> > >>> All call should success. > >>> > >>> Tested with Python grpcio==1.19.0 server/client and with go-grpc > server. I tried setting grpc.max_connection_age_grace_ms, > grpc.max_connection_age_ms, grpc.max_connection_idle_ms, > grpc.keepalive_time_ms, grpc.keepalive_permit_without_calls but they > doesn't seems to help. > >>> > >>> From my own investigation, I believe that gRPC channels doesn't drop > closed channels and tries to send calls in closed channel. There is also > TCP RST sent. > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups "grpc.io" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > >>> To post to this group, send email to [email protected]. > >>> Visit this group at https://groups.google.com/group/grpc-io. > >>> To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/e8465f2b-96f1-4ad1-816f-ee6521595179%40googlegroups.com > . > >>> For more options, visit https://groups.google.com/d/optout. > >> > >> > >> > >> -- > >> Thanks, > >> Sidhartha Thota. > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups "grpc.io" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > >> To post to this group, send email to [email protected]. > >> Visit this group at https://groups.google.com/group/grpc-io. > >> To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/CALguN-Gg0iiDSEUrO-VEOuGu5Mhemc9bZVqxqi_BcY7-pc%3DQWg%40mail.gmail.com > . > >> For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAMC1%3Djc99RwzkcgiScz84mSpxYiSMk2GGqx%3D%2BiQ-vNOJL9rVHA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
