Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5388: Only retry RPC on lost connection in send call ......................................................................
IMPALA-5388: Only retry RPC on lost connection in send call Previously, DoRpc() blacklists only a couple of conditions which shouldn't retry the RPC on exception. This is fragile as the errors could have happened after the payload has been successfully sent to the destination. Such aggressive retry behavior can lead to duplicated row batches being sent, causing wrong results in queries. This change fixes the problem by whitelisting the conditions in which the RPC can be retried. Specifically, it pattern-matches against certain errors in TSocket::write_partial() in the thrift library and only retries the RPC in those cases. With SSL enabled, we will never retry. We should investigate whether there are some cases in which it's safe to retry. This change also adds fault injection in the TransmitData() RPC caller's path to emulate different exception cases. Change-Id: I176975f2aa521d5be8a40de51067b1497923d09b Reviewed-on: http://gerrit.cloudera.org:8080/7063 Reviewed-by: Michael Ho <[email protected]> Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/runtime/backend-client.h M be/src/runtime/client-cache.h M be/src/testutil/CMakeLists.txt A be/src/testutil/fault-injection-util.cc M be/src/testutil/fault-injection-util.h A tests/custom_cluster/test_rpc_exception.py 9 files changed, 300 insertions(+), 54 deletions(-) Approvals: Impala Public Jenkins: Verified Michael Ho: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/7063 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I176975f2aa521d5be8a40de51067b1497923d09b Gerrit-PatchSet: 12 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Michael Ho <[email protected]> Gerrit-Reviewer: Alan Choi <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Juan Yu <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
