Qifan Chen has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/17872 )
Change subject: IMPALA-10811 RPC to submit query getting stuck for AWS NLB forever ...................................................................... IMPALA-10811 RPC to submit query getting stuck for AWS NLB forever This patch addresses Impala client hang due to AWS network load balancer timeout which is fixed at 350s. After the timeout due to some long query compilation for example, the connection is silently dropped and the Impala client enters the hang state. The scope of the fix applies to the following Impala clients which issue thrift RPC ExecuteStatement() followed by repeated call to GetOperationStatus() (HS2, Impyla and HUE) or a variant of it (Beeswax) to Impala backend. 1. HS2 2. Beeswax 3. Impyla 4. HUE The TCLIService protocol between the client and Impala server remains unchanged. In the fix, the backend method ImpalaServer::ExecuteStatement() starts a new thread for ImpalaServer::ExecuteStatementCommon() which can reach two states: COMPILED and DONE. The COMPILED is when the query has been successfully compiled and the DONE is for the execution to reach the end successfully or to encounter any errors. The main thread, which start the new thread, waits for the COMPILED state before advancing to another short wait period for the DONE state. If the DONE state is not reached, the control is returned back to the client and the client will issue GetOperationStatus() repeatedly to check if the execution has reached the DONE state. When Impala server detects the FINISHED execution state or there is error in servicing GetOperationStatus(), the new thread is joined and released. In addition, a cild query, which is submitted from the Impala server as an Impala client for compute stats stmt, runs synchronously in the same child query thread. Testing: TBD Change-Id: Ib57e86926a233ef13d27a9ec8d9c36d33a88a44e --- M be/src/service/child-query.cc M be/src/service/client-request-state.cc M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h 6 files changed, 251 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/17872/9 -- To view, visit http://gerrit.cloudera.org:8080/17872 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib57e86926a233ef13d27a9ec8d9c36d33a88a44e Gerrit-Change-Number: 17872 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Qifan Chen <[email protected]>
