Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19205 )
Change subject: IMPALA-13294: Add support for long polling to avoid client side wait ...................................................................... IMPALA-13294: Add support for long polling to avoid client side wait Currently, Impala does an execute call, then the client polls waiting for the operation to finish (or error out). The client sleeps between polls, and this sleep time can be a substantial percentage of a short query's execution time. To reduce this client side sleep, this implements long polling to provide an option to wait for query completion on the server side. This is controlled by the long_polling_time_ms query option. If set to greater than zero, status RPCs will wait for query completion for up to that amount of time. This defaults to off (0ms). Both Beeswax and HS2 add a wait for query completion in their get status calls (get_state for Beeswax, GetOperationStatus for HS2). This doesn't wait in the execute RPC calls (e.g. query for Beeswax, ExecuteStatement for HS2), because neither includes the query status in the response. The client will always need to do a separate status RPC. This modifies impala-shell and the beeswax client to avoid doing a sleep if the get_state/GetOperationStatus calls take longer than they would have slept. In other words, if they would have slept 50ms, then they skip that sleep if the RPC to the server took longer than 50ms. This allows the client to maintain its sleep behavior with older Impalas that don't use long polling while adapting properly to systems that do have long polling. This has the added benefit that it also adjusts for high latency to the server as well. This does not change any of the sleep times. Testing: - This adds a test case in test_hs2.py to verify the long polling behavior Change-Id: I72ca595c5dd8a33b936f078f7f7faa5b3f0f337d Reviewed-on: http://gerrit.cloudera.org:8080/19205 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M shell/impala_client.py M tests/beeswax/impala_beeswax.py M tests/hs2/test_hs2.py 12 files changed, 179 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19205 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I72ca595c5dd8a33b936f078f7f7faa5b3f0f337d Gerrit-Change-Number: 19205 Gerrit-PatchSet: 13 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
