Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19205
to look at the new patch set (#5).
Change subject: Long polling to avoid client side wait
......................................................................
Long polling to avoid client side wait
Currently, Impala does an execute call, then the client polls
waiting for the operation to finish (or error out). The client
sleeps between polls, and this sleep time can be a substantial
percentage of a short query's execution time.
To reduce this client side sleep, this implement long polling to
provide an option to wait for query completion on the server side.
This is controlled by the long_polling_time_ms query option. If
set to greater than zero, certain RPCs will wait for query
completion for up to that amount of time. This defaults to 100ms.
TODO: Decide if 100ms is reasonable.
Both Beeswax and HS2 add a wait for query completion in their
get status calls (get_state for Beeswax, GetOperationStatus for HS2).
For HS2, this also adds a wait for query completion for the
ExecStatement call. If the the query completes in this time, the
client can avoid a round-trip for the GetOperationStatus call.
There is no equivalent for Beeswax, because the "query" RPC doesn't
return query status. There is no way to avoid the extra round trip
of a get_state call.
This modifies impala-shell and the beeswax client to avoid doing a
sleep if the get_state/GetOperationStatus calls take longer than
they would have slept. In other words, if we would have slept 50ms,
then we skip that sleep if the RPC to the server took longer than
50ms. This allows the client to maintain its sleep behavior with
older Impalas that don't use long polling while adapting properly
to systems that do have long polling. This has the added benefit
that it also adjusts for high latency to the server as well.
This also modifies the function used to decide on the sleep time
between polls. Rather than using a step function with fixed
sleep lengths for different elapsed times, this switches to an
equation that smoothly increases the sleep time.
TODO: Decide if this is a sensible change to the sleep time.
Change-Id: I72ca595c5dd8a33b936f078f7f7faa5b3f0f337d
---
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M shell/impala_client.py
M tests/beeswax/impala_beeswax.py
10 files changed, 98 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/19205/5
--
To view, visit http://gerrit.cloudera.org:8080/19205
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I72ca595c5dd8a33b936f078f7f7faa5b3f0f337d
Gerrit-Change-Number: 19205
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
