Hello Quanlong Huang, Andrew Sherman, Jason Fehr, Joe McDonnell, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21803
to look at the new patch set (#17).
Change subject: IMPALA-915: Support cancel queries in frontend
......................................................................
IMPALA-915: Support cancel queries in frontend
Adds support to cancel a query during Frontend planning or metadata
operations. Frontend planning is handled by createExecRequest, so
registers Java Threads executing createExecRequest by their query ID and
provides cancelExecRequest to interrupt the Thread for a particular
query ID.
Adds support to cancel metadata operations with catalogd by sending a
Cancel request to catalogd for the relevant query ID. Registers Java
Threads executing select catalog operations by their query ID.
Cancellation is implemented by setting a boolean for the thread, and
calling Thread.interrupt to trigger InterruptedException from any wait
calls. Several ignored wait calls are updated to check the boolean and
throw an exception if the query has been cancelled, interrupting those
operations, and updates locks to lockInteruptibly.
Adds periodic checks to the planning process to interrupt planning.
They're primarily useful when planning is waiting on catalogd/HMS. If
planning gets into an algorithmically complex operation, it will not be
interrupted.
When analysis returns, cancellation is finalized in the backend. The
/cancel_query request returns once the query is cancelled. Cancelling
a request can no longer fail, so additional checks for whether the
request has been cancelled before it started executing are added.
Removes setting UpdateQueryStatus when GetExecRequest returns because
that's already handled in ImpalaServer::Execute when it calls
UnregisterQuery in response to an error, and constitutes an update race
on the status with UnregisterQuery triggered by CancelQueryHandler. We
want to use the status from CancelQueryHandler in this case as it
provides more context (about who initiated the cancel); the result of
GetExecRequest is just CancelledException. Avoids calling
UnregisterQuery in Execute if the query is already finalized to avoid
redundant "Invalid or unknown query handle" logs.
Removes check_inflight, as we can now cancel a query before it's
inflight. In the case that cancellation doesn't happen immediately -
because we're in a busy frontend loop that can't be interrupted -
/cancel will block until the frontend reaches an interruption point and
returns to the backend to finalize the query.
Enforces a delay in unregistering a query so clients have sufficient
time to poll GetOperationStatus and get back the cancellation message
rather than INVALID_QUERY_HANDLE. Modifies GetOperationStatus and GetLog
to return results for finalized queries that are not yet unregistered.
Testing:
- updates test_query_cancel_created to cancel a CREATED query
- added tests to cancel a query while metadata loading is delayed
- added local catalog mode tests for interrupting metadata operations
- removes test_query_cancel_exception, as it no longer demonstrates
relevant behavior; cancelling a query that will encounter an exception
before the exception occurs is no different than other queries
Change-Id: I0d25d4c7fb0b8dcc7dad9510db1e8dca220eeb86
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-service-client-wrapper.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/runtime/query-driver.cc
M be/src/runtime/query-driver.h
M be/src/runtime/runtime-state.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/fe-support.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/service/internal-server.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
A fe/src/main/java/org/apache/impala/catalog/CatalogCancelledException.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
A fe/src/main/java/org/apache/impala/common/CancelledException.java
A fe/src/main/java/org/apache/impala/service/Canceller.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java
M fe/src/test/java/org/apache/impala/common/QueryFixture.java
M shell/impala_client.py
M tests/beeswax/impala_beeswax.py
M tests/custom_cluster/test_hs2_fault_injection.py
M tests/custom_cluster/test_web_pages.py
M tests/util/web_pages_util.py
M tests/webserver/test_web_pages.py
46 files changed, 690 insertions(+), 263 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/21803/17
--
To view, visit http://gerrit.cloudera.org:8080/21803
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0d25d4c7fb0b8dcc7dad9510db1e8dca220eeb86
Gerrit-Change-Number: 21803
Gerrit-PatchSet: 17
Gerrit-Owner: Michael Smith <[email protected]>
Gerrit-Reviewer: Andrew Sherman <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Jason Fehr <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
