Hello Bharath Vissapragada, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#15).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
......................................................................
IMPALA-8571[WIP]: improve QueryEventHook execution
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*query-event-hook.${hook_method}.execution-rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*query-event-hook.${hook_method}.execution-exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*query-event-hook.${hook_method}.execution-timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*query-event-hook.${hook_method}.execution-submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*query-event-hook.${hook_method}.mean-execution-time*
Mean time in [ns] that ${hook_name} has taken to complete,
whether normally or by error (e.g. timeout or exception).
*query-event-hook.${hook_method}.mean-queued-time*
Mean time in [ns] between hook task submission
and hook task execution. This indicates how long a hook
task has been queued waiting to execute.
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/CMakeLists.txt
M be/src/util/backend-gflag-util.cc
A be/src/util/hook-metrics-test.cc
A be/src/util/hook-metrics.cc
A be/src/util/hook-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
19 files changed, 1,368 insertions(+), 113 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/15
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 15
Gerrit-Owner: radford nguyen <radford.ngu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fre...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: radford nguyen <radford.ngu...@gmail.com>
