Hello Bharath Vissapragada, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#14).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
......................................................................
IMPALA-8571[WIP]: improve QueryEventHook execution
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*${hookClass}.${method}.execution.rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*${hookClass}.${method}.execution.exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*${hookClass}.${method}.execution.timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*${hookClass}.${method}.submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*${hookClass}.${method}.execution.time*
Timer indicating the amount of time ${hookClass}.${method}
has taken to complete, whether normally or by error (e.g. timeout or
exception).
Because tasks can potentially be cancelled before beginning execution, the
Timer.getCount() of this timer may be less than
`${hookClass}.${method}.submissions`
*${hookClass}.${method}.queued.time*
Timer indicating the amount of time between hook task submission
and hook task execution. Since a task may be cancelled before it even begins
execution, {@link Timer#getCount()} of this timer may be less than
`${hookClass}.${method}.submissions`
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
10 files changed, 818 insertions(+), 109 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/14
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 14
Gerrit-Owner: radford nguyen <[email protected]>
Gerrit-Reviewer: Bharath Vissapragada <[email protected]>
Gerrit-Reviewer: Fredy Wijaya <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: radford nguyen <[email protected]>
