[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 23: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4411/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 23 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 28 Aug 2019 04:18:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 22: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4410/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 22 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 28 Aug 2019 04:06:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Bharath Vissapragada, Andrew Sherman, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#22).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
(WIP because still need to implement polling of metrics by backend)
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook execution*
Query event hooks are still executed in a thread-pool, but you can
now configure the type of (Java) threads used to execute them via
the newly-added backend flag `query_event_hook_use_daemon_threads`,
which takes a true/false value.
See Java Thread.setDaemon(boolean) for what a "daemon thread" entails.
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*query-event-hook.${hook_method}.execution-rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*query-event-hook.${hook_method}.execution-exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*query-event-hook.${hook_method}.execution-timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*query-event-hook.${hook_method}.execution-submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*query-event-hook.${hook_method}.mean-execution-time*
Mean time in [ns] that ${hook_name} has taken to complete,
whether normally or by error (e.g. timeout or exception).
*query-event-hook.${hook_method}.mean-queued-time*
Mean time in [ns] between hook task submission
and hook task execution. This indicates how long a hook
task has been queued waiting to execute.
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/CMakeLists.txt
M be/src/util/backend-gflag-util.cc
A be/src/util/hook-metrics-test.cc
A be/src/util/hook-metrics.cc
A be/src/util/hook-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
19 files changed, 1,364 insertions(+), 112 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/22
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 22
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Bharath Vissapragada, Andrew Sherman, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#23).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
(WIP because still need to implement polling of metrics by backend)
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook execution*
Query event hooks are still executed in a thread-pool, but you can
now configure the type of (Java) threads used to execute them via
the newly-added backend flag `query_event_hook_use_daemon_threads`,
which takes a true/false value.
See Java Thread.setDaemon(boolean) for what a "daemon thread" entails.
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*query-event-hook.${hook_method}.execution-rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*query-event-hook.${hook_method}.execution-exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*query-event-hook.${hook_method}.execution-timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*query-event-hook.${hook_method}.execution-submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*query-event-hook.${hook_method}.mean-execution-time*
Mean time in [ns] that ${hook_name} has taken to complete,
whether normally or by error (e.g. timeout or exception).
*query-event-hook.${hook_method}.mean-queued-time*
Mean time in [ns] between hook task submission
and hook task execution. This indicates how long a hook
task has been queued waiting to execute.
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/CMakeLists.txt
M be/src/util/backend-gflag-util.cc
A be/src/util/hook-metrics-test.cc
A be/src/util/hook-metrics.cc
A be/src/util/hook-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
19 files changed, 1,361 insertions(+), 112 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/23
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 23
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 21: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4409/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 21 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 28 Aug 2019 02:06:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Bharath Vissapragada, Andrew Sherman, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#21).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
(WIP because still need to implement polling of metrics by backend)
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook execution*
Query event hooks are still executed in a thread-pool, but you can
now configure the type of (Java) threads used to execute them via
the newly-added backend flag `query_event_hook_use_daemon_threads`,
which takes a true/false value.
See Java Thread.setDaemon(boolean) for what a "daemon thread" entails.
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*query-event-hook.${hook_method}.execution-rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*query-event-hook.${hook_method}.execution-exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*query-event-hook.${hook_method}.execution-timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*query-event-hook.${hook_method}.execution-submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*query-event-hook.${hook_method}.mean-execution-time*
Mean time in [ns] that ${hook_name} has taken to complete,
whether normally or by error (e.g. timeout or exception).
*query-event-hook.${hook_method}.mean-queued-time*
Mean time in [ns] between hook task submission
and hook task execution. This indicates how long a hook
task has been queued waiting to execute.
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/CMakeLists.txt
M be/src/util/backend-gflag-util.cc
A be/src/util/hook-metrics-test.cc
A be/src/util/hook-metrics.cc
A be/src/util/hook-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
19 files changed, 1,364 insertions(+), 112 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/21
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 21
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 20: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4397/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 20 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 27 Aug 2019 22:29:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Bharath Vissapragada, Andrew Sherman, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#20).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
(WIP because still need to implement polling of metrics by backend)
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook execution*
Query event hooks are still executed in a thread-pool, but you can
now configure the type of (Java) threads used to execute them via
the newly-added backend flag `query_event_hook_use_daemon_threads`,
which takes a true/false value.
See Java Thread.setDaemon(boolean) for what a "daemon thread" entails.
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*query-event-hook.${hook_method}.execution-rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*query-event-hook.${hook_method}.execution-exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*query-event-hook.${hook_method}.execution-timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*query-event-hook.${hook_method}.execution-submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*query-event-hook.${hook_method}.mean-execution-time*
Mean time in [ns] that ${hook_name} has taken to complete,
whether normally or by error (e.g. timeout or exception).
*query-event-hook.${hook_method}.mean-queued-time*
Mean time in [ns] between hook task submission
and hook task execution. This indicates how long a hook
task has been queued waiting to execute.
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
A be/src/rpc/299e-533f-3ecd-6fea/krb5kdc/.k5.KRBTEST.COM
A
be/src/rpc/299e-533f-3ecd-6fea/krb5kdc/impala-test_radimp.vpc.cloudera.com.keytab
A be/src/rpc/299e-533f-3ecd-6fea/krb5kdc/kdc.conf
A be/src/rpc/299e-533f-3ecd-6fea/krb5kdc/krb5.conf
A be/src/rpc/299e-533f-3ecd-6fea/krb5kdc/krb5cc
A be/src/rpc/299e-533f-3ecd-6fea/krb5kdc/principal
A be/src/rpc/299e-533f-3ecd-6fea/krb5kdc/principal.kadm5
A be/src/rpc/299e-533f-3ecd-6fea/krb5kdc/principal.kadm5.lock
A be/src/rpc/299e-533f-3ecd-6fea/krb5kdc/principal.ok
A be/src/rpc/54a1-4bf4-07b7-314f/krb5kdc/.k5.KRBTEST.COM
A
be/src/rpc/54a1-4bf4-07b7-314f/krb5kdc/impala-test_radimp.vpc.cloudera.com.keytab
A be/src/rpc/54a1-4bf4-07b7-314f/krb5kdc/kdc.conf
A be/src/rpc/54a1-4bf4-07b7-314f/krb5kdc/krb5.conf
A be/src/rpc/54a1-4bf4-07b7-314f/krb5kdc/krb5cc
A be/src/rpc/54a1-4bf4-07b7-314f/krb5kdc/principal
A be/src/rpc/54a1-4bf4-07b7-314f/krb5kdc/principal.kadm5
A be/src/rpc/54a1-4bf4-07b7-314f/krb5kdc/principal.kadm5.lock
A be/src/rpc/54a1-4bf4-07b7-314f/krb5kdc/principal.ok
A be/src/rpc/ab02-9d46-479e-73ba/krb5kdc/.k5.KRBTEST.COM
A
be/src/rpc/ab02-9d46-479e-73ba/krb5kdc/impala-test_radimp.vpc.cloudera.com.keytab
A be/src/rpc/ab02-9d46-479e-73ba/krb5kdc/kdc.conf
A be/src/rpc/ab02-9d46-479e-73ba/krb5kdc/krb5.conf
A be/src/rpc/ab02-9d46-479e-73ba/krb5kdc/krb5cc
A be/src/rpc/ab02-9d46-479e-73ba/krb5kdc/principal
A be/src/rpc/ab02-9d46-479e-73ba/krb5kdc/principal.kadm5
A be/src/rpc/ab02-9d46-479e-73ba/krb5kdc/principal.kadm5.lock
A be/src/rpc/ab02-9d46-479e-73ba/krb5kdc/principal.ok
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/CMakeLists.txt
M be/src/util/backend-gflag-util.cc
A be/src/util/hook-metrics-test.cc
A be/src/util/hook-metrics.cc
A be/src/util/hook-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4307/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 19 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 19 Aug 2019 23:11:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Bharath Vissapragada, Andrew Sherman, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#19).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
(WIP because still need to implement polling of metrics by backend)
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook execution*
Query event hooks are still executed in a thread-pool, but you can
now configure the type of (Java) threads used to execute them via
the newly-added backend flag `query_event_hook_use_daemon_threads`,
which takes a true/false value.
See Java Thread.setDaemon(boolean) for what a "daemon thread" entails.
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*query-event-hook.${hook_method}.execution-rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*query-event-hook.${hook_method}.execution-exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*query-event-hook.${hook_method}.execution-timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*query-event-hook.${hook_method}.execution-submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*query-event-hook.${hook_method}.mean-execution-time*
Mean time in [ns] that ${hook_name} has taken to complete,
whether normally or by error (e.g. timeout or exception).
*query-event-hook.${hook_method}.mean-queued-time*
Mean time in [ns] between hook task submission
and hook task execution. This indicates how long a hook
task has been queued waiting to execute.
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/CMakeLists.txt
M be/src/util/backend-gflag-util.cc
A be/src/util/hook-metrics-test.cc
A be/src/util/hook-metrics.cc
A be/src/util/hook-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
19 files changed, 1,364 insertions(+), 114 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/19
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 19
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
radford nguyen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13748 )
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
Patch Set 18:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13748/18/be/src/util/hook-metrics.cc
File be/src/util/hook-metrics.cc:
http://gerrit.cloudera.org:8080/#/c/13748/18/be/src/util/hook-metrics.cc@82
PS18, Line 82: void QueryEventHookMetricContainer::Update(const
TQueryEventHookMetrics& from) {
Is this all I really have to do to make sure the metrics will be published?
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 18
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Mon, 19 Aug 2019 22:16:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
radford nguyen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13748 )
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
Patch Set 18:
(10 comments)
Thanks for the feedback, asherman. Replies inline
http://gerrit.cloudera.org:8080/#/c/13748/18//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13748/18//COMMIT_MSG@31
PS18, Line 31:
> Do you want to discuss --query_event_hook_use_daemon_threads here? How woul
Yes, this is something that should be included in the commit message
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
File
fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@51
PS18, Line 51: * Rejections
> This javadoc is great, and just the sort of thing I was hoping to see.
I was not planning on publishing the html; I have just been in the habit of
using html because eclipse/Idea/etc will render it when displaying the javadoc
in a tooltip window. Everything just turns into one giant line in there if you
don't format.
Anyway, I'll switch to markdown, since it's both human-readable and can be
rendered by an IDE plugin or javadoc tool if needed.
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@102
PS18, Line 102: * ${hookClass}.${method}.execution.exceptions
> I think the metrics now have the "query-event-hook" prefix.
re: "query-event-hook" prefix
-
So this is where the split b/w backend/frontend metrics is kind of confusing.
This javadoc describes the metric names as they are collected in the frontend,
and unfortunately the names do not exactly align with the backend. Part of
this is due to the fact that the Java metrics are objects (e.g. Timers) that
can provide multiple metrics (e.g. mean time, count).
In passing the metrics to the backend, it seemed to me that the convention was
to pass the raw data (e.g. mean time, count) as needed, instead of say
translating the entire metric object to some similarly-capable object in the
backend.
The "query-event-hook" prefix only exists in the backend metric names.
re: hookClass vs. method
-
In this context, ${hookClass} is always the fully-qualified name of the java
class that implements `QueryEventHook`, and ${hookMethod} the method name
invoked on that hook. An example metric name would be:
org.foo.bar.MyQueryEventHook.onQueryComplete.execution.exceptions
I actually tried to make this very predictable and therefore amenable to
automation/generation, but if there is some way I can improve that further,
please let me know.
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@176
PS18, Line 176: // ArrayBlockingQueue contructor performs bounds-check on
queue size for us
> typo: constructor
Done
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@184
PS18, Line 184: // this executor cancels any hook tasks that
> This may seem picky but in Imapla we like comments with Capitals at the beg
Done
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@255
PS18, Line 255:* {@link ExecutionException}. This behavior is consistent
with how futures normally
> We could be more concise by not saying this about how Futures normally beha
Done
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@286
PS18, Line 286: LOG.warn("QueryEventHook {}.{} execution rejected because
the " +
> I sometimes weaselly write "probably because" as you never know :-)
Done
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
File
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java:
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java@1
PS18, Line 1: /**
> Use // style comments for the apache license please
Done
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java@80
PS18, Line 80: // make this longer to reasonably guarantee a timeout
> Is this a FIXME note, not sure I understand what this keans
Reworded as: "Sleep much longer than `hookTimeout` to reasonably guarantee a
timeout."
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java@115
PS18, Line 115:
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/13748 )
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
Patch Set 18:
(11 comments)
I think this going in the right direction
http://gerrit.cloudera.org:8080/#/c/13748/18//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13748/18//COMMIT_MSG@31
PS18, Line 31:
Do you want to discuss --query_event_hook_use_daemon_threads here? How would a
user determine what value to use for that?
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
File
fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@51
PS18, Line 51: * Rejections
This javadoc is great, and just the sort of thing I was hoping to see.
Are you expecting to publish the html that is generated from
this javadoc? If not then I think Impala code generally does not include the
formatting stuff like . I think we generally emphasize readability in
the editor.
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@102
PS18, Line 102: * ${hookClass}.${method}.execution.exceptions
I think the metrics now have the "query-event-hook" prefix.
I'm also unclear about the exact meanings of hookClass and method. Detail in
metrics is good, but so is predictability.
If I can't predict the metrics name then it is harder to write tooling that
uses it. I can see this both ways, what do you think?
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@176
PS18, Line 176: // ArrayBlockingQueue contructor performs bounds-check on
queue size for us
typo: constructor
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@184
PS18, Line 184: // this executor cancels any hook tasks that
This may seem picky but in Imapla we like comments with Capitals at the
beginning and periods at the end.
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@255
PS18, Line 255:* {@link ExecutionException}. This behavior is consistent
with how futures normally
We could be more concise by not saying this about how Futures normally behave,
as that's what we expect, right?
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@286
PS18, Line 286: LOG.warn("QueryEventHook {}.{} execution rejected because
the " +
I sometimes weaselly write "probably because" as you never know :-)
I know executor,getActiveCount() is only a snapshot but it might show something
interesting.
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
File
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java:
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java@1
PS18, Line 1: /**
Use // style comments for the apache license please
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java@49
PS18, Line 49: new QueryCompleteContext("whatever");
lol
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java@80
PS18, Line 80: // make this longer to reasonably guarantee a timeout
Is this a FIXME note, not sure I understand what this keans
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java@115
PS18, Line 115: if (expected.getCause() instanceof TimeoutException) {
if (!(expected.getCause() instanceof TimeoutException)) {
fail("TimeoutException expected but not thrown");
}
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 18
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Tue, 13 Aug 2019 18:16:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4045/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 18 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Sat, 27 Jul 2019 02:52:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Bharath Vissapragada, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#18).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
(WIP because still need to implement polling of metrics by backend)
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*query-event-hook.${hook_method}.execution-rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*query-event-hook.${hook_method}.execution-exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*query-event-hook.${hook_method}.execution-timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*query-event-hook.${hook_method}.execution-submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*query-event-hook.${hook_method}.mean-execution-time*
Mean time in [ns] that ${hook_name} has taken to complete,
whether normally or by error (e.g. timeout or exception).
*query-event-hook.${hook_method}.mean-queued-time*
Mean time in [ns] between hook task submission
and hook task execution. This indicates how long a hook
task has been queued waiting to execute.
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/CMakeLists.txt
M be/src/util/backend-gflag-util.cc
A be/src/util/hook-metrics-test.cc
A be/src/util/hook-metrics.cc
A be/src/util/hook-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
19 files changed, 1,376 insertions(+), 114 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/18
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 18
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 17: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4015/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 17 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Jul 2019 05:16:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 17: (1 comment) http://gerrit.cloudera.org:8080/#/c/13748/17/be/src/util/hook-metrics.cc File be/src/util/hook-metrics.cc: http://gerrit.cloudera.org:8080/#/c/13748/17/be/src/util/hook-metrics.cc@58 PS17, Line 58: line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 17 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 26 Jul 2019 04:36:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Bharath Vissapragada, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#17).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
(WIP because still need to implement polling of metrics by backend)
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*query-event-hook.${hook_method}.execution-rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*query-event-hook.${hook_method}.execution-exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*query-event-hook.${hook_method}.execution-timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*query-event-hook.${hook_method}.execution-submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*query-event-hook.${hook_method}.mean-execution-time*
Mean time in [ns] that ${hook_name} has taken to complete,
whether normally or by error (e.g. timeout or exception).
*query-event-hook.${hook_method}.mean-queued-time*
Mean time in [ns] between hook task submission
and hook task execution. This indicates how long a hook
task has been queued waiting to execute.
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/CMakeLists.txt
M be/src/util/backend-gflag-util.cc
A be/src/util/hook-metrics-test.cc
A be/src/util/hook-metrics.cc
A be/src/util/hook-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
19 files changed, 1,376 insertions(+), 114 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/17
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 17
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 16: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3969/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 16 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 24 Jul 2019 01:17:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 16: (1 comment) http://gerrit.cloudera.org:8080/#/c/13748/16/be/src/util/hook-metrics.cc File be/src/util/hook-metrics.cc: http://gerrit.cloudera.org:8080/#/c/13748/16/be/src/util/hook-metrics.cc@58 PS16, Line 58: line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 16 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 24 Jul 2019 00:52:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Bharath Vissapragada, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#16).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
(WIP because still need to implement polling of metrics by backend)
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*query-event-hook.${hook_method}.execution-rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*query-event-hook.${hook_method}.execution-exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*query-event-hook.${hook_method}.execution-timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*query-event-hook.${hook_method}.execution-submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*query-event-hook.${hook_method}.mean-execution-time*
Mean time in [ns] that ${hook_name} has taken to complete,
whether normally or by error (e.g. timeout or exception).
*query-event-hook.${hook_method}.mean-queued-time*
Mean time in [ns] between hook task submission
and hook task execution. This indicates how long a hook
task has been queued waiting to execute.
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/CMakeLists.txt
M be/src/util/backend-gflag-util.cc
A be/src/util/hook-metrics-test.cc
A be/src/util/hook-metrics.cc
A be/src/util/hook-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
19 files changed, 1,375 insertions(+), 113 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/16
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 16
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
radford nguyen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 15: > Patch Set 15: > > Build Failed > > https://jenkins.impala.io/job/gerrit-code-review-checks/3879/ : Initial code > review checks failed. See linked job for details on the failure. Linker failed due to undefined static member `QueryEventHook::metric_containers_` in be/src/util/hook-metrics.cc -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 15 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 15 Jul 2019 20:00:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 15: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3879/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 15 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 15 Jul 2019 12:59:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
radford nguyen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13748 )
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
Patch Set 15:
(3 comments)
Updated the review with metric code, including actual metric collection on the
frontend, metric "translation" on the backend, and the JNI plumbing to connect
the 2.
Only thing missing is the actual polling mechanism to make the connection...
currently looking at other examples (e.g. event metrics) for some guidance.
Commit `360b23bfa4eeb337cce380512f106a692bf3c49b` seems to be a good starting
point(?)
http://gerrit.cloudera.org:8080/#/c/13748/15/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/13748/15/be/src/service/impala-server.cc@75
PS15, Line 75: #include "util/hook-metrics.h"
This will be used once I determine the best way to poll for metrics from the
frontend
http://gerrit.cloudera.org:8080/#/c/13748/15/be/src/util/hook-metrics-test.cc
File be/src/util/hook-metrics-test.cc:
http://gerrit.cloudera.org:8080/#/c/13748/15/be/src/util/hook-metrics-test.cc@23
PS15, Line 23: TEST(HookMetricContainer, UpdateTest) {
Review is still WIP: I haven't actually ran this test yet, and I need to add
more coverage for backend handling of metrics.
http://gerrit.cloudera.org:8080/#/c/13748/15/be/src/util/hook-metrics.h
File be/src/util/hook-metrics.h:
http://gerrit.cloudera.org:8080/#/c/13748/15/be/src/util/hook-metrics.h@65
PS15, Line 65: TGetQueryEventHookMetricsResult* response);
Review is still WIP because I need to actually call this function with metrics
from the frontend. The JNI calls already exist, but I need some feedback on
how to poll the frontend periodically for those metrics.
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 15
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Mon, 15 Jul 2019 12:45:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 15: (3 comments) http://gerrit.cloudera.org:8080/#/c/13748/15/be/src/util/hook-metrics-test.cc File be/src/util/hook-metrics-test.cc: http://gerrit.cloudera.org:8080/#/c/13748/15/be/src/util/hook-metrics-test.cc@40 PS15, Line 40: const int64_t num_execution_submissions = 11; line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13748/15/be/src/util/hook-metrics-test.cc@42 PS15, Line 42: const double queued_mean_time = 6672; line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13748/15/be/src/util/hook-metrics.cc File be/src/util/hook-metrics.cc: http://gerrit.cloudera.org:8080/#/c/13748/15/be/src/util/hook-metrics.cc@58 PS15, Line 58: line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 15 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 15 Jul 2019 12:34:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Bharath Vissapragada, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#15).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*query-event-hook.${hook_method}.execution-rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*query-event-hook.${hook_method}.execution-exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*query-event-hook.${hook_method}.execution-timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*query-event-hook.${hook_method}.execution-submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*query-event-hook.${hook_method}.mean-execution-time*
Mean time in [ns] that ${hook_name} has taken to complete,
whether normally or by error (e.g. timeout or exception).
*query-event-hook.${hook_method}.mean-queued-time*
Mean time in [ns] between hook task submission
and hook task execution. This indicates how long a hook
task has been queued waiting to execute.
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/CMakeLists.txt
M be/src/util/backend-gflag-util.cc
A be/src/util/hook-metrics-test.cc
A be/src/util/hook-metrics.cc
A be/src/util/hook-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
19 files changed, 1,368 insertions(+), 113 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/15
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 15
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13748 )
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
Patch Set 14:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
File
fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@327
PS14, Line 327: final Future f = hookExecutor_.submit(() ->
{
> I think I'm missing something. If I just do new FutureTask<>(callHook).run
Like we discussed offline, I meant new Thread(Future).run() but we will have to
bear the thread creation cost repeatedly. So probably fine to leave it as is
(Also consider using Executors.newCachedThreadPool)
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 14
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Thu, 11 Jul 2019 18:21:36 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
radford nguyen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 14: > Patch Set 14: > > (8 comments) > > > Patch Set 14: > > > > (9 comments) > > Replied to comments -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 14 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 11 Jul 2019 00:06:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
radford nguyen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13748 )
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
Patch Set 14:
(8 comments)
> Patch Set 14:
>
> (9 comments)
Replied to comments
http://gerrit.cloudera.org:8080/#/c/13748/14/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/13748/14/be/src/service/impala-server.cc@275
PS14, Line 275: 1
> Since the hook execution is async (to query unregister) anyway, how about w
That's a good idea; I'll bump it
http://gerrit.cloudera.org:8080/#/c/13748/14/be/src/service/impala-server.cc@279
PS14, Line 279: true
> Should anyone ever have to set this to false? If not, I suggest removing th
Nobody has requested it, but I was under the impression that a user might want
to use higher-priority threads for executing hooks. They might also want to
ensure that the threads will not be killed at shutdown until the hooks have
completed executing.
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
File
fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@97
PS14, Line 97: * ${hookClass}.${method}.execution.rejections
> Where are all these metrics exposed? I don't see the plumbing to make them
I haven't completed the plumbing yet; I'm still trying to figure out how it
works. This is the main reason the CR is still WIP
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@101
PS14, Line 101: exceptions
> nit: failures?
I can change it to failures if you think that's more expressive
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@131
PS14, Line 131: private final long hookTimeout_;
: private final TimeUnit hookTimeoutUnit_;
> Just force it to be secs or something?
Yes, it is forced to `SECONDS` in the `QueryHookManager` when instantiating
this executor. I am allowing a different time unit here internally to make the
unit tests faster.
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@290
PS14, Line 290: context);
> probably helps to include query ID explicitly.
Query ID is not currently available; are you suggesting that it be passed to
the hook from the backend? I do like that idea
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@327
PS14, Line 327: final Future f = hookExecutor_.submit(() ->
{
> I don't think you need to offload this to a separate executor pool again.
I think I'm missing something. If I just do new FutureTask<>(callHook).run(),
won't that just run callHook in this thread? And doesn't that mean that we
won't get to line 337 until callHook has completed?
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
File fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java:
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java@77
PS14, Line 77: static final String BE_HOOKS_FLAG = "query_event_hook_classes";
> nit: any reason to remove private qualifier?
I used one of the constants in an error message logged by
`FixedCapacityQueryHookExecutor`. If you think it's cleaner, I can make these
private again and maybe(?) make `FCQHExecutor` an inner class
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 14
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Thu, 11 Jul 2019 00:04:35 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13748 )
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
Patch Set 14:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/13748/14/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/13748/14/be/src/service/impala-server.cc@275
PS14, Line 275: 1
Since the hook execution is async (to query unregister) anyway, how about we
bump this up a little? 3s or so?
http://gerrit.cloudera.org:8080/#/c/13748/14/be/src/service/impala-server.cc@277
PS14, Line 277: default 1.
not needed, the web UI shows the default separately. (same below)
http://gerrit.cloudera.org:8080/#/c/13748/14/be/src/service/impala-server.cc@279
PS14, Line 279: true
Should anyone ever have to set this to false? If not, I suggest removing this
flag.
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
File
fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@97
PS14, Line 97: * ${hookClass}.${method}.execution.rejections
Where are all these metrics exposed? I don't see the plumbing to make them
available on /metrics (or may be I'm missing the link somwhere)
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@101
PS14, Line 101: exceptions
nit: failures?
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@131
PS14, Line 131: private final long hookTimeout_;
: private final TimeUnit hookTimeoutUnit_;
Just force it to be secs or something?
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@290
PS14, Line 290: context);
probably helps to include query ID explicitly.
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@327
PS14, Line 327: final Future f = hookExecutor_.submit(() ->
{
I don't think you need to offload this to a separate executor pool again. You
could probably just do new FutureTask<>(Callable).run() (essentially what the
hookExecutor_ does for you) but I think that simplifies the code and make it
easy to reason about (1:1 correspondence between threadPool thread accepting
the task and the thread that actually executes the hook).
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
File fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java:
http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java@77
PS14, Line 77: static final String BE_HOOKS_FLAG = "query_event_hook_classes";
nit: any reason to remove private qualifier?
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 14
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Wed, 10 Jul 2019 17:23:10 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3834/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 14 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 08 Jul 2019 10:36:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3833/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 13 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 08 Jul 2019 10:26:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Bharath Vissapragada, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#14).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*${hookClass}.${method}.execution.rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*${hookClass}.${method}.execution.exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*${hookClass}.${method}.execution.timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*${hookClass}.${method}.submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*${hookClass}.${method}.execution.time*
Timer indicating the amount of time ${hookClass}.${method}
has taken to complete, whether normally or by error (e.g. timeout or
exception).
Because tasks can potentially be cancelled before beginning execution, the
Timer.getCount() of this timer may be less than
`${hookClass}.${method}.submissions`
*${hookClass}.${method}.queued.time*
Timer indicating the amount of time between hook task submission
and hook task execution. Since a task may be cancelled before it even begins
execution, {@link Timer#getCount()} of this timer may be less than
`${hookClass}.${method}.submissions`
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
10 files changed, 818 insertions(+), 109 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/14
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 14
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 13: (3 comments) http://gerrit.cloudera.org:8080/#/c/13748/13/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/13748/13/be/src/service/impala-server.cc@282 PS13, Line 282: DEFINE_int32(query_event_hook_queue_capacity, 16, "Maximum number of QueryEventHook tasks " line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/13748/13/be/src/service/impala-server.cc@283 PS13, Line 283: "to queue before tasks begin to get rejected. Tasks are queued when there are no free " line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/13748/13/be/src/util/backend-gflag-util.cc File be/src/util/backend-gflag-util.cc: http://gerrit.cloudera.org:8080/#/c/13748/13/be/src/util/backend-gflag-util.cc@162 PS13, Line 162: cfg.__set_query_event_hook_use_daemon_threads(FLAGS_query_event_hook_use_daemon_threads); line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 13 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 08 Jul 2019 09:47:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#13).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*${hookClass}.${method}.execution.rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*${hookClass}.${method}.execution.exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*${hookClass}.${method}.execution.timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*${hookClass}.${method}.submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*${hookClass}.${method}.execution.time*
Timer indicating the amount of time ${hookClass}.${method}
has taken to complete, whether normally or by error (e.g. timeout or
exception).
Because tasks can potentially be cancelled before beginning execution, the
Timer.getCount() of this timer may be less than
`${hookClass}.${method}.submissions`
*${hookClass}.${method}.queued.time*
Timer indicating the amount of time between hook task submission
and hook task execution. Since a task may be cancelled before it even begins
execution, {@link Timer#getCount()} of this timer may be less than
`${hookClass}.${method}.submissions`
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
10 files changed, 817 insertions(+), 109 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/13
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 13
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3831/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 11 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 08 Jul 2019 04:52:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Fredy Wijaya, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13748 to look at the new patch set (#12). Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. IMPALA-8571[WIP]: improve QueryEventHook execution This commit hardens guarantees around QueryEventHook execution by adding the following features: *hook timeout/cancellation* *hook rejection* *hook performance metrics* Testing: - added unit tests for new features - re-ran existing E2E tests Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 --- M be/src/service/impala-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java A fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java 10 files changed, 782 insertions(+), 109 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/12 -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 12 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen
