radford nguyen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 )
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution ...................................................................... Patch Set 14: (8 comments) > Patch Set 14: > > (9 comments) Replied to comments http://gerrit.cloudera.org:8080/#/c/13748/14/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/13748/14/be/src/service/impala-server.cc@275 PS14, Line 275: 1 > Since the hook execution is async (to query unregister) anyway, how about w That's a good idea; I'll bump it http://gerrit.cloudera.org:8080/#/c/13748/14/be/src/service/impala-server.cc@279 PS14, Line 279: true > Should anyone ever have to set this to false? If not, I suggest removing th Nobody has requested it, but I was under the impression that a user might want to use higher-priority threads for executing hooks. They might also want to ensure that the threads will not be killed at shutdown until the hooks have completed executing. http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java File fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java: http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@97 PS14, Line 97: * <dt>${hookClass}.${method}.execution.rejections</dt> > Where are all these metrics exposed? I don't see the plumbing to make them I haven't completed the plumbing yet; I'm still trying to figure out how it works. This is the main reason the CR is still WIP http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@101 PS14, Line 101: exceptions > nit: failures? I can change it to failures if you think that's more expressive http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@131 PS14, Line 131: private final long hookTimeout_; : private final TimeUnit hookTimeoutUnit_; > Just force it to be secs or something? Yes, it is forced to `SECONDS` in the `QueryHookManager` when instantiating this executor. I am allowing a different time unit here internally to make the unit tests faster. http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@290 PS14, Line 290: context); > probably helps to include query ID explicitly. Query ID is not currently available; are you suggesting that it be passed to the hook from the backend? I do like that idea http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@327 PS14, Line 327: final Future<QueryEventHook> f = hookExecutor_.submit(() -> { > I don't think you need to offload this to a separate executor pool again. I think I'm missing something. If I just do new FutureTask<>(callHook).run(), won't that just run callHook in this thread? And doesn't that mean that we won't get to line 337 until callHook has completed? http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java File fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java: http://gerrit.cloudera.org:8080/#/c/13748/14/fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java@77 PS14, Line 77: static final String BE_HOOKS_FLAG = "query_event_hook_classes"; > nit: any reason to remove private qualifier? I used one of the constants in an error message logged by `FixedCapacityQueryHookExecutor`. If you think it's cleaner, I can make these private again and maybe(?) make `FCQHExecutor` an inner class -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 14 Gerrit-Owner: radford nguyen <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Fredy Wijaya <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: radford nguyen <[email protected]> Gerrit-Comment-Date: Thu, 11 Jul 2019 00:04:35 +0000 Gerrit-HasComments: Yes
