[
https://issues.apache.org/jira/browse/DRILL-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883279#comment-15883279
]
ASF GitHub Bot commented on DRILL-5287:
---------------------------------------
Github user ppadma commented on a diff in the pull request:
https://github.com/apache/drill/pull/758#discussion_r103005063
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -1010,7 +1010,9 @@ public void addToEventQueue(final QueryState
newState, final Exception exception
private void recordNewState(final QueryState newState) {
state = newState;
- queryManager.updateEphemeralState(newState);
+ if
(queryContext.getOptions().getOption(ExecConstants.ZK_QUERY_STATE_UPDATE)) {
+ queryManager.updateEphemeralState(newState);
+ }
--- End diff --
For long running queries, it may not make much difference. It adds latency
of around ~50-60 msec for single query. However, with high concurrency, impact
of contention because of zookeeper updates is significant. Like I mentioned in
the JIRA, for concurrency=100, the average query response time for simple
queries is 8 sec vs 0.2 sec with these updates disabled. It does not impact
the query profile. Query profile gets updated and written at the end of the
query as usual. This option affects only running queries. In Web UI, you will
not see running queries and their state.
> Provide option to skip updates of ephemeral state changes in Zookeeper
> ----------------------------------------------------------------------
>
> Key: DRILL-5287
> URL: https://issues.apache.org/jira/browse/DRILL-5287
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.9.0
> Reporter: Padma Penumarthy
> Assignee: Padma Penumarthy
> Fix For: 1.10
>
>
> We put transient profiles in zookeeper and update state as query progresses
> and changes states. It is observed that this adds latency of ~45msec for each
> update in the query execution path. This gets even worse when high number of
> concurrent queries are in progress. For concurrency=100, the average query
> response time even for short queries is 8 sec vs 0.2 sec with these updates
> disabled. For short lived queries in a high-throughput scenario, it is of no
> value to update state changes in zookeeper. We need an option to disable
> these updates for short running operational queries.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
