[GitHub] drill pull request #758: DRILL-5287: Provide option to skip updates of ephem...
Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/758 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #758: DRILL-5287: Provide option to skip updates of ephem...
Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/758#discussion_r103624801 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java --- @@ -280,31 +284,39 @@ public void interrupted(final InterruptedException ex) { } } - QueryState updateEphemeralState(final QueryState queryState) { -switch (queryState) { + void updateEphemeralState(final QueryState queryState) { + // If query is already in zk transient store, ignore the transient state update option. + // Else, they will not be removed from transient store upon completion. + if (!inTransientStore && + !foreman.getQueryContext().getOptions().getOption(ExecConstants.QUERY_TRANSIENT_STATE_UPDATE)) { +return; + } + + switch (queryState) { case ENQUEUED: case STARTING: case RUNNING: case CANCELLATION_REQUESTED: transientProfiles.put(stringQueryId, getQueryInfo()); // store as ephemeral query profile. +inTransientStore = true; break; case COMPLETED: case CANCELED: case FAILED: try { transientProfiles.remove(stringQueryId); + inTransientStore = false; } catch(final Exception e) { logger.warn("Failure while trying to delete the estore profile for this query.", e); } - break; default: throw new IllegalStateException("unrecognized queryState " + queryState); } -return queryState; +return; --- End diff -- remove unnecessary return --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #758: DRILL-5287: Provide option to skip updates of ephem...
Github user ppadma commented on a diff in the pull request: https://github.com/apache/drill/pull/758#discussion_r103487885 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java --- @@ -280,8 +281,15 @@ public void interrupted(final InterruptedException ex) { } } - QueryState updateEphemeralState(final QueryState queryState) { -switch (queryState) { + void updateEphemeralState(final QueryState queryState) { + // If query is already in zk transient store, ignore the transient state update option. + // Else, they will not be removed from transient store upon completion. + if (transientProfiles.get(stringQueryId) == null && --- End diff -- I want to bypass the option for the queries which are already in transient store when option is enabled. Otherwise, their state will never get updated and/or will never be removed from transient store. web UI will show these queries as running forever :-) Thanks for raising a good point regarding using transientProfiles.get. I made the change to update and use in memory state instead. Please review the new diffs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #758: DRILL-5287: Provide option to skip updates of ephem...
Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/758#discussion_r103364711 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java --- @@ -280,8 +281,15 @@ public void interrupted(final InterruptedException ex) { } } - QueryState updateEphemeralState(final QueryState queryState) { -switch (queryState) { + void updateEphemeralState(final QueryState queryState) { + // If query is already in zk transient store, ignore the transient state update option. + // Else, they will not be removed from transient store upon completion. + if (transientProfiles.get(stringQueryId) == null && --- End diff -- Why not just check the option? `transientProfiles.get(stringQueryId)` is quite expensive itself ([contacts ZooKeeper and deserializes data](https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZkEphemeralStore.java#L61)). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #758: DRILL-5287: Provide option to skip updates of ephem...
Github user ppadma commented on a diff in the pull request: https://github.com/apache/drill/pull/758#discussion_r103005063 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java --- @@ -1010,7 +1010,9 @@ public void addToEventQueue(final QueryState newState, final Exception exception private void recordNewState(final QueryState newState) { state = newState; -queryManager.updateEphemeralState(newState); +if (queryContext.getOptions().getOption(ExecConstants.ZK_QUERY_STATE_UPDATE)) { + queryManager.updateEphemeralState(newState); +} --- End diff -- For long running queries, it may not make much difference. It adds latency of around ~50-60 msec for single query. However, with high concurrency, impact of contention because of zookeeper updates is significant. Like I mentioned in the JIRA, for concurrency=100, the average query response time for simple queries is 8 sec vs 0.2 sec with these updates disabled. It does not impact the query profile. Query profile gets updated and written at the end of the query as usual. This option affects only running queries. In Web UI, you will not see running queries and their state. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #758: DRILL-5287: Provide option to skip updates of ephem...
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/758#discussion_r102863394 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java --- @@ -1010,7 +1010,9 @@ public void addToEventQueue(final QueryState newState, final Exception exception private void recordNewState(final QueryState newState) { state = newState; -queryManager.updateEphemeralState(newState); +if (queryContext.getOptions().getOption(ExecConstants.ZK_QUERY_STATE_UPDATE)) { + queryManager.updateEphemeralState(newState); +} --- End diff -- How does this affect query operation for long-running queries? How does it impact the query profile? If updates are enabled, do we still do an update at query completion to finalize the profile? If not, should writing of the profile be automatically disabled if status updates are disabled? Do we do any timeout on updates? Will we notice that the query has not been updated and, say, kill the query due to timeouts? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #758: DRILL-5287: Provide option to skip updates of ephem...
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/758#discussion_r102862961 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java --- @@ -413,4 +413,8 @@ String DYNAMIC_UDF_SUPPORT_ENABLED = "exec.udf.enable_dynamic_support"; BooleanValidator DYNAMIC_UDF_SUPPORT_ENABLED_VALIDATOR = new BooleanValidator(DYNAMIC_UDF_SUPPORT_ENABLED, true, true); + + String ZK_QUERY_STATE_UPDATE_KEY = "drill.exec.zk.query.state.update"; --- End diff -- Maybe ENABLE_QUERY_STATE_UPDATE_KEY? For the key name, maybe don't include ZK. ZK is our solution today, but this option would be equally valid if we used some other mechanism. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---