[
https://issues.apache.org/jira/browse/IMPALA-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17925089#comment-17925089
]
ASF subversion and git services commented on IMPALA-13703:
----------------------------------------------------------
Commit 80a45014ea0a7b73873b00ebfb3543feebb65021 in impala's branch
refs/heads/master from Yida Wu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=80a45014e ]
IMPALA-13703: Cancel running queries before shutdown deadline
Currently, when the graceful shutdown deadline is reached, Impala
daemon exits immediately, leaving any running queries unfinished.
This approach is not quite graceful, as it may result in unreleased
resources, such as scratch files in remote storage.
This patch adds a new state in the graceful shutdown process.
Before reaching the shutdown deadline, Impala daemon will try to
cancel any remaining running queries within a configurable timelimit
flag, shutdown_query_cancel_period_s. If this time limit exceeds
20% of the total shutdown deadline, it will be automatically
capped at that value. The idea is to cancel queries only near the
end of the graceful shutdown deadline. The 20% is the threshold to
allow us to take a more aggressive way to ensure a graceful
shutdown.
If all queries are successfully canceled within this period, the
server shuts down immediately. Otherwise, it shuts down once the
deadline is reached, with queries still running.
Tests:
Passed core tests.
Added testcases test_shutdown_coordinator_cancel_query and
test_shutdown_executor_with_query_cancel_period and
test_shutdown_coordinator_and_executor_cancel_query.
Manually tested shutdown a coord or an executor with long
running queries and they were canceled.
Change-Id: I1cac2e100d329644e21fdceb0b23901b08079130
Reviewed-on: http://gerrit.cloudera.org:8080/22422
Reviewed-by: Michael Smith <[email protected]>
Reviewed-by: Abhishek Rawat <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Handle running queries before graceful shutdown deadline
> --------------------------------------------------------
>
> Key: IMPALA-13703
> URL: https://issues.apache.org/jira/browse/IMPALA-13703
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Yida Wu
> Assignee: Yida Wu
> Priority: Critical
>
> During graceful shutdown, new requests can be blocked, however, if the
> deadline is reached, running queries are not canceled before the forced
> shutdown, which can result in resources, such as files being uploaded to a
> remote system during spilling, not being released. To handle this more
> gracefully, Impala should properly terminate the queries before the deadline
> and before the process exits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
