[
https://issues.apache.org/jira/browse/IMPALA-10964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429645#comment-17429645
]
carolinchen commented on IMPALA-10964:
--------------------------------------
Hi [~stigahuang] i will reuse the skew's calculation in IMPALA-10178.
this is my solution:
1. Set skew_limit by query option.
2. Register skew_limit check in ExpireQueries(), just like resource limit check
for CPU and scanned bytes.
3. Update the actual skew value in AggregateProfile.
4. Check the actual skew value with set value in query option,
if check failed, than cancel the query with error message; if check passed,
than go on loop.
There are some details:
1. The skew check internal can be set with startup flags in coorinator.
2. The cancelled query own to skew limit will be recored in metrics page.
3. The skew limit won't work, if it is not set.
> Add query option that limits skew query in runtime
> --------------------------------------------------
>
> Key: IMPALA-10964
> URL: https://issues.apache.org/jira/browse/IMPALA-10964
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 4.0.0
> Reporter: carolinchen
> Assignee: carolinchen
> Priority: Major
> Fix For: Impala 4.0.1
>
>
> Reject queries that skew value is too big when executing the query.
> Query skew refers to the situation in which some nodes are significantly
> behind other nodes in the process of concurrent execution of SQL.
> There are two style skews:
> 1. Row skew, which may be caused by unreasonable sql or uneven task
> distributions.
> 2. Time skew, which may be caused by different capability by execnode.
> Query skew will cause two effects:
> 1. For the skew node may execute slowly, which will slow down the query
> progress .
> 2. For the skew node may exhaust lots system resources( I/O, memory, rpc),
> which will
> affect other queries in the same host/ query pool.
> When the skew value reach unreasonale range, will affect the cluster status
> and other running queries. This is a mechanism to protect the cluster from
> potentially harmful queries(eg: mem_limit).
> In our environment, the SKEW_LIMIT query option is added to limit skewed
> query.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
