[
https://issues.apache.org/jira/browse/IMPALA-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17230168#comment-17230168
]
Aman Sinha commented on IMPALA-10317:
-------------------------------------
Hugely expanding joins could indeed create runaway queries that hog resources
on the cluster but I am curious if you already considered the EXEC_TIME_LIMIT_S
option for cancelling such queries. Also note that a simple 2 table join with
very large output cardinality that is streamed back to the client (i.e no other
blocking operators after the join) is typically not a big problem. Are you
aware of any other DBMS that has join specific limits ? Would be good to
compare and contrast this approach.
> Add query option that limits join #rows at runtime
> --------------------------------------------------
>
> Key: IMPALA-10317
> URL: https://issues.apache.org/jira/browse/IMPALA-10317
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
> Reporter: Fucun Chu
> Assignee: Fucun Chu
> Priority: Major
> Attachments: query82_summary.png
>
>
> Reject queries that rows produced too bigger by join operator when executing
> the query.
> This is a mechanism to protect the cluster from potentially harmful queries.
> When the cardinality of the table is very large and the join conditions are
> very bad, the number of rows produced by the join will be very large,
> sometimes tens of billions, which affects the cluster status and other
> running queries.
> In our environment, the NUM_JOIN_ROWS_PRODUCED_LIMIT query option is added to
> limit the number of rows produced by a single join operator.
> Implementation refers to
> [IMPALA-6034|https://issues.apache.org/jira/browse/IMPALA-6034] and summary
> (see the figure below), check the join operator #rows size
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
