[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.8.patch

Added support for trigger based on TOTAL_TASKS. This will be required to avoid 
a task taking up too much of cluster resource by spinning up too many task. 

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.5.patch, HIVE-17508.6.patch, HIVE-17508.7.patch, 
> HIVE-17508.8.patch, HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.7.patch

Fixes TestHiveConf test failure

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.5.patch, HIVE-17508.6.patch, HIVE-17508.7.patch, 
> HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.6.patch

[~sershe] Can you please review the latest patch? Addressed your review 
comments, also added JDBC tests that works with global rules for tez and llap. 
Fixed test failures. 

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.5.patch, HIVE-17508.6.patch, HIVE-17508.WIP.2.patch, 
> HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.5.patch

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.5.patch, HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.4.patch

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.3.patch

Same patch rebased. 

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.WIP.2.patch, 
> HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.3.patch

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.2.patch

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Status: Patch Available  (was: Open)

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.WIP.2.patch, 
> HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.1.patch

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.WIP.2.patch, 
> HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.WIP.2.patch

Added simple expression parsing from string. Some more unit tests. 

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.WIP.patch

Attaching a WIP patch. This still has to be hooked to metastore objects. For 
query elapsed time, hive.query.timeout.seconds can be used which will fail the 
query exceeding timeout. 

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)