[
https://issues.apache.org/jira/browse/HIVE-29048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dayakar M updated HIVE-29048:
-----------------------------
Description:
Increase default pool and queue size of HS2 Background Async Threadpool from
*100* to *250:*
Currently the default value for
[hive.server2.async.exec.threads|https://github.com/apache/hive/blob/68f8734b3b7db2fdc8bc0c11ad3c37eb2e2a808c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L4233-L4234]
and
[hive.server2.async.exec.wait.queue.size|https://github.com/apache/hive/blob/68f8734b3b7db2fdc8bc0c11ad3c37eb2e2a808c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L4238-L4240]
configurations related to HS2 Async thread pool which is used for
handling/processing the client requests/queries coming to the HS2 is {*}100{*}.
Recently most of the customers reported saying below error on beeline
{noformat}
0: jdbc:hive2://localhost> show databases; Error: The background threadpool
cannot accept new task for execution, please retry the operation
(state=,code=0){noformat}
HS2 server logs shows below exception
{noformat}
Caused by: java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask@24363395 rejected from
java.util.concurrent.ThreadPoolExecutor@54ef7045[Running, pool size = 100,
active threads = 100, queued tasks = 100, completed tasks = 682504]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
~[?:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
~[?:1.8.0_342]
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
~[?:1.8.0_342]
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
~[?:1.8.0_342]
at
org.apache.hive.service.cli.session.SessionManager.submitBackgroundOperation(SessionManager.java:760)
~[hive-service-3.1.3]
at
org.apache.hive.service.cli.session.HiveSessionImpl.submitBackgroundOperation(HiveSessionImpl.java:598)
~[hive-service-3.1.3]
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:279)
~[hive-service-3.1.3] ... 50 more{noformat}
So here for most of the customer's load the default value *100* is not
sufficient so their queries are getting rejected. An average around 20+
customers are reported this issue per year. May be most of the other
organization's customers are also facing this issue.
So I am proposing to increase this default value from *100* to *250* for smooth
handling of customer load.
was:
Increase default pool and queue size of HS2 Background Async Threadpool from
*100* to *250:*
Currently the default value for
[hive.server2.async.exec.threads|https://github.com/apache/hive/blob/68f8734b3b7db2fdc8bc0c11ad3c37eb2e2a808c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L4233-L4234]
and
[hive.server2.async.exec.wait.queue.size|https://github.com/apache/hive/blob/68f8734b3b7db2fdc8bc0c11ad3c37eb2e2a808c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L4238-L4240]
configurations related to HS2 Async thread pool which is used for
handling/processing the client requests/queries coming to the HS2 is {*}100{*}.
Recently most of the customers reported saying below error on beeline
{noformat}
0: jdbc:hive2://localhost> show databases; Error: The background threadpool
cannot accept new task for execution, please retry the operation
(state=,code=0){noformat}
HS2 server logs shows below exception
{noformat}
Caused by: java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask@24363395 rejected from
java.util.concurrent.ThreadPoolExecutor@54ef7045[Running, pool size = 100,
active threads = 100, queued tasks = 100, completed tasks = 682504]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
~[?:1.8.0_342] at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
~[?:1.8.0_342] at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
~[?:1.8.0_342] at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
~[?:1.8.0_342] at
org.apache.hive.service.cli.session.SessionManager.submitBackgroundOperation(SessionManager.java:760)
~[hive-service-3.1.3] at
org.apache.hive.service.cli.session.HiveSessionImpl.submitBackgroundOperation(HiveSessionImpl.java:598)
~[hive-service-3.1.3] at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:279)
~[hive-service-3.1.3] ... 50 more{noformat}
So here for most of the customer's load the default value *100* is not
sufficient so their queries are getting rejected. An average around 20+
customers are reported this issue per year. May be most of the other
organization's customers are also facing this issue.
So I am proposing to increase this default value from *100* to *250* for smooth
handling of customer load.
> Increase default pool and queue size of HS2 Background Async Threadpool.
> ------------------------------------------------------------------------
>
> Key: HIVE-29048
> URL: https://issues.apache.org/jira/browse/HIVE-29048
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Dayakar M
> Assignee: Dayakar M
> Priority: Minor
>
> Increase default pool and queue size of HS2 Background Async Threadpool from
> *100* to *250:*
> Currently the default value for
> [hive.server2.async.exec.threads|https://github.com/apache/hive/blob/68f8734b3b7db2fdc8bc0c11ad3c37eb2e2a808c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L4233-L4234]
> and
> [hive.server2.async.exec.wait.queue.size|https://github.com/apache/hive/blob/68f8734b3b7db2fdc8bc0c11ad3c37eb2e2a808c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L4238-L4240]
> configurations related to HS2 Async thread pool which is used for
> handling/processing the client requests/queries coming to the HS2 is
> {*}100{*}.
>
> Recently most of the customers reported saying below error on beeline
> {noformat}
> 0: jdbc:hive2://localhost> show databases; Error: The background threadpool
> cannot accept new task for execution, please retry the operation
> (state=,code=0){noformat}
>
> HS2 server logs shows below exception
> {noformat}
> Caused by: java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@24363395 rejected from
> java.util.concurrent.ThreadPoolExecutor@54ef7045[Running, pool size = 100,
> active threads = 100, queued tasks = 100, completed tasks = 682504]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> ~[?:1.8.0_342]
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> ~[?:1.8.0_342]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> ~[?:1.8.0_342]
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
> ~[?:1.8.0_342]
> at
> org.apache.hive.service.cli.session.SessionManager.submitBackgroundOperation(SessionManager.java:760)
> ~[hive-service-3.1.3]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.submitBackgroundOperation(HiveSessionImpl.java:598)
> ~[hive-service-3.1.3]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:279)
> ~[hive-service-3.1.3] ... 50 more{noformat}
> So here for most of the customer's load the default value *100* is not
> sufficient so their queries are getting rejected. An average around 20+
> customers are reported this issue per year. May be most of the other
> organization's customers are also facing this issue.
>
> So I am proposing to increase this default value from *100* to *250* for
> smooth handling of customer load.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
