[
https://issues.apache.org/jira/browse/SPARK-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Wang updated SPARK-13935:
-----------------------------
Description:
We run a sql like "insert overwrite table store_returns partition
(sr_returned_date) select xxxxxx" using beeline then it will block other
beeline connection while invoke the Hive method via
"ClientWrapper.loadDynamicPartitions".
The reason is that "withHiveState" will lock "clientLoader". Sadly when a new
client comes, it will invoke "setConf" methods which is also sychronized with
"clientLoader".
So the problem is that if the first sql took very long time to run, then all
other client could not connect to thrift server successfully.
We tested on release 1.5.1. After checking the codes, i think latest release
has same issue, too.
was:
We run a sql like "insert overwrite table store_returns partition
(sr_returned_date) select xxxxxx" using beeline then it will block other
beeline connection while invoke the Hive method via
"ClientWrapper.loadDynamicPartitions".
The reason is that "withHiveState" will lock "clientLoader". Sadly when a new
client comes, it will invoke "setConf" methods which is also sychronized with
"clientLoader".
So the problem is that if the first sql took very long time to run, then all
other client could not connect to thrift server successfully.
We tested on release 1.5.1. not sure if latest release has same issue.
> Other clients' connection hang up when someone do huge load
> -----------------------------------------------------------
>
> Key: SPARK-13935
> URL: https://issues.apache.org/jira/browse/SPARK-13935
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0, 1.5.2, 1.6.0, 1.6.1
> Reporter: Tao Wang
> Priority: Critical
>
> We run a sql like "insert overwrite table store_returns partition
> (sr_returned_date) select xxxxxx" using beeline then it will block other
> beeline connection while invoke the Hive method via
> "ClientWrapper.loadDynamicPartitions".
> The reason is that "withHiveState" will lock "clientLoader". Sadly when a new
> client comes, it will invoke "setConf" methods which is also sychronized with
> "clientLoader".
> So the problem is that if the first sql took very long time to run, then all
> other client could not connect to thrift server successfully.
> We tested on release 1.5.1. After checking the codes, i think latest release
> has same issue, too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
