[
https://issues.apache.org/jira/browse/HIVE-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948374#comment-16948374
]
Qinghui Xu commented on HIVE-22306:
-----------------------------------
Thanks for the feedback, and that's a very good point.
What I'm currently doing:
* It turns out that thrift nonblocking server implementations are not
production ready (there's currently no secure implementations).
I'm trying first to add SASL support in nonblocking servers, as it will enable
the use of most common security mechanisms, especially kerberos. Here's the
work in progress: THRIFT-4889, and there is a PR for it. It would be great if
hive guys can help to review it so that I can get it merged.
* I'll try to use the thrift nonblocking sasl server implementation
(THRIFT-4889) in our hive fork, and test it in our preprod environment
(eventually go to prod).
* If everything is doing good, I'll propose a new feature (PR) for hive
metastore to run with sasl and (optional) nonblocking mode
* Eventually remove the blocking server for sasl mode, and have only
nonblocking server.
* Eventually use nonblocking server for other modes without sasl.
> Use nonblocking thrift server for metastore
> -------------------------------------------
>
> Key: HIVE-22306
> URL: https://issues.apache.org/jira/browse/HIVE-22306
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Qinghui Xu
> Priority: Major
>
> Currently hive metastore's threads are blocking for network io (it's using
> `TThreadPoolServer` behind the scene), which means with increasing use cases
> (in our tech stack there are different services relying on it, hiveserver2,
> spark, presto, and more, all with a significant number of users), to handle
> all connections it needs either a big thread pool or many instances with
> smaller thread pools. And often, those metastores will see their thread pool
> saturated, while the cpu usage is still quite low, just because most
> connections stay idle and only run a query from time to time. This is thus a
> great misuse of the computation resources.
> Thus I propose to use a non blocking threading model, and run computation
> asynchronously.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
