[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179202#comment-15179202
]
Hive QA commented on HIVE-13129:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790941/HIVE-13129.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9766 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7157/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7157/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7157/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790941 - PreCommit-HIVE-TRUNK-Build
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.2.patch, HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177859#comment-15177859
]
Chaoyu Tang commented on HIVE-13129:
+1
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.2.patch, HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175771#comment-15175771
]
Naveen Gangam commented on HIVE-13129:
--
It makes sense to attach the HMS connection lifecycle to the service's
lifecycle.
+1 for me pending pre-commit tests.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.2.patch, HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169470#comment-15169470
]
Aihua Xu commented on HIVE-13129:
-
If it's used, it's OK. But right now it's not used after that.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169312#comment-15169312
]
Chaoyu Tang commented on HIVE-13129:
IMO, it should not be a big problem. This HMS connection can be used to serve
possible HMS requests related to HS2 itself, it can live as long as HS2 does.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169084#comment-15169084
]
Aihua Xu commented on HIVE-13129:
-
When HiveServer2 stops, eventually the connection will be released. But if the
server keeps running, we will always have that connection to HMS during the
lifetime while it never gets used after that. And we will have one connection
for each HiveServer2 instance.
That can be considered as leaking I guess.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167726#comment-15167726
]
Hive QA commented on HIVE-13129:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789598/HIVE-13129.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9826 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7091/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7091/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7091/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12789598 - PreCommit-HIVE-TRUNK-Build
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167565#comment-15167565
]
Aihua Xu commented on HIVE-13129:
-
Only one HMS connection will be created but never released and not used later.
It will accumulate leaking HMS resource and database resource if HiveServer
keeps restarting.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167489#comment-15167489
]
Naveen Gangam commented on HIVE-13129:
--
[~aihuaxu] This should only be creating one HMS connection, during HS2 startup,
over the lifetime of the HS2. Do you see this causing an incremental leak? or
are you proactively closing that HMS connection because you think its never
used again?
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163299#comment-15163299
]
Aihua Xu commented on HIVE-13129:
-
+ [~thejas] [~ngangam] [~ctang.ma] You may have more background on this and can
you guys take a look at this simple change?
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163295#comment-15163295
]
Aihua Xu commented on HIVE-13129:
-
Attach the patch-1: we will close the HMS connection right after getting
authorization policy since we are not using the connection in that service main
thread anymore. All the incoming queries will be handled by thread pool and
will use separate HMS connections.
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
