[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[ https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972619#comment-16972619 ] David Mollitor commented on HIVE-22469: --- [~mgergely] Thank you for the review! Pushed to master. > Lower Metastore DB Connection Pool Size in QTests > - > > Key: HIVE-22469 > URL: https://issues.apache.org/jira/browse/HIVE-22469 > Project: Hive > Issue Type: Improvement > Components: Test, Tests >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch, > HIVE-22469.3.patch > > > Hive Metastore uses the 'HikariCP' database connection pool. The default > number of connections to the database is 10. For the Qtests, connecting to a > local DerbyDB, there need not be more than 4 connections. Anymore simply adds > undo overhead and by looking at the QTest logs, I see a bunch of 'connection > refused' from HikariCP. It may be the case that the standalone DB does not > support that many concurrent connections anyway. > [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore] > I also unified all of the JDBC connection strings to ensure that each test is > using an in-memory database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[ https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971369#comment-16971369 ] Miklos Gergely commented on HIVE-22469: --- +1 > Lower Metastore DB Connection Pool Size in QTests > - > > Key: HIVE-22469 > URL: https://issues.apache.org/jira/browse/HIVE-22469 > Project: Hive > Issue Type: Improvement > Components: Test, Tests >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch, > HIVE-22469.3.patch > > > Hive Metastore uses the 'HikariCP' database connection pool. The default > number of connections to the database is 10. For the Qtests, connecting to a > local DerbyDB, there need not be more than 4 connections. Anymore simply adds > undo overhead and by looking at the QTest logs, I see a bunch of 'connection > refused' from HikariCP. It may be the case that the standalone DB does not > support that many concurrent connections anyway. > [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore] > I also unified all of the JDBC connection strings to ensure that each test is > using an in-memory database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[ https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970860#comment-16970860 ] David Mollitor commented on HIVE-22469: --- [~mgergely] Can you also take a look at this one please? :) > Lower Metastore DB Connection Pool Size in QTests > - > > Key: HIVE-22469 > URL: https://issues.apache.org/jira/browse/HIVE-22469 > Project: Hive > Issue Type: Improvement > Components: Test, Tests >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch, > HIVE-22469.3.patch > > > Hive Metastore uses the 'HikariCP' database connection pool. The default > number of connections to the database is 10. For the Qtests, connecting to a > local DerbyDB, there need not be more than 4 connections. Anymore simply adds > undo overhead and by looking at the QTest logs, I see a bunch of 'connection > refused' from HikariCP. It may be the case that the standalone DB does not > support that many concurrent connections anyway. > [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore] > I also unified all of the JDBC connection strings to ensure that each test is > using an in-memory database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[
https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970700#comment-16970700
]
Hive QA commented on HIVE-22469:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985381/HIVE-22469.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17668 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19360/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19360/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19360/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985381 - PreCommit-HIVE-Build
> Lower Metastore DB Connection Pool Size in QTests
> -
>
> Key: HIVE-22469
> URL: https://issues.apache.org/jira/browse/HIVE-22469
> Project: Hive
> Issue Type: Improvement
> Components: Test, Tests
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch,
> HIVE-22469.3.patch
>
>
> Hive Metastore uses the 'HikariCP' database connection pool. The default
> number of connections to the database is 10. For the Qtests, connecting to a
> local DerbyDB, there need not be more than 4 connections. Anymore simply adds
> undo overhead and by looking at the QTest logs, I see a bunch of 'connection
> refused' from HikariCP. It may be the case that the standalone DB does not
> support that many concurrent connections anyway.
> [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[
https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970688#comment-16970688
]
Hive QA commented on HIVE-22469:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
48s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 2m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19360/dev-support/hive-personality.sh
|
| git revision | master / 4d4ab36 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19360/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Lower Metastore DB Connection Pool Size in QTests
> -
>
> Key: HIVE-22469
> URL: https://issues.apache.org/jira/browse/HIVE-22469
> Project: Hive
> Issue Type: Improvement
> Components: Test, Tests
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch,
> HIVE-22469.3.patch
>
>
> Hive Metastore uses the 'HikariCP' database connection pool. The default
> number of connections to the database is 10. For the Qtests, connecting to a
> local DerbyDB, there need not be more than 4 connections. Anymore simply adds
> undo overhead and by looking at the QTest logs, I see a bunch of 'connection
> refused' from HikariCP. It may be the case that the standalone DB does not
> support that many concurrent connections anyway.
> [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[
https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970509#comment-16970509
]
Hive QA commented on HIVE-22469:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985351/HIVE-22469.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 17669 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=275)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=276)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths
(batchId=276)
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSessionSparkSessionTimeout
(batchId=277)
org.apache.hive.hcatalog.streaming.TestStreaming.testConcurrentTransactionBatchCommits
(batchId=220)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=301)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=301)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=301)
org.apache.hive.service.server.TestInformationSchemaWithPrivilege.test
(batchId=283)
org.apache.hive.streaming.TestStreaming.testConcurrentTransactionBatchCommits
(batchId=362)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19354/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19354/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19354/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985351 - PreCommit-HIVE-Build
> Lower Metastore DB Connection Pool Size in QTests
> -
>
> Key: HIVE-22469
> URL: https://issues.apache.org/jira/browse/HIVE-22469
> Project: Hive
> Issue Type: Improvement
> Components: Test, Tests
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch,
> HIVE-22469.3.patch
>
>
> Hive Metastore uses the 'HikariCP' database connection pool. The default
> number of connections to the database is 10. For the Qtests, connecting to a
> local DerbyDB, there need not be more than 4 connections. Anymore simply adds
> undo overhead and by looking at the QTest logs, I see a bunch of 'connection
> refused' from HikariCP. It may be the case that the standalone DB does not
> support that many concurrent connections anyway.
> [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[
https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970419#comment-16970419
]
Hive QA commented on HIVE-22469:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
42s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 2m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19354/dev-support/hive-personality.sh
|
| git revision | master / 3b4c2fe |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19354/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Lower Metastore DB Connection Pool Size in QTests
> -
>
> Key: HIVE-22469
> URL: https://issues.apache.org/jira/browse/HIVE-22469
> Project: Hive
> Issue Type: Improvement
> Components: Test, Tests
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch,
> HIVE-22469.2.patch
>
>
> Hive Metastore uses the 'HikariCP' database connection pool. The default
> number of connections to the database is 10. For the Qtests, connecting to a
> local DerbyDB, there need not be more than 2 connections. Anymore simply adds
> undo overhead and by looking at the QTest logs, I see a bunch of 'connection
> refused' from HikariCP. It may be the case that the standalone DB does not
> support that many concurrent connections anyway.
> [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[
https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970073#comment-16970073
]
Hive QA commented on HIVE-22469:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985246/HIVE-22469.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19347/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19347/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19347/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12985246/HIVE-22469.2.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985246 - PreCommit-HIVE-Build
> Lower Metastore DB Connection Pool Size in QTests
> -
>
> Key: HIVE-22469
> URL: https://issues.apache.org/jira/browse/HIVE-22469
> Project: Hive
> Issue Type: Improvement
> Components: Test, Tests
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch
>
>
> Hive Metastore uses the 'HikariCP' database connection pool. The default
> number of connections to the database is 10. For the Qtests, connecting to a
> local DerbyDB, there need not be more than 2 connections. Anymore simply adds
> undo overhead and by looking at the QTest logs, I see a bunch of 'connection
> refused' from HikariCP. It may be the case that the standalone DB does not
> support that many concurrent connections anyway.
> [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[
https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969980#comment-16969980
]
Hive QA commented on HIVE-22469:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985246/HIVE-22469.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17665 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=111)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=364)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=275)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths
(batchId=275)
org.apache.hive.hcatalog.streaming.TestStreaming.testConcurrentTransactionBatchCommits
(batchId=219)
org.apache.hive.streaming.TestStreaming.testConcurrentTransactionBatchCommits
(batchId=361)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19345/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19345/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19345/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985246 - PreCommit-HIVE-Build
> Lower Metastore DB Connection Pool Size in QTests
> -
>
> Key: HIVE-22469
> URL: https://issues.apache.org/jira/browse/HIVE-22469
> Project: Hive
> Issue Type: Improvement
> Components: Test, Tests
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch
>
>
> Hive Metastore uses the 'HikariCP' database connection pool. The default
> number of connections to the database is 10. For the Qtests, connecting to a
> local DerbyDB, there need not be more than 2 connections. Anymore simply adds
> undo overhead and by looking at the QTest logs, I see a bunch of 'connection
> refused' from HikariCP. It may be the case that the standalone DB does not
> support that many concurrent connections anyway.
> [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[
https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969931#comment-16969931
]
Hive QA commented on HIVE-22469:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
38s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 2m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19345/dev-support/hive-personality.sh
|
| git revision | master / e7d2cd2 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19345/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Lower Metastore DB Connection Pool Size in QTests
> -
>
> Key: HIVE-22469
> URL: https://issues.apache.org/jira/browse/HIVE-22469
> Project: Hive
> Issue Type: Improvement
> Components: Test, Tests
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch
>
>
> Hive Metastore uses the 'HikariCP' database connection pool. The default
> number of connections to the database is 10. For the Qtests, connecting to a
> local DerbyDB, there need not be more than 2 connections. Anymore simply adds
> undo overhead and by looking at the QTest logs, I see a bunch of 'connection
> refused' from HikariCP. It may be the case that the standalone DB does not
> support that many concurrent connections anyway.
> [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[
https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969533#comment-16969533
]
David Mollitor commented on HIVE-22469:
---
Managing 10 connections, bunch of extra noise in the logs.
{code}
2019-11-06T11:45:30,741 DEBUG [HikariPool-3 housekeeper] pool.HikariPool:
HikariPool-3 - Pool stats (total=10, active=0, idle=10, waiting=0)
2019-11-06T11:45:30,741 DEBUG [HikariPool-2 housekeeper] pool.HikariPool:
HikariPool-2 - Pool stats (total=10, active=1, idle=9, waiting=0)
2019-11-06T11:45:40,809 DEBUG [HikariPool-1 connection closer] pool.PoolBase:
HikariPool-1 - Closing connection
org.apache.derby.impl.jdbc.EmbedConnection@442851558 (XID = 184), (SESSIONID =
19), (DATABASE =
/home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db),
(DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:40,812 DEBUG [HikariPool-1 connection adder] pool.HikariPool:
HikariPool-1 - Added connection
org.apache.derby.impl.jdbc.EmbedConnection@2095252197 (XID = 17066), (SESSIONID
= 3), (DATABASE =
/home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db),
(DRDAID = null)
2019-11-06T11:45:49,020 DEBUG [HikariPool-1 connection closer] pool.PoolBase:
HikariPool-1 - Closing connection
org.apache.derby.impl.jdbc.EmbedConnection@416110958 (XID = 174), (SESSIONID =
9), (DATABASE =
/home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db),
(DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:49,022 DEBUG [HikariPool-1 connection adder] pool.HikariPool:
HikariPool-1 - Added connection
org.apache.derby.impl.jdbc.EmbedConnection@73417930 (XID = 17068), (SESSIONID =
5), (DATABASE =
/home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db),
(DRDAID = null)
2019-11-06T11:45:49,363 DEBUG [HikariPool-1 connection closer] pool.PoolBase:
HikariPool-1 - Closing connection
org.apache.derby.impl.jdbc.EmbedConnection@1998766497 (XID = 180), (SESSIONID =
15), (DATABASE =
/home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db),
(DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:49,364 DEBUG [HikariPool-1 connection adder] pool.HikariPool:
HikariPool-1 - Added connection
org.apache.derby.impl.jdbc.EmbedConnection@1163742961 (XID = 17070), (SESSIONID
= 7), (DATABASE =
/home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db),
(DRDAID = null)
2019-11-06T11:45:52,762 DEBUG [HikariPool-1 housekeeper] pool.HikariPool:
HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
2019-11-06T11:45:53,260 DEBUG [HikariPool-1 connection closer] pool.PoolBase:
HikariPool-1 - Closing connection
org.apache.derby.impl.jdbc.EmbedConnection@478356754 (XID = 178), (SESSIONID =
13), (DATABASE =
/home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db),
(DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:53,261 DEBUG [HikariPool-1 connection adder] pool.HikariPool:
HikariPool-1 - Added connection
org.apache.derby.impl.jdbc.EmbedConnection@1959963107 (XID = 17072), (SESSIONID
= 9), (DATABASE =
/home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db),
(DRDAID = null)
2019-11-06T11:46:00,298 DEBUG [HikariPool-1 connection closer] pool.PoolBase:
HikariPool-1 - Closing connection
org.apache.derby.impl.jdbc.EmbedConnection@605147393 (XID = 182), (SESSIONID =
17), (DATABASE =
/home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db),
(DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:46:00,299 DEBUG [HikariPool-1 connection adder] pool.HikariPool:
HikariPool-1 - Added connection
org.apache.derby.impl.jdbc.EmbedConnection@245074122 (XID = 17074), (SESSIONID
= 11), (DATABASE =
/home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db),
(DRDAID = null)
2019-11-06T11:46:00,741 DEBUG [HikariPool-3 housekeeper] pool.HikariPool:
HikariPool-3 - Pool stats (total=10, active=0, idle=10, waiting=0)
2019-11-06T11:46:00,741 DEBUG [HikariPool-2 housekeeper] pool.HikariPool:
HikariPool-2 - Pool stats (total=10, active=1, idle=9, waiting=0)
{code}
> Lower Metastore DB Connection Pool Size in QTests
> -
>
> Key: HIVE-22469
> URL: https://issues.apache.org/jira/browse/HIVE-22469
> Project: Hive
> Issue Type: Improvement
> Components: Test, Tests
>Affects Versions: 3.2.0
>
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[
https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969530#comment-16969530
]
David Mollitor commented on HIVE-22469:
---
Example...
{code:none}
2019-11-06T11:23:23,325 DEBUG [Heartbeater-2] txn.TxnHandler: Going to rollback
2019-11-06T11:23:23,325 WARN [Heartbeater-2] pool.ProxyConnection:
HikariPool-2 - Connection org.apache.derby.impl.jdbc.EmbedConnection@1601499507
(XID = 12138), (SESSIONID = 71), (DATABASE =
/home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db),
(DRDAID = null) marked as broken because of SQLSTATE(08003), ErrorCode(4)
java.sql.SQLNonTransientConnectionException: No current connection.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
~[derby-10.14.1.0.jar:?]
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source) ~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source) ~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.Util.noCurrentConnection(Unknown Source)
~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.EmbedConnection.checkIfClosed(Unknown
Source) ~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.EmbedConnection.setupContextStack(Unknown
Source) ~[derby-10.14.1.0.jar:?]
at org.apache.derby.impl.jdbc.EmbedConnection.rollback(Unknown Source)
~[derby-10.14.1.0.jar:?]
at
com.zaxxer.hikari.pool.ProxyConnection.rollback(ProxyConnection.java:362)
~[HikariCP-2.6.1.jar:?]
at
com.zaxxer.hikari.pool.HikariProxyConnection.rollback(HikariProxyConnection.java)
~[HikariCP-2.6.1.jar:?]
{code}
> Lower Metastore DB Connection Pool Size in QTests
> -
>
> Key: HIVE-22469
> URL: https://issues.apache.org/jira/browse/HIVE-22469
> Project: Hive
> Issue Type: Improvement
> Components: Test, Tests
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch
>
>
> Hive Metastore uses the 'HikariCP' database connection pool. The default
> number of connections to the database is 10. For the Qtests, connecting to a
> local DerbyDB, there need not be more than 2 connections. Anymore simply adds
> undo overhead and by looking at the QTest logs, I see a bunch of 'connection
> refused' from HikariCP. It may be the case that the standalone DB does not
> support that many concurrent connections anyway.
> [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
