Re: Updating Wiki about Hikari Configuration Properties
That did the trick! Thank you. I have updated the page. Chris Nauroth On Fri, Oct 28, 2022 at 11:47 PM Denys Kuzmenko wrote: > Hi Chris, > > Previously I just gave you edit permissions for the requested page, but now > granted Hive space permissions. Please check again and let me know. > > On Fri, Oct 28, 2022 at 7:51 PM Chris Nauroth wrote: > > > Thank you, Denys. Unfortunately, I still don't see the Edit button like I > > do on other projects where I have access (Hadoop, Zookeeper). I just > > realized my Confluence login is actually just "cnauroth", so maybe I > caused > > some confusion by stating it as "cnaur...@apache.org"? > > > > Ayush, to further clarify, yes, I am planning to backport HIVE-17317 onto > > versions 3.1.2 and 3.1.3 of Dataproc's Hive distro. I've already tested > > locally and confirmed that I can use hikaricp.minimumIdle to limit the > > number of connections. I'd also be happy to send an upstream pull request > > against branch-3.1 for the backport if the community would find that > > valuable. Let me know. > > > > When I make the Confluence edit, I'll clarify which versions have this > > support. > > > > Chris Nauroth > > > > > > On Fri, Oct 28, 2022 at 4:37 AM Ayush Saxena wrote: > > > > > Hi Chris, > > > > > > Just curious: > > >> > > >> BTW, the reason I discovered this is that I recently upgraded a > cluster > > >> from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). > > > > > > > > > As you said the prefix change happened in HIVE-17317 which is in > > > 4.0.0-alpha-1 [1] and you migrated to Hive 3.x, Then how did you face > > > this problem? Some patched version of Hive, or some missing FixVersion > in > > > the Jira? > > > > > > Good to mention the version post which the change in prefix happens in > > the > > > wiki as well, to avoid future confusions due to versions. > > > > > > -Ayush > > > > > > [1] https://issues.apache.org/jira/browse/HIVE-17317 > > > > > > On Fri, 28 Oct 2022 at 17:02, Denys Kuzmenko > > > wrote: > > > > > >> Hi Chris, > > >> > > >> Please try now > > >> > > >> On Fri, Oct 28, 2022 at 12:43 PM Stamatis Zampetakis < > zabe...@gmail.com > > > > > >> wrote: > > >> > > >> > Hive PMC members can provide edit rights to the wiki. > > >> > > > >> > @Naveen, Dennys, Adam: Can someone please give write privileges to > > >> Chris? > > >> > > > >> > Best, > > >> > Stamatis > > >> > > > >> > On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth > > >> wrote: > > >> > > > >> > > Hi everyone, > > >> > > > > >> > > Regarding this page: > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration > > >> > > > > >> > > It states that the metastore's Hikari connection pool can be > > >> configured > > >> > by > > >> > > specifying properties prefixed as "hikari". This is not quite > > >> correct. In > > >> > > HIVE-17317, there was a bug fix made to the Hikari integration > such > > >> that > > >> > > the proper prefix is "hikaricp". For example: > > >> > > > > >> > > > > >> > > hikaricp.minimumIdle > > >> > > 4 > > >> > > false > > >> > > Dataproc Cluster Properties > > >> > > > > >> > > > > >> > > Could you please grant access to me (cnaur...@apache.org) to > update > > >> the > > >> > > page? If you prefer not to grant access, could a Hive committer > make > > >> the > > >> > > change for me? > > >> > > > > >> > > BTW, the reason I discovered this is that I recently upgraded a > > >> cluster > > >> > > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). > After > > >> the > > >> > > upgrade, I found that HiveMetaStore was generating far more > database > > >> > > connections at baseline, putting extra burden on the database. It > > >> appears > > >> > > that BoneCP default behavior (4 idle connections) is different > from > > >> > > HikariCP default behavior (idle connections equal to max > connections > > >> > which > > >> > > is 10). This put me down the path of wanting to control Hikari's > > >> > > minimumIdle setting and then finding this discrepancy in the > > >> > documentation. > > >> > > > > >> > > Passing on this information in case others are seeing unusually > high > > >> > > connection counts after an upgrade to 3.x. > > >> > > > > >> > > Chris Nauroth > > >> > > > > >> > > > >> > > > > > >
Re: Updating Wiki about Hikari Configuration Properties
Hi Chris, Previously I just gave you edit permissions for the requested page, but now granted Hive space permissions. Please check again and let me know. On Fri, Oct 28, 2022 at 7:51 PM Chris Nauroth wrote: > Thank you, Denys. Unfortunately, I still don't see the Edit button like I > do on other projects where I have access (Hadoop, Zookeeper). I just > realized my Confluence login is actually just "cnauroth", so maybe I caused > some confusion by stating it as "cnaur...@apache.org"? > > Ayush, to further clarify, yes, I am planning to backport HIVE-17317 onto > versions 3.1.2 and 3.1.3 of Dataproc's Hive distro. I've already tested > locally and confirmed that I can use hikaricp.minimumIdle to limit the > number of connections. I'd also be happy to send an upstream pull request > against branch-3.1 for the backport if the community would find that > valuable. Let me know. > > When I make the Confluence edit, I'll clarify which versions have this > support. > > Chris Nauroth > > > On Fri, Oct 28, 2022 at 4:37 AM Ayush Saxena wrote: > > > Hi Chris, > > > > Just curious: > >> > >> BTW, the reason I discovered this is that I recently upgraded a cluster > >> from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). > > > > > > As you said the prefix change happened in HIVE-17317 which is in > > 4.0.0-alpha-1 [1] and you migrated to Hive 3.x, Then how did you face > > this problem? Some patched version of Hive, or some missing FixVersion in > > the Jira? > > > > Good to mention the version post which the change in prefix happens in > the > > wiki as well, to avoid future confusions due to versions. > > > > -Ayush > > > > [1] https://issues.apache.org/jira/browse/HIVE-17317 > > > > On Fri, 28 Oct 2022 at 17:02, Denys Kuzmenko > > wrote: > > > >> Hi Chris, > >> > >> Please try now > >> > >> On Fri, Oct 28, 2022 at 12:43 PM Stamatis Zampetakis > > >> wrote: > >> > >> > Hive PMC members can provide edit rights to the wiki. > >> > > >> > @Naveen, Dennys, Adam: Can someone please give write privileges to > >> Chris? > >> > > >> > Best, > >> > Stamatis > >> > > >> > On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth > >> wrote: > >> > > >> > > Hi everyone, > >> > > > >> > > Regarding this page: > >> > > > >> > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration > >> > > > >> > > It states that the metastore's Hikari connection pool can be > >> configured > >> > by > >> > > specifying properties prefixed as "hikari". This is not quite > >> correct. In > >> > > HIVE-17317, there was a bug fix made to the Hikari integration such > >> that > >> > > the proper prefix is "hikaricp". For example: > >> > > > >> > > > >> > > hikaricp.minimumIdle > >> > > 4 > >> > > false > >> > > Dataproc Cluster Properties > >> > > > >> > > > >> > > Could you please grant access to me (cnaur...@apache.org) to update > >> the > >> > > page? If you prefer not to grant access, could a Hive committer make > >> the > >> > > change for me? > >> > > > >> > > BTW, the reason I discovered this is that I recently upgraded a > >> cluster > >> > > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). After > >> the > >> > > upgrade, I found that HiveMetaStore was generating far more database > >> > > connections at baseline, putting extra burden on the database. It > >> appears > >> > > that BoneCP default behavior (4 idle connections) is different from > >> > > HikariCP default behavior (idle connections equal to max connections > >> > which > >> > > is 10). This put me down the path of wanting to control Hikari's > >> > > minimumIdle setting and then finding this discrepancy in the > >> > documentation. > >> > > > >> > > Passing on this information in case others are seeing unusually high > >> > > connection counts after an upgrade to 3.x. > >> > > > >> > > Chris Nauroth > >> > > > >> > > >> > > >
Re: Updating Wiki about Hikari Configuration Properties
Thank you, Denys. Unfortunately, I still don't see the Edit button like I do on other projects where I have access (Hadoop, Zookeeper). I just realized my Confluence login is actually just "cnauroth", so maybe I caused some confusion by stating it as "cnaur...@apache.org"? Ayush, to further clarify, yes, I am planning to backport HIVE-17317 onto versions 3.1.2 and 3.1.3 of Dataproc's Hive distro. I've already tested locally and confirmed that I can use hikaricp.minimumIdle to limit the number of connections. I'd also be happy to send an upstream pull request against branch-3.1 for the backport if the community would find that valuable. Let me know. When I make the Confluence edit, I'll clarify which versions have this support. Chris Nauroth On Fri, Oct 28, 2022 at 4:37 AM Ayush Saxena wrote: > Hi Chris, > > Just curious: >> >> BTW, the reason I discovered this is that I recently upgraded a cluster >> from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). > > > As you said the prefix change happened in HIVE-17317 which is in > 4.0.0-alpha-1 [1] and you migrated to Hive 3.x, Then how did you face > this problem? Some patched version of Hive, or some missing FixVersion in > the Jira? > > Good to mention the version post which the change in prefix happens in the > wiki as well, to avoid future confusions due to versions. > > -Ayush > > [1] https://issues.apache.org/jira/browse/HIVE-17317 > > On Fri, 28 Oct 2022 at 17:02, Denys Kuzmenko > wrote: > >> Hi Chris, >> >> Please try now >> >> On Fri, Oct 28, 2022 at 12:43 PM Stamatis Zampetakis >> wrote: >> >> > Hive PMC members can provide edit rights to the wiki. >> > >> > @Naveen, Dennys, Adam: Can someone please give write privileges to >> Chris? >> > >> > Best, >> > Stamatis >> > >> > On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth >> wrote: >> > >> > > Hi everyone, >> > > >> > > Regarding this page: >> > > >> > > >> > > >> > >> https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration >> > > >> > > It states that the metastore's Hikari connection pool can be >> configured >> > by >> > > specifying properties prefixed as "hikari". This is not quite >> correct. In >> > > HIVE-17317, there was a bug fix made to the Hikari integration such >> that >> > > the proper prefix is "hikaricp". For example: >> > > >> > > >> > > hikaricp.minimumIdle >> > > 4 >> > > false >> > > Dataproc Cluster Properties >> > > >> > > >> > > Could you please grant access to me (cnaur...@apache.org) to update >> the >> > > page? If you prefer not to grant access, could a Hive committer make >> the >> > > change for me? >> > > >> > > BTW, the reason I discovered this is that I recently upgraded a >> cluster >> > > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). After >> the >> > > upgrade, I found that HiveMetaStore was generating far more database >> > > connections at baseline, putting extra burden on the database. It >> appears >> > > that BoneCP default behavior (4 idle connections) is different from >> > > HikariCP default behavior (idle connections equal to max connections >> > which >> > > is 10). This put me down the path of wanting to control Hikari's >> > > minimumIdle setting and then finding this discrepancy in the >> > documentation. >> > > >> > > Passing on this information in case others are seeing unusually high >> > > connection counts after an upgrade to 3.x. >> > > >> > > Chris Nauroth >> > > >> > >> >
Re: Updating Wiki about Hikari Configuration Properties
Hi Chris, Just curious: > > BTW, the reason I discovered this is that I recently upgraded a cluster > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). As you said the prefix change happened in HIVE-17317 which is in 4.0.0-alpha-1 [1] and you migrated to Hive 3.x, Then how did you face this problem? Some patched version of Hive, or some missing FixVersion in the Jira? Good to mention the version post which the change in prefix happens in the wiki as well, to avoid future confusions due to versions. -Ayush [1] https://issues.apache.org/jira/browse/HIVE-17317 On Fri, 28 Oct 2022 at 17:02, Denys Kuzmenko wrote: > Hi Chris, > > Please try now > > On Fri, Oct 28, 2022 at 12:43 PM Stamatis Zampetakis > wrote: > > > Hive PMC members can provide edit rights to the wiki. > > > > @Naveen, Dennys, Adam: Can someone please give write privileges to Chris? > > > > Best, > > Stamatis > > > > On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth > wrote: > > > > > Hi everyone, > > > > > > Regarding this page: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration > > > > > > It states that the metastore's Hikari connection pool can be configured > > by > > > specifying properties prefixed as "hikari". This is not quite correct. > In > > > HIVE-17317, there was a bug fix made to the Hikari integration such > that > > > the proper prefix is "hikaricp". For example: > > > > > > > > > hikaricp.minimumIdle > > > 4 > > > false > > > Dataproc Cluster Properties > > > > > > > > > Could you please grant access to me (cnaur...@apache.org) to update > the > > > page? If you prefer not to grant access, could a Hive committer make > the > > > change for me? > > > > > > BTW, the reason I discovered this is that I recently upgraded a cluster > > > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). After > the > > > upgrade, I found that HiveMetaStore was generating far more database > > > connections at baseline, putting extra burden on the database. It > appears > > > that BoneCP default behavior (4 idle connections) is different from > > > HikariCP default behavior (idle connections equal to max connections > > which > > > is 10). This put me down the path of wanting to control Hikari's > > > minimumIdle setting and then finding this discrepancy in the > > documentation. > > > > > > Passing on this information in case others are seeing unusually high > > > connection counts after an upgrade to 3.x. > > > > > > Chris Nauroth > > > > > >
Re: Updating Wiki about Hikari Configuration Properties
Hi Chris, Please try now On Fri, Oct 28, 2022 at 12:43 PM Stamatis Zampetakis wrote: > Hive PMC members can provide edit rights to the wiki. > > @Naveen, Dennys, Adam: Can someone please give write privileges to Chris? > > Best, > Stamatis > > On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth wrote: > > > Hi everyone, > > > > Regarding this page: > > > > > > > https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration > > > > It states that the metastore's Hikari connection pool can be configured > by > > specifying properties prefixed as "hikari". This is not quite correct. In > > HIVE-17317, there was a bug fix made to the Hikari integration such that > > the proper prefix is "hikaricp". For example: > > > > > > hikaricp.minimumIdle > > 4 > > false > > Dataproc Cluster Properties > > > > > > Could you please grant access to me (cnaur...@apache.org) to update the > > page? If you prefer not to grant access, could a Hive committer make the > > change for me? > > > > BTW, the reason I discovered this is that I recently upgraded a cluster > > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). After the > > upgrade, I found that HiveMetaStore was generating far more database > > connections at baseline, putting extra burden on the database. It appears > > that BoneCP default behavior (4 idle connections) is different from > > HikariCP default behavior (idle connections equal to max connections > which > > is 10). This put me down the path of wanting to control Hikari's > > minimumIdle setting and then finding this discrepancy in the > documentation. > > > > Passing on this information in case others are seeing unusually high > > connection counts after an upgrade to 3.x. > > > > Chris Nauroth > > >
Re: Updating Wiki about Hikari Configuration Properties
Hive PMC members can provide edit rights to the wiki. @Naveen, Dennys, Adam: Can someone please give write privileges to Chris? Best, Stamatis On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth wrote: > Hi everyone, > > Regarding this page: > > > https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration > > It states that the metastore's Hikari connection pool can be configured by > specifying properties prefixed as "hikari". This is not quite correct. In > HIVE-17317, there was a bug fix made to the Hikari integration such that > the proper prefix is "hikaricp". For example: > > > hikaricp.minimumIdle > 4 > false > Dataproc Cluster Properties > > > Could you please grant access to me (cnaur...@apache.org) to update the > page? If you prefer not to grant access, could a Hive committer make the > change for me? > > BTW, the reason I discovered this is that I recently upgraded a cluster > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). After the > upgrade, I found that HiveMetaStore was generating far more database > connections at baseline, putting extra burden on the database. It appears > that BoneCP default behavior (4 idle connections) is different from > HikariCP default behavior (idle connections equal to max connections which > is 10). This put me down the path of wanting to control Hikari's > minimumIdle setting and then finding this discrepancy in the documentation. > > Passing on this information in case others are seeing unusually high > connection counts after an upgrade to 3.x. > > Chris Nauroth >