[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283177#comment-14283177 ] Brock Noland commented on HIVE-9119: FYI this [broke TestMTQueries|https://issues.apache.org/jira/browse/HIVE-1869?focusedCommentId=14283175page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14283175]. Fixing in an old jira HIVE-1869 for that class. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Labels: TODOC15 Fix For: 0.15.0 Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch, HIVE-9119.4.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274870#comment-14274870 ] Lefty Leverenz commented on HIVE-9119: -- Doc note: This adds two configuration parameters and modifies time units for a third in HiveConf.java. They should be documented in the Locking section of Configuration Properties. * new: *hive.zookeeper.connection.max.retries* * new: *hive.zookeeper.connection.basesleeptime* * modified: *hive.zookeeper.session.timeout* * doc here: [Configuration Properties -- Locking | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Locking] ** [hive.zookeeper.session.timeout | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.zookeeper.session.timeout] ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Labels: TODOC15 Fix For: 0.15.0 Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch, HIVE-9119.4.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273824#comment-14273824 ] Na Yang commented on HIVE-9119: --- Thank you [~xuefuz] for helping review and submit the patch! Best Regards, Na ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Labels: TODOC15 Fix For: 0.15.0 Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch, HIVE-9119.4.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272806#comment-14272806 ] Xuefu Zhang commented on HIVE-9119: --- +1 patch looks good to me. Thanks for fixing this long haunting issue, and now the zookeeper related code is much cleaner also. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch, HIVE-9119.4.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1428#comment-1428 ] Na Yang commented on HIVE-9119: --- Those 3 tests all passed successfully on my local environment with the HIVE-9114.patch applied. Do not know why it is failing on the build machine. The error logs do not seem to be related to the ZooKeeperHiveLockManager changes in the patch. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch, HIVE-9119.4.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[
https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265091#comment-14265091
]
Hive QA commented on HIVE-9119:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690109/HIVE-9119.4.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6721 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2257/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2257/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2257/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690109 - PreCommit-HIVE-TRUNK-Build
ZooKeeperHiveLockManager does not use zookeeper in the proper way
-
Key: HIVE-9119
URL: https://issues.apache.org/jira/browse/HIVE-9119
Project: Hive
Issue Type: Improvement
Components: Locking
Affects Versions: 0.13.0, 0.14.0, 0.13.1
Reporter: Na Yang
Assignee: Na Yang
Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch,
HIVE-9119.4.patch
ZooKeeperHiveLockManager does not use zookeeper in the proper way.
Currently a new zookeeper client instance is created for each
getlock/releaselock query which sometimes causes the number of open
connections between
HiveServer2 and ZooKeeper exceed the max connection number that zookeeper
server allows.
To use zookeeper as a distributed lock, there is no need to create a new
zookeeper instance for every getlock try. A single zookeeper instance could
be reused and shared by ZooKeeperHiveLockManagers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262040#comment-14262040 ] Lefty Leverenz commented on HIVE-9119: -- +1 for the configuration parameter definitions. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260937#comment-14260937 ] Lefty Leverenz commented on HIVE-9119: -- Thanks for the changes, [~nyang]. One new question: When you changed the default of *hive.zookeeper.session.timeout* to use a TimeValidator, did an extra zero slip into the value or was the original not in milliseconds? (600*1000 - 600ms.) Also, you might want to split the description of *hive.zookeeper.connection.basesleeptime* into two lines. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261561#comment-14261561 ] Na Yang commented on HIVE-9119: --- [~leftylev], thank you for reviewing the patch. I uploaded a new patch according to your suggestion and also changed the hive.zookeeper.session.timeout to 60ms. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[
https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261802#comment-14261802
]
Hive QA commented on HIVE-9119:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689573/HIVE-9119.3.patch
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 6700 tests executed
*Failed tests:*
{noformat}
TestHBaseCliDriver-hbase_binary_map_queries_prefix.q-hbase_stats.q-ppd_key_ranges.q
- did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_handler_bulk.q-hbase_scan_params.q-hbase_single_sourced_multi_insert.q
- did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_handler_snapshot.q-hbase_stats3.q-hbase_custom_key3.q
- did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_ppd_key_range.q-hbase_custom_key.q-hbase_pushdown.q -
did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_stats2.q-hbase_custom_key2.q-hbase_timestamp.q - did
not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_stats_empty_partition.q-external_table_ppd.q-hbase_ppd_join.q
- did not produce a TEST-*.xml file
TestHBaseNegativeCliDriver - did not produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build//testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build//console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689573 - PreCommit-HIVE-TRUNK-Build
ZooKeeperHiveLockManager does not use zookeeper in the proper way
-
Key: HIVE-9119
URL: https://issues.apache.org/jira/browse/HIVE-9119
Project: Hive
Issue Type: Improvement
Components: Locking
Affects Versions: 0.13.0, 0.14.0, 0.13.1
Reporter: Na Yang
Assignee: Na Yang
Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch
ZooKeeperHiveLockManager does not use zookeeper in the proper way.
Currently a new zookeeper client instance is created for each
getlock/releaselock query which sometimes causes the number of open
connections between
HiveServer2 and ZooKeeper exceed the max connection number that zookeeper
server allows.
To use zookeeper as a distributed lock, there is no need to create a new
zookeeper instance for every getlock try. A single zookeeper instance could
be reused and shared by ZooKeeperHiveLockManagers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[
https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260673#comment-14260673
]
Hive QA commented on HIVE-9119:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689389/HIVE-9119.2.patch
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 6700 tests
executed
*Failed tests:*
{noformat}
TestHBaseCliDriver-hbase_binary_map_queries_prefix.q-hbase_stats.q-ppd_key_ranges.q
- did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_handler_bulk.q-hbase_scan_params.q-hbase_single_sourced_multi_insert.q
- did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_handler_snapshot.q-hbase_stats3.q-hbase_custom_key3.q
- did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_ppd_key_range.q-hbase_custom_key.q-hbase_pushdown.q -
did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_stats2.q-hbase_custom_key2.q-hbase_timestamp.q - did
not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_stats_empty_partition.q-external_table_ppd.q-hbase_ppd_join.q
- did not produce a TEST-*.xml file
TestHBaseNegativeCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2214/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2214/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2214/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689389 - PreCommit-HIVE-TRUNK-Build
ZooKeeperHiveLockManager does not use zookeeper in the proper way
-
Key: HIVE-9119
URL: https://issues.apache.org/jira/browse/HIVE-9119
Project: Hive
Issue Type: Improvement
Components: Locking
Affects Versions: 0.13.0, 0.14.0, 0.13.1
Reporter: Na Yang
Assignee: Na Yang
Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch
ZooKeeperHiveLockManager does not use zookeeper in the proper way.
Currently a new zookeeper client instance is created for each
getlock/releaselock query which sometimes causes the number of open
connections between
HiveServer2 and ZooKeeper exceed the max connection number that zookeeper
server allows.
To use zookeeper as a distributed lock, there is no need to create a new
zookeeper instance for every getlock try. A single zookeeper instance could
be reused and shared by ZooKeeperHiveLockManagers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258873#comment-14258873 ] Na Yang commented on HIVE-9119: --- [~leftylev], thank you very much for reviewing the patch. I will take your suggestions. Regarding to your questions, please see my answer below: 2.What are the units for hive.zookeeper.connection.basesleeptime? A TimeValidator could be used here – see comment on HIVE-6679 for an example. - [Na]: The unit is millisecond. I will follow the example when I upload a new patch. 3. Is the omission of an E for ZOOKEEPR deliberate in HIVE_ZOOKEEPR_CONNECTION_BASESLEEPTIME? It occurs once later in the code, also without the E. -[Na]: It is a typo. I wil correct it in the new patch. 4. Just curious: What's initial about the basesleeptime? -[Na]: CuratorFramework uses ExponentialBackoffRetryPolicy to reconnect to the ZooKeeper server. This retry policy retries a set number of times with increasing sleep time between retries. The basesleeptime is the sleep time for the first retry. I will explain it more clearly in the new patch. Currently, the qtests do not run properly with the CuratorFramework change. So I need to work on that and upload a new patch with these doc changes later on. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[
https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256775#comment-14256775
]
Lefty Leverenz commented on HIVE-9119:
--
Doc review of patch 1 (config params): Looks good overall, just a few quibbles
and questions.
{code}
+
HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES(hive.zookeeper.connection.max.retries,
3,
+Max number of times to retry when connecting to the zookeeper
server.),
+
HIVE_ZOOKEEPR_CONNECTION_BASESLEEPTIME(hive.zookeeper.connection.basesleeptime,
1000,
+Initial amount of time to wait between retries when connecting to the
zookeeper server.),
{code}
# Please use camel case for ZooKeeper server in both descriptions.
# What are the units for *hive.zookeeper.connection.basesleeptime*? A
TimeValidator could be used here -- see comment on HIVE-6679 for an example.
# Is the omission of an E for ZOOKEEPR deliberate in
HIVE_ZOOKEEPR_CONNECTION_BASESLEEPTIME? It occurs once later in the code, also
without the E.
# Just curious: What's initial about the basesleeptime?
Reference:
* [ZooKeeper website uses camel case | http://zookeeper.apache.org]
* [HIVE-6679 comment -- TimeValidator example |
https://issues.apache.org/jira/browse/HIVE-6679?focusedCommentId=14248013page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14248013]
* Another example of TimeValidator, with milliseconds:
{code}
HIVE_LOCALIZE_RESOURCE_WAIT_INTERVAL(hive.localize.resource.wait.interval,
5000ms,
new TimeValidator(TimeUnit.MILLISECONDS),
{code}
ZooKeeperHiveLockManager does not use zookeeper in the proper way
-
Key: HIVE-9119
URL: https://issues.apache.org/jira/browse/HIVE-9119
Project: Hive
Issue Type: Improvement
Components: Locking
Affects Versions: 0.13.0, 0.14.0, 0.13.1
Reporter: Na Yang
Assignee: Na Yang
Attachments: HIVE-9119.1.patch
ZooKeeperHiveLockManager does not use zookeeper in the proper way.
Currently a new zookeeper client instance is created for each
getlock/releaselock query which sometimes causes the number of open
connections between
HiveServer2 and ZooKeeper exceed the max connection number that zookeeper
server allows.
To use zookeeper as a distributed lock, there is no need to create a new
zookeeper instance for every getlock try. A single zookeeper instance could
be reused and shared by ZooKeeperHiveLockManagers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[
https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256192#comment-14256192
]
Hive QA commented on HIVE-9119:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12688616/HIVE-9119.1.patch
{color:red}ERROR:{color} -1 due to 155 failed/errored test(s), 4134 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-alter_char2.q-insert_compressed.q-load_dyn_part15.q-and-12-more -
did not produce a TEST-*.xml file
TestCliDriver-alter_table_not_sorted.q-authorization_update.q-dynamic_partition_pruning.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-authorization_1_sql_std.q-drop_index_removes_partition_dirs.q-udf_date_sub.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_createtab.q-groupby_ppr.q-limit_pushdown_negative.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_nonsql.q-rcfile_merge1.q-stats2.q-and-12-more -
did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_stdconfigauth.q-drop_partitions_filter2.q-bucketmapjoin1.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-authorization_create_table_owner_privs.q-create_func1.q-partition_wise_fileformat.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-auto_join18_multi_distinct.q-udf_repeat.q-list_bucket_query_multiskew_2.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-auto_join30.q-join_reorder3.q-lock1.q-and-12-more - did not
produce a TEST-*.xml file
TestCliDriver-auto_join4.q-mapjoin_decimal.q-input_dynamicserde.q-and-12-more -
did not produce a TEST-*.xml file
TestCliDriver-auto_join9.q-acid_vectorization.q-udf_floor.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-auto_sortmerge_join_13.q-alter_partition_clusterby_sortby.q-rcfile_merge2.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-autogen_colalias.q-tez_joins_explain.q-unicode_notation.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-avro_add_column.q-orc_wide_table.q-query_with_semi.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-avro_joins.q-disallow_incompatible_type_change_off.q-udf_max.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-bool_literal.q-udf_hash.q-groupby4_map.q-and-12-more - did not
produce a TEST-*.xml file
TestCliDriver-bucketcontext_5.q-udf_to_float.q-udf_locate.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-bucketmapjoin3.q-udf_round_3.q-udf_between.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-columnstats_partlvl_dp.q-smb_mapjoin_21.q-insert_overwrite_local_directory_1.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-columnstats_tbllvl.q-limit0.q-udf_exp.q-and-12-more - did not
produce a TEST-*.xml file
TestCliDriver-compute_stats_string.q-nullgroup4_multi_distinct.q-udf_concat_ws.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-cp_mj_rc.q-skewjoin_union_remove_2.q-udf_stddev_pop.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-create_genericudf.q-join13.q-notable_alias3.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-create_or_replace_view.q-skewjoinopt8.q-join_cond_pushdown_3.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-date_udf.q-transform_ppr2.q-udf_from_unixtime.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-decimal_6.q-vectorization_limit.q-authorization_index.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-describe_xpath.q-parquet_types.q-partcols1.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-dynpart_sort_optimization2.q-skewjoin_mapjoin4.q-list_bucket_dml_6.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-enforce_order.q-bucketcontext_4.q-stats_publisher_error_1.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-escape_distributeby1.q-create_view_partitioned.q-groupby_sort_7.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-gby_star.q-protectmode.q-udf_regexp_replace.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-groupby10.q-timestamp_comparison.q-tez_union.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-groupby12.q-udf2.q-udf_bigint.q-and-12-more - did not produce a
TEST-*.xml file
TestCliDriver-groupby3_map.q-plan_json.q-correlationoptimizer10.q-and-12-more -
did not produce a TEST-*.xml file
TestCliDriver-groupby4.q-reducesink_dedup.q-update_all_partitioned.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-groupby_complex_types.q-varchar_serde.q-groupby7_map_skew.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-groupby_grouping_sets5.q-desc_tbl_part_cols.q-decimal_2.q-and-12-more
- did not produce a TEST-*.xml file
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255487#comment-14255487 ] Na Yang commented on HIVE-9119: --- According to a ZooKeeper test that I did, increasing the number of ZooKeeper client instances does not help improving the ZooKeeper operation requests performance in a multi-thread environment. The major load is on the ZooKeeper server side. Therefore, a singleton ZooKeeper client is a good choice for the ZookeeperHiveLockManager implementation. A zookeeper client/connection pool is not needed in this case. In addition, if a zookeeper client/connection pool is used and the pool side is set too big by the user, that can cause overhead on zookeeper server and even make the ZooKeeperHiveLockManager not workable if the zookeeper client number exceeds the number that the zookeeper server could handle. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251278#comment-14251278 ] Na Yang commented on HIVE-9119: --- This JIRA and HIVE-8135 are trying to address the same issue. The problem is whether a ZooKeeperClient pool is needed or a singleton client is able to do the work. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
