[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283177#comment-14283177 ] Brock Noland commented on HIVE-9119: FYI this [broke TestMTQueries|https://issues.apache.org/jira/browse/HIVE-1869?focusedCommentId=14283175page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14283175]. Fixing in an old jira HIVE-1869 for that class. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Labels: TODOC15 Fix For: 0.15.0 Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch, HIVE-9119.4.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274870#comment-14274870 ] Lefty Leverenz commented on HIVE-9119: -- Doc note: This adds two configuration parameters and modifies time units for a third in HiveConf.java. They should be documented in the Locking section of Configuration Properties. * new: *hive.zookeeper.connection.max.retries* * new: *hive.zookeeper.connection.basesleeptime* * modified: *hive.zookeeper.session.timeout* * doc here: [Configuration Properties -- Locking | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Locking] ** [hive.zookeeper.session.timeout | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.zookeeper.session.timeout] ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Labels: TODOC15 Fix For: 0.15.0 Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch, HIVE-9119.4.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273824#comment-14273824 ] Na Yang commented on HIVE-9119: --- Thank you [~xuefuz] for helping review and submit the patch! Best Regards, Na ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Labels: TODOC15 Fix For: 0.15.0 Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch, HIVE-9119.4.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272806#comment-14272806 ] Xuefu Zhang commented on HIVE-9119: --- +1 patch looks good to me. Thanks for fixing this long haunting issue, and now the zookeeper related code is much cleaner also. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch, HIVE-9119.4.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1428#comment-1428 ] Na Yang commented on HIVE-9119: --- Those 3 tests all passed successfully on my local environment with the HIVE-9114.patch applied. Do not know why it is failing on the build machine. The error logs do not seem to be related to the ZooKeeperHiveLockManager changes in the patch. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch, HIVE-9119.4.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265091#comment-14265091 ] Hive QA commented on HIVE-9119: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12690109/HIVE-9119.4.patch {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6721 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2257/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2257/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2257/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12690109 - PreCommit-HIVE-TRUNK-Build ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch, HIVE-9119.4.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262040#comment-14262040 ] Lefty Leverenz commented on HIVE-9119: -- +1 for the configuration parameter definitions. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260937#comment-14260937 ] Lefty Leverenz commented on HIVE-9119: -- Thanks for the changes, [~nyang]. One new question: When you changed the default of *hive.zookeeper.session.timeout* to use a TimeValidator, did an extra zero slip into the value or was the original not in milliseconds? (600*1000 - 600ms.) Also, you might want to split the description of *hive.zookeeper.connection.basesleeptime* into two lines. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261561#comment-14261561 ] Na Yang commented on HIVE-9119: --- [~leftylev], thank you for reviewing the patch. I uploaded a new patch according to your suggestion and also changed the hive.zookeeper.session.timeout to 60ms. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261802#comment-14261802 ] Hive QA commented on HIVE-9119: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12689573/HIVE-9119.3.patch {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 6700 tests executed *Failed tests:* {noformat} TestHBaseCliDriver-hbase_binary_map_queries_prefix.q-hbase_stats.q-ppd_key_ranges.q - did not produce a TEST-*.xml file TestHBaseCliDriver-hbase_handler_bulk.q-hbase_scan_params.q-hbase_single_sourced_multi_insert.q - did not produce a TEST-*.xml file TestHBaseCliDriver-hbase_handler_snapshot.q-hbase_stats3.q-hbase_custom_key3.q - did not produce a TEST-*.xml file TestHBaseCliDriver-hbase_ppd_key_range.q-hbase_custom_key.q-hbase_pushdown.q - did not produce a TEST-*.xml file TestHBaseCliDriver-hbase_stats2.q-hbase_custom_key2.q-hbase_timestamp.q - did not produce a TEST-*.xml file TestHBaseCliDriver-hbase_stats_empty_partition.q-external_table_ppd.q-hbase_ppd_join.q - did not produce a TEST-*.xml file TestHBaseNegativeCliDriver - did not produce a TEST-*.xml file {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build//testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build//console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12689573 - PreCommit-HIVE-TRUNK-Build ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch, HIVE-9119.3.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260673#comment-14260673 ] Hive QA commented on HIVE-9119: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12689389/HIVE-9119.2.patch {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 6700 tests executed *Failed tests:* {noformat} TestHBaseCliDriver-hbase_binary_map_queries_prefix.q-hbase_stats.q-ppd_key_ranges.q - did not produce a TEST-*.xml file TestHBaseCliDriver-hbase_handler_bulk.q-hbase_scan_params.q-hbase_single_sourced_multi_insert.q - did not produce a TEST-*.xml file TestHBaseCliDriver-hbase_handler_snapshot.q-hbase_stats3.q-hbase_custom_key3.q - did not produce a TEST-*.xml file TestHBaseCliDriver-hbase_ppd_key_range.q-hbase_custom_key.q-hbase_pushdown.q - did not produce a TEST-*.xml file TestHBaseCliDriver-hbase_stats2.q-hbase_custom_key2.q-hbase_timestamp.q - did not produce a TEST-*.xml file TestHBaseCliDriver-hbase_stats_empty_partition.q-external_table_ppd.q-hbase_ppd_join.q - did not produce a TEST-*.xml file TestHBaseNegativeCliDriver - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2214/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2214/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2214/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12689389 - PreCommit-HIVE-TRUNK-Build ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258873#comment-14258873 ] Na Yang commented on HIVE-9119: --- [~leftylev], thank you very much for reviewing the patch. I will take your suggestions. Regarding to your questions, please see my answer below: 2.What are the units for hive.zookeeper.connection.basesleeptime? A TimeValidator could be used here – see comment on HIVE-6679 for an example. - [Na]: The unit is millisecond. I will follow the example when I upload a new patch. 3. Is the omission of an E for ZOOKEEPR deliberate in HIVE_ZOOKEEPR_CONNECTION_BASESLEEPTIME? It occurs once later in the code, also without the E. -[Na]: It is a typo. I wil correct it in the new patch. 4. Just curious: What's initial about the basesleeptime? -[Na]: CuratorFramework uses ExponentialBackoffRetryPolicy to reconnect to the ZooKeeper server. This retry policy retries a set number of times with increasing sleep time between retries. The basesleeptime is the sleep time for the first retry. I will explain it more clearly in the new patch. Currently, the qtests do not run properly with the CuratorFramework change. So I need to work on that and upload a new patch with these doc changes later on. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256775#comment-14256775 ] Lefty Leverenz commented on HIVE-9119: -- Doc review of patch 1 (config params): Looks good overall, just a few quibbles and questions. {code} + HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES(hive.zookeeper.connection.max.retries, 3, +Max number of times to retry when connecting to the zookeeper server.), + HIVE_ZOOKEEPR_CONNECTION_BASESLEEPTIME(hive.zookeeper.connection.basesleeptime, 1000, +Initial amount of time to wait between retries when connecting to the zookeeper server.), {code} # Please use camel case for ZooKeeper server in both descriptions. # What are the units for *hive.zookeeper.connection.basesleeptime*? A TimeValidator could be used here -- see comment on HIVE-6679 for an example. # Is the omission of an E for ZOOKEEPR deliberate in HIVE_ZOOKEEPR_CONNECTION_BASESLEEPTIME? It occurs once later in the code, also without the E. # Just curious: What's initial about the basesleeptime? Reference: * [ZooKeeper website uses camel case | http://zookeeper.apache.org] * [HIVE-6679 comment -- TimeValidator example | https://issues.apache.org/jira/browse/HIVE-6679?focusedCommentId=14248013page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14248013] * Another example of TimeValidator, with milliseconds: {code} HIVE_LOCALIZE_RESOURCE_WAIT_INTERVAL(hive.localize.resource.wait.interval, 5000ms, new TimeValidator(TimeUnit.MILLISECONDS), {code} ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256192#comment-14256192 ] Hive QA commented on HIVE-9119: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12688616/HIVE-9119.1.patch {color:red}ERROR:{color} -1 due to 155 failed/errored test(s), 4134 tests executed *Failed tests:* {noformat} TestCliDriver-alter_char2.q-insert_compressed.q-load_dyn_part15.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-alter_table_not_sorted.q-authorization_update.q-dynamic_partition_pruning.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_1_sql_std.q-drop_index_removes_partition_dirs.q-udf_date_sub.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_cli_createtab.q-groupby_ppr.q-limit_pushdown_negative.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_cli_nonsql.q-rcfile_merge1.q-stats2.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_cli_stdconfigauth.q-drop_partitions_filter2.q-bucketmapjoin1.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_create_table_owner_privs.q-create_func1.q-partition_wise_fileformat.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-auto_join18_multi_distinct.q-udf_repeat.q-list_bucket_query_multiskew_2.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-auto_join30.q-join_reorder3.q-lock1.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-auto_join4.q-mapjoin_decimal.q-input_dynamicserde.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-auto_join9.q-acid_vectorization.q-udf_floor.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-auto_sortmerge_join_13.q-alter_partition_clusterby_sortby.q-rcfile_merge2.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-autogen_colalias.q-tez_joins_explain.q-unicode_notation.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-avro_add_column.q-orc_wide_table.q-query_with_semi.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-avro_joins.q-disallow_incompatible_type_change_off.q-udf_max.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-bool_literal.q-udf_hash.q-groupby4_map.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-bucketcontext_5.q-udf_to_float.q-udf_locate.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-bucketmapjoin3.q-udf_round_3.q-udf_between.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-columnstats_partlvl_dp.q-smb_mapjoin_21.q-insert_overwrite_local_directory_1.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-columnstats_tbllvl.q-limit0.q-udf_exp.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-compute_stats_string.q-nullgroup4_multi_distinct.q-udf_concat_ws.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-cp_mj_rc.q-skewjoin_union_remove_2.q-udf_stddev_pop.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-create_genericudf.q-join13.q-notable_alias3.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-create_or_replace_view.q-skewjoinopt8.q-join_cond_pushdown_3.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-date_udf.q-transform_ppr2.q-udf_from_unixtime.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-decimal_6.q-vectorization_limit.q-authorization_index.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-describe_xpath.q-parquet_types.q-partcols1.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-dynpart_sort_optimization2.q-skewjoin_mapjoin4.q-list_bucket_dml_6.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-enforce_order.q-bucketcontext_4.q-stats_publisher_error_1.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-escape_distributeby1.q-create_view_partitioned.q-groupby_sort_7.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-gby_star.q-protectmode.q-udf_regexp_replace.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-groupby10.q-timestamp_comparison.q-tez_union.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-groupby12.q-udf2.q-udf_bigint.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-groupby3_map.q-plan_json.q-correlationoptimizer10.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-groupby4.q-reducesink_dedup.q-update_all_partitioned.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-groupby_complex_types.q-varchar_serde.q-groupby7_map_skew.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-groupby_grouping_sets5.q-desc_tbl_part_cols.q-decimal_2.q-and-12-more - did not produce a TEST-*.xml file
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255487#comment-14255487 ] Na Yang commented on HIVE-9119: --- According to a ZooKeeper test that I did, increasing the number of ZooKeeper client instances does not help improving the ZooKeeper operation requests performance in a multi-thread environment. The major load is on the ZooKeeper server side. Therefore, a singleton ZooKeeper client is a good choice for the ZookeeperHiveLockManager implementation. A zookeeper client/connection pool is not needed in this case. In addition, if a zookeeper client/connection pool is used and the pool side is set too big by the user, that can cause overhead on zookeeper server and even make the ZooKeeperHiveLockManager not workable if the zookeeper client number exceeds the number that the zookeeper server could handle. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251278#comment-14251278 ] Na Yang commented on HIVE-9119: --- This JIRA and HIVE-8135 are trying to address the same issue. The problem is whether a ZooKeeperClient pool is needed or a singleton client is able to do the work. ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)