[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800138#comment-16800138
 ] 

Hive QA commented on HIVE-21469:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963553/HIVE-21469.10.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15836 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16661/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16661/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16661/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963553 - PreCommit-HIVE-Build

> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.10.patch, 
> HIVE-21469.2.patch, HIVE-21469.3.patch, HIVE-21469.4.patch, 
> HIVE-21469.5.patch, HIVE-21469.6.patch, HIVE-21469.7.patch, 
> HIVE-21469.8.patch, HIVE-21469.9.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs.  Check first if 
> the node exists before trying to create it.
> {code:java}
>   try {
> curatorFramework.delete().forPath(zLock.getPath());
>   } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
>   }
> {code}
> There has historically been a quite a few bugs regarding leaked locks.  The 
> Driver will signal the session {{Thread}} by performing an interrupt.  That 
> interrupt can happen any time and it can kill a create/delete action within 
> the ZK framework.  We can see one example of workaround for this.  If the ZK 
> action is interrupted, simply do it again.  Well, what if it's interrupted 
> yet again?  The lock will be leaked.  Also, when the {{InterruptedException}} 
> is caught in the try block, the thread's interrupted flag is cleared.  The 
> flag is not reset in this code and therefore we lose the fact that this 
> thread has been interrupted.  This flag should be preserved so that other 
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
>   Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
>   } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
>   String name = ((ZooKeeperHiveLock)hiveLock).getPath();
>   throw new LockException("Node " + name + " can not be deleted after 
> " + numRetriesForUnLock + " attempts.",
>   e);
> }
>   }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete 
> the lock (again, for fear of leaking it).  This sleep should be 
> uninterruptible.  If we need to get the lock deleted, and there's a problem, 
> interrupting the sleep will cause the code to eventually exit and locks will 
> be leaked.
> It also requires a bunch more TLC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800128#comment-16800128
 ] 

Hive QA commented on HIVE-21469:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
20s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} ql: The patch generated 0 new + 46 unchanged - 37 
fixed = 46 total (was 83) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
45s{color} | {color:green} ql generated 0 new + 2249 unchanged - 6 fixed = 2249 
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 50s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16661/dev-support/hive-personality.sh
 |
| git revision | master / 2fa22bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16661/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.10.patch, 
> HIVE-21469.2.patch, HIVE-21469.3.patch, HIVE-21469.4.patch, 
> HIVE-21469.5.patch, HIVE-21469.6.patch, HIVE-21469.7.patch, 
> HIVE-21469.8.patch, HIVE-21469.9.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action i

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800094#comment-16800094
 ] 

Hive QA commented on HIVE-21469:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
32s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green} ql: The patch generated 0 new + 46 unchanged - 37 
fixed = 46 total (was 83) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
45s{color} | {color:green} ql generated 0 new + 2249 unchanged - 6 fixed = 2249 
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 41s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16660/dev-support/hive-personality.sh
 |
| git revision | master / 2fa22bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16660/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch, 
> HIVE-21469.3.patch, HIVE-21469.4.patch, HIVE-21469.5.patch, 
> HIVE-21469.6.patch, HIVE-21469.7.patch, HIVE-21469.8.patch, HIVE-21469.9.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799899#comment-16799899
 ] 

Hive QA commented on HIVE-21469:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963533/HIVE-21469.8.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15836 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16657/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16657/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16657/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963533 - PreCommit-HIVE-Build

> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch, 
> HIVE-21469.3.patch, HIVE-21469.4.patch, HIVE-21469.5.patch, 
> HIVE-21469.6.patch, HIVE-21469.7.patch, HIVE-21469.8.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs.  Check first if 
> the node exists before trying to create it.
> {code:java}
>   try {
> curatorFramework.delete().forPath(zLock.getPath());
>   } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
>   }
> {code}
> There has historically been a quite a few bugs regarding leaked locks.  The 
> Driver will signal the session {{Thread}} by performing an interrupt.  That 
> interrupt can happen any time and it can kill a create/delete action within 
> the ZK framework.  We can see one example of workaround for this.  If the ZK 
> action is interrupted, simply do it again.  Well, what if it's interrupted 
> yet again?  The lock will be leaked.  Also, when the {{InterruptedException}} 
> is caught in the try block, the thread's interrupted flag is cleared.  The 
> flag is not reset in this code and therefore we lose the fact that this 
> thread has been interrupted.  This flag should be preserved so that other 
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
>   Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
>   } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
>   String name = ((ZooKeeperHiveLock)hiveLock).getPath();
>   throw new LockException("Node " + name + " can not be deleted after 
> " + numRetriesForUnLock + " attempts.",
>   e);
> }
>   }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete 
> the lock (again, for fear of leaking it).  This sleep should be 
> uninterruptible.  If we need to get the lock deleted, and there's a problem, 
> interrupting the sleep will cause the code to eventually exit and locks will 
> be leaked.
> It also requires a bunch more TLC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799890#comment-16799890
 ] 

Hive QA commented on HIVE-21469:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
17s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
42s{color} | {color:red} ql: The patch generated 1 new + 46 unchanged - 37 
fixed = 47 total (was 83) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
25s{color} | {color:green} ql generated 0 new + 2250 unchanged - 5 fixed = 2250 
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 49s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16657/dev-support/hive-personality.sh
 |
| git revision | master / 2fa22bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16657/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16657/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch, 
> HIVE-21469.3.patch, HIVE-21469.4.patch, HIVE-21469.5.patch, 
> HIVE-21469.6.patch, HIVE-21469.7.patch, HIVE-21469.8.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even thoug

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799772#comment-16799772
 ] 

Hive QA commented on HIVE-21469:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963513/HIVE-21469.7.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15806 tests 
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=167)

[mapjoin_emit_interval.q,vector_auto_smb_mapjoin_14.q,deleteAnalyze.q,database.q,vector_bround.q,smb_mapjoin_6.q,vector_reduce_groupby_decimal.q,vectorized_dynamic_partition_pruning.q,cbo_views.q,vectorization_part.q,schema_evol_text_vecrow_part_all_primitive_llap_io.q,dynamic_partition_pruning.q,cte_mat_1.q,cluster.q,vector_char_mapjoin1.q,schema_evol_text_nonvec_part_llap_io.q,cte_5.q,subquery_shared_alias.q,vector_decimal_2.q,vector_outer_reference_windowed.q,bucketmapjoin7.q,vector_string_concat.q,subquery_nested_subquery.q,limit_pushdown3.q,vector_outer_join2.q,vectorized_context.q,metadata_only_queries.q,union6.q,cbo_subq_in.q,vectorized_timestamp_funcs.q]
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16655/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16655/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16655/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963513 - PreCommit-HIVE-Build

> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch, 
> HIVE-21469.3.patch, HIVE-21469.4.patch, HIVE-21469.5.patch, 
> HIVE-21469.6.patch, HIVE-21469.7.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs.  Check first if 
> the node exists before trying to create it.
> {code:java}
>   try {
> curatorFramework.delete().forPath(zLock.getPath());
>   } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
>   }
> {code}
> There has historically been a quite a few bugs regarding leaked locks.  The 
> Driver will signal the session {{Thread}} by performing an interrupt.  That 
> interrupt can happen any time and it can kill a create/delete action within 
> the ZK framework.  We can see one example of workaround for this.  If the ZK 
> action is interrupted, simply do it again.  Well, what if it's interrupted 
> yet again?  The lock will be leaked.  Also, when the {{InterruptedException}} 
> is caught in the try block, the thread's interrupted flag is cleared.  The 
> flag is not reset in this code and therefore we lose the fact that this 
> thread has been interrupted.  This flag should be preserved so that other 
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
>   Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
>   } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
>   String name = ((ZooKeeperHiveLock)hiveLock).getPath();
>   throw new LockException("Node " + name + " c

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799747#comment-16799747
 ] 

Hive QA commented on HIVE-21469:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
9s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
43s{color} | {color:red} ql: The patch generated 1 new + 47 unchanged - 36 
fixed = 48 total (was 83) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
25s{color} | {color:red} ql generated 1 new + 2250 unchanged - 5 fixed = 2251 
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 49s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.lockPrimitive(HiveLockObject,
 HiveLockMode, boolean, Collection) might ignore java.lang.Exception  At 
ZooKeeperHiveLockManager.java:ignore java.lang.Exception  At 
ZooKeeperHiveLockManager.java:[line 459] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16655/dev-support/hive-personality.sh
 |
| git revision | master / 2fa22bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16655/yetus/diff-checkstyle-ql.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16655/yetus/new-findbugs-ql.html
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16655/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch, 
> HIVE-21469.3.patch, HIVE-21469.4.patch, HIVE-21469.5.patch, 
> HIVE-21469.6.patch, HIVE-21469.7.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the pa

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799537#comment-16799537
 ] 

Hive QA commented on HIVE-21469:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963442/HIVE-21469.6.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15836 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into1] 
(batchId=99)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into2] 
(batchId=101)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into3] 
(batchId=99)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into4] 
(batchId=99)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[lockneg_try_drop_locked_db]
 (batchId=100)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16646/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16646/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16646/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963442 - PreCommit-HIVE-Build

> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch, 
> HIVE-21469.3.patch, HIVE-21469.4.patch, HIVE-21469.5.patch, HIVE-21469.6.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs.  Check first if 
> the node exists before trying to create it.
> {code:java}
>   try {
> curatorFramework.delete().forPath(zLock.getPath());
>   } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
>   }
> {code}
> There has historically been a quite a few bugs regarding leaked locks.  The 
> Driver will signal the session {{Thread}} by performing an interrupt.  That 
> interrupt can happen any time and it can kill a create/delete action within 
> the ZK framework.  We can see one example of workaround for this.  If the ZK 
> action is interrupted, simply do it again.  Well, what if it's interrupted 
> yet again?  The lock will be leaked.  Also, when the {{InterruptedException}} 
> is caught in the try block, the thread's interrupted flag is cleared.  The 
> flag is not reset in this code and therefore we lose the fact that this 
> thread has been interrupted.  This flag should be preserved so that other 
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
>   Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
>   } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
>   String name = ((ZooKeeperHiveLock)hiveLock).getPath();
>   throw new LockException("Node " + name + " can not be deleted after 
> " + numRetriesForUnLock + " attempts.",
>   e);
> }
>   }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete 
> the lock (again, for fear of leaking it).  This sleep should be 
> uninterruptible.  If we need to get the lock deleted, and the

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799526#comment-16799526
 ] 

Hive QA commented on HIVE-21469:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
19s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} ql: The patch generated 0 new + 49 unchanged - 34 
fixed = 49 total (was 83) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
20s{color} | {color:green} ql generated 0 new + 2251 unchanged - 4 fixed = 2251 
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16646/dev-support/hive-personality.sh
 |
| git revision | master / 2fa22bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16646/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch, 
> HIVE-21469.3.patch, HIVE-21469.4.patch, HIVE-21469.5.patch, HIVE-21469.6.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs.  Check first if 
> the node exists before trying to create it

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799186#comment-16799186
 ] 

Hive QA commented on HIVE-21469:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963405/HIVE-21469.5.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 15199 tests 
executed
*Failed tests:*
{noformat}
TestBlobstoreCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=278)
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=15)

[avro_joins.q,parquet_vectorization_8.q,serde_reported_schema.q,mm_bhif.q,annotate_stats_join_pkfk.q,udf_unix_timestamp.q,describe_comment_nonascii.q,orc_analyze.q,schema_evol_orc_acidvec_part_update.q,partition_date.q,stats15.q,tez_join_result_complex.q,input36.q,alter_numbuckets_partitioned_table2_h23.q,transform_ppr1.q,spark_vectorized_dynamic_partition_pruning.q,fullouter_mapjoin_1_optimized.q,materialized_view_create_rewrite_time_window.q,unionDistinct_2.q,udaf_histogram_numeric.q,get_splits_0.q,load_data_using_job.q,auto_join26.q,list_bucket_dml_3.q,parquet_types_vectorization.q,show_materialized_views.q,smb_mapjoin_22.q,vector_decimal_6.q,autoColumnStats_8.q,input5.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=33)

[mapjoin_emit_interval.q,partition_char.q,deleteAnalyze.q,input_part8.q,udf_levenshtein.q,parquet_vectorization_decimal_date.q,avrotblsjoin.q,cbo_views.q,vectorization_part.q,dynamic_partition_pruning.q,alter_rename_table.q,cluster.q,input44.q,parquet_vectorization_9.q,schema_evol_text_nonvec_part_llap_io.q,subquery_shared_alias.q,orc_ppd_str_conversion.q,bucketmapjoin7.q,subquery_nested_subquery.q,limit_pushdown3.q,input_testxpath.q,vector_outer_join2.q,input.q,join19.q,encryption_ctas.q,union6.q,lock2.q,load_hdfs_file_with_space_in_the_name.q,partition_boolexpr.q,vectorized_timestamp_funcs.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=40)

[lateral_view_noalias.q,semijoin_reddedup.q,sysdb.q,alter_char2.q,parquet_vectorization_11.q,partialdhj.q,infer_bucket_sort_dyn_part.q,partition_varchar2.q,rename_table_update_column_stats.q,bucketcontext_8.q,input_testxpath2.q,udf_mask_last_n.q,create_view_partitioned.q,vector_windowing_order_null.q,alter_view_rename.q,input17.q,date_comparison.q,udf_notequal.q,ppd_gby_join.q,show_create_table_partitioned.q,udaf_binarysetfunctions.q,tez_union.q,druidmini_test_ts.q,approx_distinct.q,materialized_view_rewrite_ssb.q,dynpart_merge.q,udf_substr.q,comments.q,correlationoptimizer12.q,timestamp_comparison2.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=61)

[multi_insert_partitioned.q,parquet_types.q,udf_to_unix_timestamp.q,schema_evol_stats.q,avro_nullable_union.q,parquet_ppd_timestamp.q,merge1.q,authorization_9.q,sample5.q,transform3.q,encryption_select_read_only_unencrypted_tbl.q,join_array.q,windowing_distinct.q,orc_ppd_schema_evol_3a.q,orc_merge5.q,skewjoin_mapjoin7.q,leadlag.q,auto_join22.q,vector_groupby_reduce.q,union8.q,lineage2.q,correlationoptimizer1.q,spark_combine_equivalent_work_2.q,input4_cb_delim.q,mapjoin_distinct.q,vector_decimal_mapjoin.q,udf_not.q,avro_schema_evolution_native.q,udf_mask_show_last_n.q,materialized_view_rewrite_8.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=71)

[schema_evol_orc_acid_table_update.q,join35.q,stats_date.q,udf_lessthan.q,char_nested_types.q,infer_bucket_sort_map_operators.q,load_dyn_part8.q,part_inherit_tbl_props_empty.q,order_by_expr_2.q,decimal_5.q,exim_25_export_parentpath_has_inaccessible_children.q,alter_merge_orc.q,bucketcontext_2.q,vector_char_cast.q,show_create_table_db_table.q,udf_array.q,vector_custom_udf_configure.q,parquet_map_of_maps.q,autoColumnStats_6.q,archive_excludeHadoop20.q,orc_merge4.q,multi_insert_with_join.q,varchar_2.q,union7.q,materialized_view_rewrite_9.q,char_1.q,schema_evol_orc_acidvec_part.q,partition_varchar1.q,cbo_rp_gby.q,rcfile_union.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=101)

[udf_array_contains_wrong2.q,alter_partition_coltype_invalidcolname.q,authorization_role_cycles2.q,archive_corrupt.q,ptf_window_boundaries2.q,merge_negative_3.q,clusterbysortby.q,fallbackauth_create_func2.q,clustern3.q,invalid_select_expression.q,subquery_select_aggregate.q,authorization_ctas2.q,dynamic_partitions_with_whitelist.q,authorization_public_create.q,set_hiveconf_validation2.q,create_or_replace_view5.q,invalid_stddev_samp_syntax.q,select_udtf_alias.q,windowing_leadlag_in_udaf.q,authorization_truncate.q,analyze_view.q,exim_14_nonpart_part.q,orc_change_fileformat.q,delete_non_acid_table.q,create_with_constraints_validate.q,orc_repl

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799134#comment-16799134
 ] 

Hive QA commented on HIVE-21469:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
26s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
44s{color} | {color:red} ql: The patch generated 1 new + 49 unchanged - 34 
fixed = 50 total (was 83) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
43s{color} | {color:red} ql generated 1 new + 2252 unchanged - 3 fixed = 2253 
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m  2s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Write to static field 
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.metrics 
from instance method 
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.setContext(HiveLockManagerCtx)
  At ZooKeeperHiveLockManager.java:from instance method 
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.setContext(HiveLockManagerCtx)
  At ZooKeeperHiveLockManager.java:[line 124] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16635/dev-support/hive-personality.sh
 |
| git revision | master / 2fa22bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16635/yetus/diff-checkstyle-ql.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16635/yetus/new-findbugs-ql.html
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16635/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch, 
> HIVE-21469.3.patch, HIVE-21469.4.patch, HIVE-21469.5.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSIST

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798685#comment-16798685
 ] 

Hive QA commented on HIVE-21469:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963345/HIVE-21469.4.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 918 failed/errored test(s), 15360 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=101)

[udf_array_contains_wrong2.q,alter_partition_coltype_invalidcolname.q,authorization_role_cycles2.q,archive_corrupt.q,ptf_window_boundaries2.q,merge_negative_3.q,clusterbysortby.q,fallbackauth_create_func2.q,clustern3.q,invalid_select_expression.q,subquery_select_aggregate.q,authorization_ctas2.q,dynamic_partitions_with_whitelist.q,authorization_public_create.q,set_hiveconf_validation2.q,create_or_replace_view5.q,invalid_stddev_samp_syntax.q,select_udtf_alias.q,windowing_leadlag_in_udaf.q,authorization_truncate.q,analyze_view.q,exim_14_nonpart_part.q,orc_change_fileformat.q,delete_non_acid_table.q,create_with_constraints_validate.q,orc_replace_columns1_acid.q,describe_xpath3.q,input4_2.q,archive_multi1.q,orc_replace_columns2_acid.q,invalid_varchar_length_3.q,authorization_create_tbl.q,add_partition_with_whitelist.q,check_constraint_qual_name.q,authorization_uri_altertab_setloc.q,groupby_grouping_id1.q,create_udaf_failure.q,invalid_t_alter1.q,subquery_select_distinct.q,udf_size_wrong_type.q,exim_04_nonpart_noncompat_colnumber.q,archive_partspec2.q,join2.q,exim_03_nonpart_noncompat_colschema.q,input2.q,authorization_fail_6.q,ptf_negative_WhereWithRankCond.q,dyn_part4.q,spark_task_failure.q,authorization_uri_insert_local.q,materialized_view_no_supported_op_rewrite_2.q,drop_database_cascade.q,duplicate_alias_in_transform.q,alter_view_failure5.q,subquery_corr_select.q,partition_column_names_with_leading_and_trailing_spaces.q,view_delete.q,load_orc_negative3.q,alter_table_wrong_table.q,subquery_scalar_multi_rows.q,authorization_create_role_no_admin.q,exim_00_unsupported_schema.q,semijoin4.q,compile_processor.q,create_unknown_genericudf.q,udf_function_does_not_implement_udf.q,orc_type_promotion3_acid.q,udf_local_resource.q,merge_negative_4.q,lockneg_query_tbl_in_locked_db.q,udaf_collect_set_unsupported.q,update_non_acid_table.q,insert_into2.q,exchange_partition_neg_partition_exists3.q,join28.q,alter_view_failure7.q,invalid_cast_to_binary_1.q,change_hive_tmp_table_space.q,fileformat_void_output.q,wrong_column_type.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=99)

[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,mm_update.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,authorization_jdbc_keystore.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,materialized_view_name_collusion.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,merge_column_mismatch.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alte

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798661#comment-16798661
 ] 

Hive QA commented on HIVE-21469:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
27s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
41s{color} | {color:red} ql: The patch generated 1 new + 49 unchanged - 34 
fixed = 50 total (was 83) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
32s{color} | {color:red} ql generated 1 new + 2252 unchanged - 3 fixed = 2253 
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 41s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Write to static field 
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.metrics 
from instance method 
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.setContext(HiveLockManagerCtx)
  At ZooKeeperHiveLockManager.java:from instance method 
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.setContext(HiveLockManagerCtx)
  At ZooKeeperHiveLockManager.java:[line 124] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16626/dev-support/hive-personality.sh
 |
| git revision | master / b953c4e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16626/yetus/diff-checkstyle-ql.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16626/yetus/new-findbugs-ql.html
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16626/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch, 
> HIVE-21469.3.patch, HIVE-21469.4.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> 

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798559#comment-16798559
 ] 

Hive QA commented on HIVE-21469:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963319/HIVE-21469.3.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16622/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16622/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16622/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:49:30.153
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16622/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:49:30.155
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 38682a4 HIVE-21446 : Hive Server going OOM during hive external 
table replications. (Mahesh Kumar Behera, reviewed by Sankar Hariappan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 38682a4 HIVE-21446 : Hive Server going OOM during hive external 
table replications. (Mahesh Kumar Behera, reviewed by Sankar Hariappan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:49:31.217
+ rm -rf ../yetus_PreCommit-HIVE-Build-16622
+ mkdir ../yetus_PreCommit-HIVE-Build-16622
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16622
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16622/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not 
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockMode.java: does 
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObj.java: does 
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java: 
does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLock.java:
 does not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java:
 does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q 
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc2085600430980394434.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc2085600430980394434.exe, 
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
 
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator  Version 3.5.2
protoc-jar: executing: [/tmp/protoc5705060335407942899.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator  Version 3.5.2
Output file 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
 does not exist: must build 
/data/hiveptest/working/

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796771#comment-16796771
 ] 

Hive QA commented on HIVE-21469:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963004/HIVE-21469.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 135 failed/errored test(s), 12413 tests 
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=267)
TestAutoPurgeTables - did not produce a TEST-*.xml file (likely timed out) 
(batchId=254)
TestBlobstoreCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=278)
TestClearDanglingScratchDir - did not produce a TEST-*.xml file (likely timed 
out) (batchId=254)
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=1)

[udf_upper.q,ctas_date.q,materialized_view_rewrite_part_2.q,groupby_grouping_sets3.q,vector_decimal_5.q,vector_case_when_conversion.q,bucket_map_join_spark4.q,timestamp_2.q,schema_evol_orc_acid_table_update_llap_io.q,date_join1.q,constprog_type.q,timestamp_ints_casts.q,udf_negative.q,orc_merge_diff_fs.q,udf_substring_index.q,results_cache_lifetime.q,cross_prod_3.q,masking_12.q,diff_part_input_formats.q,auto_join_without_localtask.q,join46.q,ctas_uses_table_location.q,tez_bmj_schema_evolution.q,bucketmapjoin4.q,udf_context_aware.q,authorization_non_id.q,mapjoin_test_outer.q,vectorization_9.q,input15.q,udf_PI.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=10)

[msck_repair_acid.q,parquet_ppd_decimal.q,authorization_load.q,udf_md5.q,alter_view_as_select.q,groupby_sort_6.q,limit0.q,select_as_omitted.q,decimal_udf.q,list_bucket_query_oneskew_3.q,vector_null_projection.q,smb_mapjoin_12.q,vector_cast_constant.q,encryption_insert_partition_static.q,semijoin3.q,reloadJar.q,orc_merge_incompat3.q,groupby2_limit.q,sort_merge_join_desc_1.q,vector_fullouter_mapjoin_1_optimized.q,ba_table3.q,mi.q,input0.q,parquet_map_of_arrays_of_ints.q,auto_join11.q,delete_whole_partition.q,authorization_view_disable_cbo_3.q,avro_date.q,udf_like.q,compute_stats_decimal.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=11)

[cbo_rp_lineage2.q,join_emit_interval.q,druidkafkamini_avro.q,mm_iow_temp.q,mapjoin2.q,schema_evol_text_vec_table_llap_io.q,vector_windowing_navfn.q,vectorization_12.q,vector_number_compare_projection.q,parquet_table_with_subschema.q,parquet_join2.q,authorization_8.q,exim_07_all_part_over_nonoverlap.q,parquet_ppd_char.q,udf_locate.q,nullgroup4_multi_distinct.q,join_rc.q,bucket6.q,schema_evol_text_vecrow_table.q,udf_likeany.q,orc_ppd_char.q,udf_boolean.q,udf_xpath_double.q,orc_ppd_same_table_multiple_aliases.q,materialized_view_drop.q,alter2.q,vector_full_outer_join.q,typechangetest.q,distinct_66.q,rcfile_merge4.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=12)

[skewjoinopt15.q,vector_coalesce.q,udf_elt.q,join44.q,udf5.q,udf_case_thrift.q,inputddl2.q,join_constraints_optimization.q,correlationoptimizer13.q,udf_testlength.q,parquet_vectorization_1.q,ppd_repeated_alias.q,correlationoptimizer4.q,ppd_windowing2.q,udf_union.q,masking_1_newdb.q,vector_leftsemi_mapjoin.q,udf_string.q,repl_1_drop.q,cbo_rp_insert.q,ppd_clusterby.q,udf_trunc_number.q,union10.q,vector_if_expr.q,load_dyn_part7.q,llap_acid2.q,groupby_constcolval.q,acid_mapjoin.q,explainanalyze_5.q,distinct_windowing.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=13)

[load_dyn_part3.q,autoColumnStats_4.q,drop_table.q,stat_estimate_drill.q,auto_join33.q,parquet_ppd_varchar.q,udf_sha2.q,groupby5_map_skew.q,merge4.q,storage_format_descriptor.q,mapjoin_hook.q,multi_column_in_single.q,schema_evol_orc_nonvec_table.q,cbo_rp_subq_in.q,authorization_view_disable_cbo_4.q,list_bucket_dml_2.q,cbo_rp_semijoin.q,char_2.q,non_ascii_literal2.q,load_part_authsuccess.q,auto_sortmerge_join_15.q,explain_rearrange.q,stats_nonpart.q,varchar_union1.q,vectorized_mapjoin3.q,input21.q,vector_udf2.q,groupby_cube_multi_gby.q,annotate_stats_limit.q,union34.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=14)

[auto_join18.q,partition_coltype_literals.q,input1_limit.q,udf_divide.q,input40.q,udf_round_2_auto_stats.q,correlationoptimizer8.q,auto_sortmerge_join_14.q,udf_array_contains.q,union22.q,non_ascii_literal1.q,bucket_map_join_tez2.q,sample_islocalmode_hook.q,literal_decimal.q,constprog2.q,parquet_external_time.q,infer_bucket_sort_reducers_power_two.q,input20.q,smb_join_partition_key.q,union_remove_14.q,vector_count.q,udf_if.q,input38.q,decimal_trailing.q,load_fs_overwrite.q,notable_alias2.q,join_reorder.q,constant_prop.q,sample1.q,bucketmapjoin8.q]
TestCliDriver - did not produ

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796662#comment-16796662
 ] 

Hive QA commented on HIVE-21469:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
11s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} ql: The patch generated 0 new + 10 unchanged - 23 
fixed = 10 total (was 33) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16581/dev-support/hive-personality.sh
 |
| git revision | master / 230db04 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16581/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs.  Check first if 
> the node exists before trying to create it.
> {code:java}
>   try {
> curatorFramework.delete().forPath(zLock.getPath());
>   } catch (InterruptedException ie) {
>

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795836#comment-16795836
 ] 

Hive QA commented on HIVE-21469:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962898/HIVE-21469.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 132 failed/errored test(s), 12450 tests 
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=267)
TestAutoPurgeTables - did not produce a TEST-*.xml file (likely timed out) 
(batchId=254)
TestBlobstoreCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=278)
TestClearDanglingScratchDir - did not produce a TEST-*.xml file (likely timed 
out) (batchId=254)
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=1)

[udf_upper.q,ctas_date.q,materialized_view_rewrite_part_2.q,groupby_grouping_sets3.q,vector_decimal_5.q,vector_case_when_conversion.q,bucket_map_join_spark4.q,timestamp_2.q,schema_evol_orc_acid_table_update_llap_io.q,date_join1.q,constprog_type.q,timestamp_ints_casts.q,udf_negative.q,orc_merge_diff_fs.q,udf_substring_index.q,results_cache_lifetime.q,cross_prod_3.q,masking_12.q,diff_part_input_formats.q,auto_join_without_localtask.q,join46.q,ctas_uses_table_location.q,tez_bmj_schema_evolution.q,bucketmapjoin4.q,udf_context_aware.q,authorization_non_id.q,mapjoin_test_outer.q,vectorization_9.q,input15.q,udf_PI.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=10)

[msck_repair_acid.q,parquet_ppd_decimal.q,authorization_load.q,udf_md5.q,alter_view_as_select.q,groupby_sort_6.q,limit0.q,select_as_omitted.q,decimal_udf.q,list_bucket_query_oneskew_3.q,vector_null_projection.q,smb_mapjoin_12.q,vector_cast_constant.q,encryption_insert_partition_static.q,semijoin3.q,reloadJar.q,orc_merge_incompat3.q,groupby2_limit.q,sort_merge_join_desc_1.q,vector_fullouter_mapjoin_1_optimized.q,ba_table3.q,mi.q,input0.q,parquet_map_of_arrays_of_ints.q,auto_join11.q,delete_whole_partition.q,authorization_view_disable_cbo_3.q,avro_date.q,udf_like.q,compute_stats_decimal.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=11)

[cbo_rp_lineage2.q,join_emit_interval.q,druidkafkamini_avro.q,mm_iow_temp.q,mapjoin2.q,schema_evol_text_vec_table_llap_io.q,vector_windowing_navfn.q,vectorization_12.q,vector_number_compare_projection.q,parquet_table_with_subschema.q,parquet_join2.q,authorization_8.q,exim_07_all_part_over_nonoverlap.q,parquet_ppd_char.q,udf_locate.q,nullgroup4_multi_distinct.q,join_rc.q,bucket6.q,schema_evol_text_vecrow_table.q,udf_likeany.q,orc_ppd_char.q,udf_boolean.q,udf_xpath_double.q,orc_ppd_same_table_multiple_aliases.q,materialized_view_drop.q,alter2.q,vector_full_outer_join.q,typechangetest.q,distinct_66.q,rcfile_merge4.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=12)

[skewjoinopt15.q,vector_coalesce.q,udf_elt.q,join44.q,udf5.q,udf_case_thrift.q,inputddl2.q,join_constraints_optimization.q,correlationoptimizer13.q,udf_testlength.q,parquet_vectorization_1.q,ppd_repeated_alias.q,correlationoptimizer4.q,ppd_windowing2.q,udf_union.q,masking_1_newdb.q,vector_leftsemi_mapjoin.q,udf_string.q,repl_1_drop.q,cbo_rp_insert.q,ppd_clusterby.q,udf_trunc_number.q,union10.q,vector_if_expr.q,load_dyn_part7.q,llap_acid2.q,groupby_constcolval.q,acid_mapjoin.q,explainanalyze_5.q,distinct_windowing.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=13)

[load_dyn_part3.q,autoColumnStats_4.q,drop_table.q,stat_estimate_drill.q,auto_join33.q,parquet_ppd_varchar.q,udf_sha2.q,groupby5_map_skew.q,merge4.q,storage_format_descriptor.q,mapjoin_hook.q,multi_column_in_single.q,schema_evol_orc_nonvec_table.q,cbo_rp_subq_in.q,authorization_view_disable_cbo_4.q,list_bucket_dml_2.q,cbo_rp_semijoin.q,char_2.q,non_ascii_literal2.q,load_part_authsuccess.q,auto_sortmerge_join_15.q,explain_rearrange.q,stats_nonpart.q,varchar_union1.q,vectorized_mapjoin3.q,input21.q,vector_udf2.q,groupby_cube_multi_gby.q,annotate_stats_limit.q,union34.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=14)

[auto_join18.q,partition_coltype_literals.q,input1_limit.q,udf_divide.q,input40.q,udf_round_2_auto_stats.q,correlationoptimizer8.q,auto_sortmerge_join_14.q,udf_array_contains.q,union22.q,non_ascii_literal1.q,bucket_map_join_tez2.q,sample_islocalmode_hook.q,literal_decimal.q,constprog2.q,parquet_external_time.q,infer_bucket_sort_reducers_power_two.q,input20.q,smb_join_partition_key.q,union_remove_14.q,vector_count.q,udf_if.q,input38.q,decimal_trailing.q,load_fs_overwrite.q,notable_alias2.q,join_reorder.q,constant_prop.q,sample1.q,bucketmapjoin8.q]
TestCliDriver - did not produ

[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795733#comment-16795733
 ] 

Hive QA commented on HIVE-21469:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
11s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
41s{color} | {color:red} ql: The patch generated 3 new + 10 unchanged - 23 
fixed = 13 total (was 33) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 28s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16567/dev-support/hive-personality.sh
 |
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16567/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16567/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
>  Issue Type: Improvement
>  Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
>   @Override
>   public void setContext(HiveLockManagerCtx ctx) throws LockException {
>  try {
>   curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
>   parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
>   try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" 
> +  parent, new byte[0]);
>   } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() != 
> KeeperException.Code.NODEEXISTS) {
>   LOG.warn("Unexpected ZK exception when creating parent node /" + 
> parent, e);
> }
>   }
> {code}
> Every time a new session is created and this {{setContext}} method is called, 
> it attempts to create the root node.  I have seen that, even though the root 
> node exists, an create node action is written to the ZK logs.  Check first if 
> the node exists before trying to create it.
> {code:java}
>   try {
> curatorFramework.delete