[jira] [Commented] (HIVE-22751) Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper
[ https://issues.apache.org/jira/browse/HIVE-22751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022374#comment-17022374 ] Hive QA commented on HIVE-22751: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991397/HIVE-22751.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17945 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20302/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20302/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20302/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12991397 - PreCommit-HIVE-Build > Move locking in HiveServer2::isDeregisteredWithZooKeeper to > ZooKeeperHiveHelper > --- > > Key: HIVE-22751 > URL: https://issues.apache.org/jira/browse/HIVE-22751 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22751.1.patch > > > [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/server/HiveServer2.java#L620] > [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/SessionManager.java#L597] > > When queries are run in beeline and closed, it causes unwanted delays in > shutting down beeline. Here is the threaddump from server side, which shows > HiveServer2 lock contention. > > It would be good to move synchronization to > "zooKeeperHelper.isDeregisteredWithZooKeeper" > > {noformat} > "main" #1 prio=5 os_prio=0 tid=0x7f78b0078800 nid=0x2d1c waiting on > condition [0x7f78b968c000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xac8d5ff0> (a > java.util.concurrent.FutureTask) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429) > at java.util.concurrent.FutureTask.get(FutureTask.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:187) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) > - locked <0xa9c5f2a8> (a java.lang.Object) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) > at > org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:790) > at > org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:763) > at > org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:687) > - locked <0xa99bd568> (a > org.apache.hive.service.server.HiveServer2) > at > org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016) > at > org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137) > at > org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294) > at > org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:318) > at org.apache.hadoop.util.RunJar.main(RunJar.java:232) > "HiveServer2-HttpHandler-Pool: Thread-50" #50 prio=5 os_prio=0 > tid=0x7f78b3e60800 nid=0x2fa7 waiting for monitor entry > [0x7f7884edf000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hive.service.server.HiveServer2.isDeregisteredWithZooKeeper(HiveServer2.java:600) > - waiting to lock <0xa99bd568> (a > org.apache.hive.service.server.HiveServer2) > at > org.apache.hive.service.cli.session.SessionManager.closeSessionInternal(SessionManager.java:631) > at > org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:621) > - locked <0xaa1970b0> (a >
[jira] [Commented] (HIVE-22751) Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper
[ https://issues.apache.org/jira/browse/HIVE-22751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022314#comment-17022314 ] Hive QA commented on HIVE-22751: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 25s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 53s{color} | {color:blue} standalone-metastore/metastore-common in master has 37 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 45s{color} | {color:blue} service in master has 50 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 59s{color} | {color:red} service generated 1 new + 50 unchanged - 0 fixed = 51 total (was 50) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 28m 12s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:service | | | Inconsistent synchronization of org.apache.hive.service.server.HiveServer2.zooKeeperHelper; locked 62% of time Unsynchronized access at HiveServer2.java:62% of time Unsynchronized access at HiveServer2.java:[line 620] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20302/dev-support/hive-personality.sh | | git revision | master / 6357dbc | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-20302/yetus/new-findbugs-service.html | | modules | C: standalone-metastore/metastore-common service U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-20302/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Move locking in HiveServer2::isDeregisteredWithZooKeeper to > ZooKeeperHiveHelper > --- > > Key: HIVE-22751 > URL: https://issues.apache.org/jira/browse/HIVE-22751 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22751.1.patch > > > [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/server/HiveServer2.java#L620] >
[jira] [Commented] (HIVE-22751) Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper
[ https://issues.apache.org/jira/browse/HIVE-22751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019829#comment-17019829 ] Anishek Agarwal commented on HIVE-22751: +1 pending tests. > Move locking in HiveServer2::isDeregisteredWithZooKeeper to > ZooKeeperHiveHelper > --- > > Key: HIVE-22751 > URL: https://issues.apache.org/jira/browse/HIVE-22751 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22751.1.patch > > > [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/server/HiveServer2.java#L620] > [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/SessionManager.java#L597] > > When queries are run in beeline and closed, it causes unwanted delays in > shutting down beeline. Here is the threaddump from server side, which shows > HiveServer2 lock contention. > > It would be good to move synchronization to > "zooKeeperHelper.isDeregisteredWithZooKeeper" > > {noformat} > "main" #1 prio=5 os_prio=0 tid=0x7f78b0078800 nid=0x2d1c waiting on > condition [0x7f78b968c000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xac8d5ff0> (a > java.util.concurrent.FutureTask) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429) > at java.util.concurrent.FutureTask.get(FutureTask.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:187) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) > - locked <0xa9c5f2a8> (a java.lang.Object) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) > at > org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:790) > at > org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:763) > at > org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:687) > - locked <0xa99bd568> (a > org.apache.hive.service.server.HiveServer2) > at > org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016) > at > org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137) > at > org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294) > at > org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:318) > at org.apache.hadoop.util.RunJar.main(RunJar.java:232) > "HiveServer2-HttpHandler-Pool: Thread-50" #50 prio=5 os_prio=0 > tid=0x7f78b3e60800 nid=0x2fa7 waiting for monitor entry > [0x7f7884edf000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hive.service.server.HiveServer2.isDeregisteredWithZooKeeper(HiveServer2.java:600) > - waiting to lock <0xa99bd568> (a > org.apache.hive.service.server.HiveServer2) > at > org.apache.hive.service.cli.session.SessionManager.closeSessionInternal(SessionManager.java:631) > at > org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:621) > - locked <0xaa1970b0> (a > org.apache.hive.service.cli.session.SessionManager) > at > org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:244) > at > org.apache.hive.service.cli.thrift.ThriftCLIService.CloseSession(ThriftCLIService.java:527) > at > org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1517) > at > org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1502) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) > at > org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:237) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >