[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607780#comment-17607780 ] Bryan Beaudreault commented on HBASE-24896: --- This Jira is quite old, so i've filed https://issues.apache.org/jira/browse/HBASE-27381 for further investigation/discussion > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.2 > > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16020" #81 daemon > prio=5 os_prio=0 cpu=53.85ms elapsed=315627.86s allocated=81984B > defined_classes=3 tid=0x7f3da0981590 nid=0x62d8 in Object.wait() > [0x7f3d9bd8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfoBuilder.(RegionInfoBuilder.java:49) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3231) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeOpenRegionProcedures(RSRpcServices.java:3755) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.lambda$executeProcedures$2(RSRpcServices.java:3827) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices$$Lambda$173/0x0017c0e40040.accept(Unknown > Source) > at java.util.ArrayList.forEach(java.base@11.0.6/ArrayList.java:1540) > at > java.util.Collections$UnmodifiableCollection.forEach(java.base@11.0.6/Collections.java:1085) > at >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607758#comment-17607758 ] Bryan Beaudreault commented on HBASE-24896: --- Trying some googling, and seeing stuff like [https://stackoverflow.com/questions/28631656/runnable-thread-state-but-in-object-wait.] This indicates: > Such deadlocks may be caused by a [typical > bug|https://bugs.openjdk.java.net/browse/JDK-8037567] when a subclass is > referenced from a static initializer. By no means conclusive, but if that's true then the UNDEFINED static constant in RegionInfo references a subclass of the interface and could be the reason this continues triggering. I'm removing this variable from our fork to see if it resolves the issue – we're seeing a relatively high percentage of new centos8 servers hit this. > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.2 > > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16020" #81 daemon > prio=5 os_prio=0 cpu=53.85ms elapsed=315627.86s allocated=81984B > defined_classes=3 tid=0x7f3da0981590 nid=0x62d8 in Object.wait() > [0x7f3d9bd8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfoBuilder.(RegionInfoBuilder.java:49) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3231) > at >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607735#comment-17607735 ] Bryan Beaudreault commented on HBASE-24896: --- We ran into this on a few servers recently, running version 2.4.6. So it has the fix for this jira, but our thread dumps were pretty identical to above – {code:java} org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3331) and org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:76) and at org.apache.hadoop.hbase.client.RegionInfoBuilder.build(RegionInfoBuilder.java:110) at org.apache.hadoop.hbase.client.RegionInfoBuilder.(RegionInfoBuilder.java:34) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3331) and (new) org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3404) {code} > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.2 > > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16020" #81 daemon > prio=5 os_prio=0 cpu=53.85ms elapsed=315627.86s allocated=81984B > defined_classes=3 tid=0x7f3da0981590 nid=0x62d8 in Object.wait() > [0x7f3d9bd8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfoBuilder.(RegionInfoBuilder.java:49) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3231) > at >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199088#comment-17199088 ] Hudson commented on HBASE-24896: Results for branch master [build #57 on builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/57/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/57/General_20Nightly_20Build_20Report/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/57/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/57/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.2 > > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16020" #81 daemon > prio=5 os_prio=0 cpu=53.85ms elapsed=315627.86s allocated=81984B > defined_classes=3 tid=0x7f3da0981590 nid=0x62d8 in Object.wait() > [0x7f3d9bd8c000] >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199073#comment-17199073 ] Hudson commented on HBASE-24896: Results for branch branch-2.3 [build #53 on builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/53/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/53/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/53/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/53/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/53/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.2 > > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199068#comment-17199068 ] Hudson commented on HBASE-24896: Results for branch branch-2 [build #49 on builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/49/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/49/General_20Nightly_20Build_20Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/49/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/49/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (x) {color:red}-1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/49/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.2 > > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198937#comment-17198937 ] Viraj Jasani commented on HBASE-24896: -- Thanks [~ndimiduk] [~stack]. Forward-ported changes to master and branch-2 after getting build results from respective PRs. > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.2 > > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16020" #81 daemon > prio=5 os_prio=0 cpu=53.85ms elapsed=315627.86s allocated=81984B > defined_classes=3 tid=0x7f3da0981590 nid=0x62d8 in Object.wait() > [0x7f3d9bd8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfoBuilder.(RegionInfoBuilder.java:49) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3231) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeOpenRegionProcedures(RSRpcServices.java:3755) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.lambda$executeProcedures$2(RSRpcServices.java:3827) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices$$Lambda$173/0x0017c0e40040.accept(Unknown > Source) > at java.util.ArrayList.forEach(java.base@11.0.6/ArrayList.java:1540) > at > java.util.Collections$UnmodifiableCollection.forEach(java.base@11.0.6/Collections.java:1085) > at >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198742#comment-17198742 ] Hudson commented on HBASE-24896: Results for branch branch-2.3 [build #49 on builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/49/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/49/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/49/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/49/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/49/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.2 > > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198021#comment-17198021 ] Nick Dimiduk commented on HBASE-24896: -- For the watchers, I have a new PR up: https://github.com/apache/hbase/pull/2417 Since this bug results in a rare but insidious live-lock on region server startup, if this patch is acceptable, I think we should sink 2.3.2RC1 in favor of including this fix. FYI [~vjasani]. > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Priority: Major > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16020" #81 daemon > prio=5 os_prio=0 cpu=53.85ms elapsed=315627.86s allocated=81984B > defined_classes=3 tid=0x7f3da0981590 nid=0x62d8 in Object.wait() > [0x7f3d9bd8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfoBuilder.(RegionInfoBuilder.java:49) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3231) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeOpenRegionProcedures(RSRpcServices.java:3755) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.lambda$executeProcedures$2(RSRpcServices.java:3827) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices$$Lambda$173/0x0017c0e40040.accept(Unknown > Source) > at java.util.ArrayList.forEach(java.base@11.0.6/ArrayList.java:1540) > at > java.util.Collections$UnmodifiableCollection.forEach(java.base@11.0.6/Collections.java:1085) > at >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197859#comment-17197859 ] Nick Dimiduk commented on HBASE-24896: -- This locked up a ITBLL test on 2.3.2RC1. Host in question was assigned meta, master never gave up on that assignment, retried thousands of times. > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Priority: Major > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16020" #81 daemon > prio=5 os_prio=0 cpu=53.85ms elapsed=315627.86s allocated=81984B > defined_classes=3 tid=0x7f3da0981590 nid=0x62d8 in Object.wait() > [0x7f3d9bd8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfoBuilder.(RegionInfoBuilder.java:49) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3231) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeOpenRegionProcedures(RSRpcServices.java:3755) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.lambda$executeProcedures$2(RSRpcServices.java:3827) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices$$Lambda$173/0x0017c0e40040.accept(Unknown > Source) > at java.util.ArrayList.forEach(java.base@11.0.6/ArrayList.java:1540) > at > java.util.Collections$UnmodifiableCollection.forEach(java.base@11.0.6/Collections.java:1085) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3827) > at >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193724#comment-17193724 ] Michael Stack commented on HBASE-24896: --- Was meant for an earlier incarnation of the patch. Purged it since that PR did not land and is not likely to. > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Priority: Major > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16020" #81 daemon > prio=5 os_prio=0 cpu=53.85ms elapsed=315627.86s allocated=81984B > defined_classes=3 tid=0x7f3da0981590 nid=0x62d8 in Object.wait() > [0x7f3d9bd8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfoBuilder.(RegionInfoBuilder.java:49) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3231) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeOpenRegionProcedures(RSRpcServices.java:3755) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.lambda$executeProcedures$2(RSRpcServices.java:3827) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices$$Lambda$173/0x0017c0e40040.accept(Unknown > Source) > at java.util.ArrayList.forEach(java.base@11.0.6/ArrayList.java:1540) > at > java.util.Collections$UnmodifiableCollection.forEach(java.base@11.0.6/Collections.java:1085) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3827) > at >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193719#comment-17193719 ] Nick Dimiduk commented on HBASE-24896: -- Was the release note here intended for a different Jira? Or maybe intended as a comment instead of release note? > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Priority: Major > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16020" #81 daemon > prio=5 os_prio=0 cpu=53.85ms elapsed=315627.86s allocated=81984B > defined_classes=3 tid=0x7f3da0981590 nid=0x62d8 in Object.wait() > [0x7f3d9bd8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfoBuilder.(RegionInfoBuilder.java:49) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3231) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeOpenRegionProcedures(RSRpcServices.java:3755) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.lambda$executeProcedures$2(RSRpcServices.java:3827) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices$$Lambda$173/0x0017c0e40040.accept(Unknown > Source) > at java.util.ArrayList.forEach(java.base@11.0.6/ArrayList.java:1540) > at > java.util.Collections$UnmodifiableCollection.forEach(java.base@11.0.6/Collections.java:1085) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3827) > at >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181370#comment-17181370 ] Michael Stack commented on HBASE-24896: --- Update. Helpful back and forth on PR entertaining different possible fixes but I need to repro the issue so can say for certain if an approach fixes the problem first. This pointer via [~bharathv] was particularly useful https://docs.oracle.com/javase/specs/jls/se8/html/jls-12.html#jls-12.4.2 Meantime, filing subissue to make the problematic RI.UNDEFINED IA.Private as suggested in review will move MRI out of RIB since nice-to-have. > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.2 > > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16020" #81 daemon > prio=5 os_prio=0 cpu=53.85ms elapsed=315627.86s allocated=81984B > defined_classes=3 tid=0x7f3da0981590 nid=0x62d8 in Object.wait() > [0x7f3d9bd8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfoBuilder.(RegionInfoBuilder.java:49) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3231) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeOpenRegionProcedures(RSRpcServices.java:3755) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.lambda$executeProcedures$2(RSRpcServices.java:3827) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices$$Lambda$173/0x0017c0e40040.accept(Unknown >
[jira] [Commented] (HBASE-24896) 'Stuck' in static initialization creating RegionInfo instance
[ https://issues.apache.org/jira/browse/HBASE-24896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17180045#comment-17180045 ] Michael Stack commented on HBASE-24896: --- I put up a PR that undoes some of the circular references static initializing... > 'Stuck' in static initialization creating RegionInfo instance > - > > Key: HBASE-24896 > URL: https://issues.apache.org/jira/browse/HBASE-24896 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.1 >Reporter: Michael Stack >Priority: Major > Attachments: hbasedn192-jstack-0.webarchive, > hbasedn192-jstack-1.webarchive, hbasedn192-jstack-2.webarchive > > > We ran into the following deadlocked server in testing. The priority handlers > seem stuck across multiple thread dumps. Seven of the ten total priority > threads have this state: > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16020" #82 daemon > prio=5 os_prio=0 cpu=0.70ms elapsed=315627.86s allocated=3744B > defined_classes=0 tid=0x7f3da0983040 nid=0x62d9 in Object.wait() > [0x7f3d9bc8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3478) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > {code} > The anomalous three are as follows: > h3. #1 > {code:java} > "RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020" #77 daemon > prio=5 os_prio=0 cpu=175.98ms elapsed=315627.86s allocated=2153K > defined_classes=14 tid=0x7f3da0ae6ec0 nid=0x62d4 in Object.wait() > [0x7f3d9c19] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfo.(RegionInfo.java:72) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3327) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1491) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2912) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44856) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code} > ...which is the creation of the UNDEFINED in RegionInfo here: > {color:#808000}@InterfaceAudience.Public{color}{color:#80}public > interface {color}RegionInfo {color:#80}extends > {color}Comparable { > RegionInfo {color:#660e7a}UNDEFINED {color}= > RegionInfoBuilder.newBuilder(TableName.valueOf({color:#008000}"__UNDEFINED__"{color})).build(); > > h3. #2 > {code:java} > "RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16020" #81 daemon > prio=5 os_prio=0 cpu=53.85ms elapsed=315627.86s allocated=81984B > defined_classes=3 tid=0x7f3da0981590 nid=0x62d8 in Object.wait() > [0x7f3d9bd8c000] >java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.client.RegionInfoBuilder.(RegionInfoBuilder.java:49) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3231) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeOpenRegionProcedures(RSRpcServices.java:3755) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.lambda$executeProcedures$2(RSRpcServices.java:3827) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices$$Lambda$173/0x0017c0e40040.accept(Unknown > Source) > at java.util.ArrayList.forEach(java.base@11.0.6/ArrayList.java:1540) > at > java.util.Collections$UnmodifiableCollection.forEach(java.base@11.0.6/Collections.java:1085) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3827) > at >