[
https://issues.apache.org/jira/browse/HBASE-17023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15737730#comment-15737730
]
Hadoop QA commented on HBASE-17023:
-----------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 54s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 44s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 81m 39s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 116m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12842659/HBASE-17023.v1-branch-1.patch
|
| JIRA Issue | HBASE-17023 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 326cbb511865 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 / c2801a2 |
| Default Java | 1.7.0_80 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_111
/usr/lib/jvm/java-7-oracle:1.7.0_80 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4865/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4865/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4865/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Region left unassigned due to AM and SSH each thinking others would do the
> assignment work
> ------------------------------------------------------------------------------------------
>
> Key: HBASE-17023
> URL: https://issues.apache.org/jira/browse/HBASE-17023
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
> Affects Versions: 1.1.0
> Reporter: Stephen Yuan Jiang
> Assignee: Stephen Yuan Jiang
> Attachments: HBASE-17023.v0-branch-1.1.patch,
> HBASE-17023.v1-branch-1.patch
>
>
> Another Assignment Manager and SSH issue. This issue is similar to
> HBASE-13330, except this time the code path goes through ClosedRegionHandler
> and we should apply the same fix of HBASE-13330 to ClosedRegionHandler.
> Basically, the AssignmentManager thinks the ServerShutdownHandler would
> assign the region and the ServerShutdownHandler thinks that the
> AssignmentManager would assign the region. The region
> (23e0186c4d2b5cc09f25de35fe174417) ultimately never gets assigned. Below is
> an analysis from the logs that captures the flow of events.
> 1. The AssignmentManager had initially assigned this region to
> {{rs42.prod.foo.com,16020,1476293566365}}.
> 2. The {{rs42.prod.foo.com,16020,1476293566365}} stops and sends the CLOSE
> request to master.
> 3. ServerShutdownHandler(SSH) runs to assign this region to
> {{rs44.prod.foo.com,16020,1476294287692}}, but assign failed.
> 4. When the master restarted it did a scan of the meta to learn about the
> regions in the cluster. It found this region still being assigned to
> {{rs42} from the meta record.
> 5. However, this {{rs42}} server was not alive anymore. So, the
> AssignmentManager queued up a ServerShutdownHandling task for this (that
> asynchronously executes):
> 6. In the meantime, the AssignmentManager proceeded to read the RIT nodes
> from ZK. It found this region as well is in RS_ZK_REGION_FAILED_OPEN in the
> {{rs44}} RS.
> 7. The region was moved to CLOSED state:
> {noformat}
> 2016-10-12 17:45:11,637 DEBUG [AM.ZK.Worker-pool2-t6]
> master.AssignmentManager: Handling RS_ZK_REGION_FAILED_OPEN,
> server=rs44.prod.foo.com,16020,1476294287692,
> region=23e0186c4d2b5cc09f25de35fe174417,
> current_state={23e0186c4d2b5cc09f25de35fe174417 state=PENDING_OPEN,
> ts=1476294311564, server=rs44.prod.foo.com,16020,1476294287692}
> 2016-10-12 17:45:11,637 INFO [AM.ZK.Worker-pool2-t6] master.RegionStates:
> Transition {23e0186c4d2b5cc09f25de35fe174417 state=PENDING_OPEN,
> ts=1476294311564, server=rs44.prod.foo.com,16020,1476294287692} to
> {23e0186c4d2b5cc09f25de35fe174417 state=CLOSED, ts=1476294311637,
> server=rs44.prod.foo.com,16020,1476294287692}
> 2016-10-12 17:45:11,637 WARN [AM.ZK.Worker-pool2-t6] master.RegionStates:
> 23e0186c4d2b5cc09f25de35fe174417 moved to CLOSED on
> rs44.prod.foo.com,16020,1476294287692, expected
> rs42.prod.foo.com,16020,1476293566365
> {noformat}
> 8. After that the AssignmentManager tried to assign it again. However, the
> assignment didn't happen because the ServerShutdownHandling task queued
> earlier didn't yet execute:
> {noformat}
> 2016-10-12 17:45:11,637 DEBUG [AM.ZK.Worker-pool2-t6]
> master.AssignmentManager: Found an existing plan for
> table1,3025965238305402_2,1468091325259.23e0186c4d2b5cc09f25de35fe174417.
> destination server is rs44.prod.foo.com,16020,1476294287692 accepted as a
> dest server = false
> 2016-10-12 17:45:11,697 DEBUG [AM.ZK.Worker-pool2-t6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> table1,3025965238305402_2,1468091325259.23e0186c4d2b5cc09f25de35fe174417.;
> generated random
> plan=hri=table1,3025965238305402_2,1468091325259.23e0186c4d2b5cc09f25de35fe174417.,
> src=, dest=rs28.prod.foo.com,16020,1476294291314; 10 (online=11) available
> servers, forceNewPlan=true
> 2016-10-12 17:45:11,697 DEBUG [AM.ZK.Worker-pool2-t6]
> handler.ClosedRegionHandler: Handling CLOSED event for
> 23e0186c4d2b5cc09f25de35fe174417
> 2016-10-12 17:45:11,697 WARN [AM.ZK.Worker-pool2-t6] master.RegionStates:
> 23e0186c4d2b5cc09f25de35fe174417 moved to CLOSED on
> rs44.prod.foo.com,16020,1476294287692, expected
> rs42.prod.foo.com,16020,1476293566365
> 2016-10-12 17:45:11,697 INFO [AM.ZK.Worker-pool2-t6]
> master.AssignmentManager: Skip assigning
> table1,3025965238305402_2,1468091325259.23e0186c4d2b5cc09f25de35fe174417.,
> it's host rs42.prod.foo.com,16020,1476293566365 is dead but not processed yet
> 2016-10-12 17:45:11,884 INFO [MASTER_SERVER_OPERATIONS-server01:16000-3]
> master.RegionStates: Transitioning {23e0186c4d2b5cc09f25de35fe174417
> state=CLOSED, ts=1476294311697, server=rs44.prod.foo.com,16020,1476294287692}
> will be handled by SSH for rs42.prod.foo.com,16020,1476293566365
> {noformat}
> 9. When the ServerShutdownHandling task reaches to this region, it also
> skipped the region in question. This was because this region was in RIT, and
> the ServerShutdownHandling task thinks that the AssignmentManager would
> assign it as part of handling the RIT nodes:
> {noformat}
> 2016-10-12 17:45:11,892 INFO [MASTER_SERVER_OPERATIONS-server01:16000-3]
> handler.ServerShutdownHandler: Skip assigning region in transition on other
> server{23e0186c4d2b5cc09f25de35fe174417 state=CLOSED, ts=1476294311697,
> server=rs44.prod.foo.com,16020,1476294287692}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
