[
https://issues.apache.org/jira/browse/HBASE-13337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-13337:
--------------------------------
Attachment: HBASE-13337.patch
Attached a patch that checks for such a scenario in case a region is forced to
assign with a new plan. [~sreenivasulureddy], could you give it a try?
> Table regions are not assigning back, after restarting all regionservers at
> once.
> ---------------------------------------------------------------------------------
>
> Key: HBASE-13337
> URL: https://issues.apache.org/jira/browse/HBASE-13337
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
> Affects Versions: 2.0.0
> Reporter: Y. SREENIVASULU REDDY
> Assignee: Jimmy Xiang
> Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-13337.patch
>
>
> Regions of the table are continouly in state=FAILED_CLOSE.
> {noformat}
> Region State
>
> RIT time (ms)
> 8f62e819b356736053e06240f7f7c6fd
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd.
> state=FAILED_CLOSE, ts=Thu Mar 26 15:05:36 IST 2015 (113s ago),
> server=VM1,16040,1427362531818 113929
> caf59209ae65ea80fca6bdc6996a7d68
> t1,dddddddd,1427362431330.caf59209ae65ea80fca6bdc6996a7d68.
> state=FAILED_CLOSE, ts=Thu Mar 26 15:05:36 IST 2015 (113s ago),
> server=VM2,16040,1427362533691 113929
> db52a74988f71e5cf257bbabf31f26f3
> t1,44444444,1427362431330.db52a74988f71e5cf257bbabf31f26f3.
> state=FAILED_CLOSE, ts=Thu Mar 26 15:05:36 IST 2015 (113s ago),
> server=VM3,16040,1427362533691 113920
> 43f3a65b9f9ff283f598c5450feab1f8
> t1,88888888,1427362431330.43f3a65b9f9ff283f598c5450feab1f8.
> state=FAILED_CLOSE, ts=Thu Mar 26 15:05:36 IST 2015 (113s ago),
> server=VM1,16040,1427362531818 113920
> {noformat}
> *Steps to reproduce:*
> 1. Start HBase cluster with more than one regionserver.
> 2. Create a table with precreated regions. (lets say 15 regions)
> 3. Make sure the regions are well balanced.
> 4. Restart all the Regionservers process at once across the cluster, except
> HMaster process
> 5. After restarting the Regionservers, successfully will connect to the
> HMaster.
> *Bug:*
> But no regions are assigning back to the Regionservers.
> *Master log shows as follows:*
> {noformat}
> 2015-03-26 15:05:36,201 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStates: Transition {8f62e819b356736053e06240f7f7c6fd
> state=OFFLINE, ts=1427362536106, server=VM2,16040,1427362242602} to
> {8f62e819b356736053e06240f7f7c6fd state=PENDING_OPEN, ts=1427362536201,
> server=VM1,16040,1427362531818}
> 2015-03-26 15:05:36,202 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStateStore: Updating row
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd. with
> state=PENDING_OPEN&sn=VM1,16040,1427362531818
> 2015-03-26 15:05:36,244 DEBUG [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Force region state offline
> {8f62e819b356736053e06240f7f7c6fd state=PENDING_OPEN, ts=1427362536201,
> server=VM1,16040,1427362531818}
> 2015-03-26 15:05:36,244 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStates: Transition {8f62e819b356736053e06240f7f7c6fd
> state=PENDING_OPEN, ts=1427362536201, server=VM1,16040,1427362531818} to
> {8f62e819b356736053e06240f7f7c6fd state=PENDING_CLOSE, ts=1427362536244,
> server=VM1,16040,1427362531818}
> 2015-03-26 15:05:36,244 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStateStore: Updating row
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd. with
> state=PENDING_CLOSE
> 2015-03-26 15:05:36,248 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=1 of 10
> 2015-03-26 15:05:36,248 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=2 of 10
> 2015-03-26 15:05:36,249 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=3 of 10
> 2015-03-26 15:05:36,249 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=4 of 10
> 2015-03-26 15:05:36,249 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=5 of 10
> 2015-03-26 15:05:36,250 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=6 of 10
> 2015-03-26 15:05:36,250 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=7 of 10
> 2015-03-26 15:05:36,250 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=8 of 10
> 2015-03-26 15:05:36,251 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=9 of 10
> 2015-03-26 15:05:36,251 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=10 of 10
> 2015-03-26 15:05:36,251 WARN [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStates: Failed to open/close 8f62e819b356736053e06240f7f7c6fd on
> VM1,16040,1427362531818, set to FAILED_CLOSE
> 2015-03-26 15:05:36,251 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStates: Transition {8f62e819b356736053e06240f7f7c6fd
> state=PENDING_CLOSE, ts=1427362536244, server=VM1,16040,1427362531818} to
> {8f62e819b356736053e06240f7f7c6fd state=FAILED_CLOSE, ts=1427362536251,
> server=VM1,16040,1427362531818}
> 2015-03-26 15:05:36,251 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStateStore: Updating row
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd. with
> state=FAILED_CLOSE
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
