[
https://issues.apache.org/jira/browse/HBASE-20792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526320#comment-16526320
]
Duo Zhang commented on HBASE-20792:
-----------------------------------
A bit strange
this is what we have on master procedure page
{noformat}
ModifyTableProcedure table=IntegrationTestBigLinkedList Thu Jun 28 21:42:05 CST
2018 Thu Jun 28 21:42:28 CST 2018
{noformat}
The procedure is started at 21:42:05 and finished at 21:42:28.
But in the log for ITBLL, at 21:42:05 we have already failed...
{noformat}
2018-06-28 21:42:05,276 WARN [ChaosMonkey] policies.Policy: ionPolicy.java:59)
at
org.apache.hadoop.hbase.chaos.policies.PeriodicPolicy.run(PeriodicPolicy.java:41)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: The operation: MODIFY on
table: IntegrationTestBigLinkedList has not completed after 613179ms
at
org.apache.hadoop.hbase.client.HBaseAdmin$TableFuture$TableWaitForStateCallable.throwTimeoutException(HBaseAdmin.java:3661)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitForState(HBaseAdmin.java:3603)
at
org.apache.hadoop.hbase.client.HBaseAdmin$TableFuture.waitForSchemaUpdate(HBaseAdmin.java:3719)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ModifyTableFuture.postOperationResult(HBaseAdmin.java:1957)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ModifyTableFuture.postOperationResult(HBaseAdmin.java:1935)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:3429)
at org.apache.hadoop.hbase.client.HBaseAdmin.get(HBaseAdmin.java:2150)
... 5 more
{noformat}
And the log is messed up so it is a bit hard to find out the root cause... And
maybe we'd better add the procedure id in the exception message for better
debugging?
> info:servername and info:sn inconsistent for OPEN region
> --------------------------------------------------------
>
> Key: HBASE-20792
> URL: https://issues.apache.org/jira/browse/HBASE-20792
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
> Reporter: Josh Elser
> Assignee: Josh Elser
> Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: HBASE-20792.patch, TestRegionMoveAndAbandon.java,
> hbase-hbase-master-ctr-e138-1518143905142-380753-01-000004.hwx.site.log
>
>
> Next problem we've run into after HBASE-20752 and HBASE-20708
> After a rolling restart of a cluster, we'll see situations where a collection
> of regions will simply not be assigned out to the RS. I was able to reproduce
> this my mimic the restart patterns our tests do internally (ignore whether
> this is the best way to restart nodes for now :)). The general pattern is
> this:
> {code:java}
> for rs in regionservers:
> stop(server, rs, RS)
> for master in masters:
> stop(server, master, MASTER)
> sleep(15)
> for master in masters:
> start(server, master, MASTER)
> for rs in regionservers:
> start(server, rs, RS){code}
> Looking at meta, we can see why the Master is ignoring some regions:
> {noformat}
> test
> column=table:state, timestamp=1529871718998, value=\x08\x00
> test,,1529871718122.0297f680df6dc0166a44f9536346268e.
> column=info:regioninfo, timestamp=1529967103390, value={ENCODED =>
> 0297f680df6dc0166a44f9536346268e, NAME =>
> 'test,,1529871718122.0297f680df6dc0166a44f9536346268e.', STARTKEY
> => '', ENDKEY =>
> ''}
> test,,1529871718122.0297f680df6dc0166a44f9536346268e.
> column=info:seqnumDuringOpen, timestamp=1529967103390,
> value=\x00\x00\x00\x00\x00\x00\x00*
> test,,1529871718122.0297f680df6dc0166a44f9536346268e.
> column=info:server, timestamp=1529967103390,
> value=ctr-e138-1518143905142-378097-02-000012.hwx.site:16020
> test,,1529871718122.0297f680df6dc0166a44f9536346268e.
> column=info:serverstartcode, timestamp=1529967103390, value=1529966776248
> test,,1529871718122.0297f680df6dc0166a44f9536346268e. column=info:sn,
> timestamp=1529967096482,
> value=ctr-e138-1518143905142-378097-02-000006.hwx.site,16020,1529966755170
> test,,1529871718122.0297f680df6dc0166a44f9536346268e.
> column=info:state, timestamp=1529967103390, value=OPEN{noformat}
> The region is marked as {{OPEN}}. The master doesn't know any better.
> However, the interesting bit is that {{info:server}} and {{info:sn}} are
> inconsistent (which, according to the javadoc should not be possible for an
> {{OPEN}} region).{{}}
> This doesn't happen every time, but I caught it yesterday on the 2nd or 3rd
> attempt, so I'm hopeful it's not a bear to repro.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
