[
https://issues.apache.org/jira/browse/HBASE-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14008235#comment-14008235
]
stack commented on HBASE-11094:
-------------------------------
bq. Release Note: You have to drain all remaining split log work items before
rolling upgrade/change distributedLogReplay config
How does operator know when this has been done?
Did we decide the exception subpackage an anti-pattern? Or maybe it was ok for
new exceptions? (Im talking about location for
RegionServerConfigMismatchException).
+ * @param isOpenForReplay
Above is the name of the method, not the name of the attribute. Suggest rename
as openForReplay
Yeah, change this name too:
+ optional bool isOpenForDistributedLogReplay = 4;
Its not possible to mix old and new style log splitting in the one cluster?
So, IIRC, its the RS that does the replay of log edits? The master has already
opened the regions before it assigns out the split log task? Yeah, I suppose
its not possible for a mix of both types of replay in a cluster.
Worse case would be crash just after the Master had been upgraded? In this
case, all regionservers would respond that the regions could not be opened in
replay mode? Right?
Or, if a crash after the M and the RS have been rolling restarted. Only one RS
will be able to open regions. It could take a while for the M to figure this
out going by the below:
needNewPlan = true;
+ if(t instanceof RegionServerConfigMismatchException) {
+ // try another server & reset retry count
+ i--;
+ }
Could the rolling restart proceed during this time and in this way we'd get
ourselves out of the spot?
What happens on non-upgraded servers when we pass the code path that this is
inserted into?
+ // RS & Master have to see the same configuration values
+ if((slt.getMode() == ZooKeeperProtos.SplitLogTask.RecoveryMode.UNKNOWN)
+ || (slt.getMode() ==
ZooKeeperProtos.SplitLogTask.RecoveryMode.LOG_REPLAY &&
+ !HLogSplitter.isDistributedLogReplay(conf))
+ || (slt.getMode() ==
ZooKeeperProtos.SplitLogTask.RecoveryMode.LOG_SPLITTING &&
+ HLogSplitter.isDistributedLogReplay(conf))) {
+ LOG.debug("Didn't grab Task=" + path + " because recovery mode isn't
expected. Current " +
+ "task has recovery mode=" + slt.getMode());
+ return;
+ }
The solution is pretty clean. What would happen in the above scenarios?
We could go another version without turning on this feature but when we go to
turn it on the next time, will we run into a new issue that precludes our
making it the default?
What if we turn it on by default but add to the release notes that IF YOU ARE
GOING TO ROLLING RESTART, turn this off? That too exotic?
Good stuff lads.
> Distributed log replay is incompatible for rolling restarts
> -----------------------------------------------------------
>
> Key: HBASE-11094
> URL: https://issues.apache.org/jira/browse/HBASE-11094
> Project: HBase
> Issue Type: Sub-task
> Reporter: Enis Soztutar
> Assignee: Jeffrey Zhong
> Priority: Blocker
> Fix For: 0.99.0
>
> Attachments: hbase-11094-v2.patch, hbase-11094.patch
>
>
> 0.99.0 comes with dist log replay by default (HBASE-10888). However, reading
> the code and discussing this with Jeffrey, we realized that the dist log
> replay code is not compatible with rolling upgrades from 0.98.0 and 1.0.0.
> The issue is that, the region server looks at it own configuration to decide
> whether the region should be opened in replay mode or not. The open region
> RPC does not contain that info. So if dist log replay is enabled on master,
> the master will assign the region and schedule replay tasks. If the region is
> opened in a RS that does not have this conf enabled, then it will happily
> open the region in normal mode (not replay mode) causing possible (transient)
> data loss.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
