[
https://issues.apache.org/jira/browse/HBASE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14994605#comment-14994605
]
Matteo Bertozzi edited comment on HBASE-14783 at 11/6/15 11:03 PM:
-------------------------------------------------------------------
there are 3 problems here
- on a not clean shutdown your cluster is in a non consistent state, starting
the master and ignoring the exception is a bad idea. you have run hbck or you
can restart the cluster and do a clean shutdown before the downgrade.
- on a clean shutdown and restart with lower version you will not have
procedure running. so nothing to load and you'll not hit this problem
the 3rd problem is related to how we load, completed procedures (but not yet
deleted) are loaded using convert() which creates a Procedure instance, but we
don't really need that. in fact we remove that instance later when we realize
that we just need the result. so even in case of a clean shutdown we may get
the exception because we are trying to call convert().
was (Author: mbertozzi):
there are 3 problems here
- on a not clean shutdown your cluster is in a non consistent state, starting
the master and ignoring the exception is a bad idea. you have run hbck or you
can restart the cluster and do a clean shutdown before the downgrade.
- on a clean shutdown and restart with lower version you will not have
procedure running. so nothing to load and you'll not hit this problem
the 3rd is how do we load, completed procedures (but not yet deleted) are
loaded using convert() which creates a Procedure instance, but we don't really
need that. in fact we remove that instance later when we realize that we just
need the result.
> Proc-V2: Master aborts when downgrading from 1.3 to 1.1
> -------------------------------------------------------
>
> Key: HBASE-14783
> URL: https://issues.apache.org/jira/browse/HBASE-14783
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Stephen Yuan Jiang
>
> I was running ITBLL with 1.3 deployed on a 6 node cluster.
> Then I stopped the cluster, deployed 1.1 release and tried to start cluster.
> However, master failed to start due to:
> {code}
> 2015-11-06 00:58:40,351 FATAL [eval-test-2:20000.activeMasterManager]
> master.HMaster: Failed to become active master
> java.io.IOException: The procedure class
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be
> accessible and have an empty constructor
> at
> org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:548)
> at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:640)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:105)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:82)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:298)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:275)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:434)
> at
> org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1208)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1107)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:694)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:186)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1713)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:191)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:536)
> ... 12 more
> {code}
> The cause was that ServerCrashProcedure, written in some WAL file under
> MasterProcWALs from first run, was absent in 1.1 release.
> After a brief discussion with Stephen, I am logging this JIRA to solicit
> discussion on how customer experience can be improved if downgrade of hbase
> is performed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
