[
https://issues.apache.org/jira/browse/HBASE-25902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367980#comment-17367980
]
Viraj Jasani commented on HBASE-25902:
--------------------------------------
After some digging, here is what I could come up with:
# For HBase 2, if we have to update CFs to meta, we better do it from master.
When we upgrade one RS to HBase 2 on an already running HBase 1 cluster and
when meta move to that RS, I believe we should not yet update Table descriptors
in HDFS. Let's give this privilege to master only because it has well defined
procedure to perform this task.
# After upgrading all RS to HBase 2.4, when master comes to 2.4, it starts
TableStateManager, which is responsible for migrating table state from
Zookeeper to meta table. This happens only when HBase 2 master comes up for the
first time. This is the first place where we query table CF, which is missing
in meta table. So what do we do? We can park this migration for now and let it
be completed "next time" (step 6 to cover this). We don't want to add any hack,
let next active master init take care of this.
# Now active master initialization goes through the rest of the routine:
## Create assign procedures for offline regions
## Wake up suspended procedures
## Update meta region replica if required
## Scan meta and create snapshot of region assignment
## Start all mandatory chores
## Wait for namespace region to come online
## Initialize Cluster Schema service
## Pre master init hook in Coproc
## Initialize Register observers
## Set master as *Initialized*
# In above steps, while initializing Cluster Schema service (sub-step 7), we
face issue again because it tries to start TableNamespace manager, which tries
to check if namespace table is enabled (using table CF in meta) and we again
get NSCFE. It's important that we bypass this because master is not yet marked
as Initialized and without it being Initialized, using ModifyTable procedure is
of no good use to us. It should be started only after master is marked
*Initialized* (and we don't want to mess this up by introducing any hacks).
# Now, after master is marked Initialized, we submit Modify table procedure
and wait for it to complete, which consist of these sub-steps (or child
procedures):
## Pre-flight checks
## Prepare to modify : check Table descriptor if necessary
## Unassign procedure for closing excess replicas if necessary
## Update Table descriptor in master's in-memory, as well as in .tabledesc in
HDFS
## Remove replica cols in necessary
## Run Coproc actions
## Reopen regions procedure
## Assign procedure for replicas if necessary
## Remove CFs from HDFS if necessary
# We are done with adding missing CFs in meta table. However, as mentioned in
above steps, we have bypassed two of the important steps just so that we can
let master state be completely initialized and then submit Modify table
procedure. Hence, now we need to abort active master itself. This is one time
abort, and when new active master takes over, it won't get missing CFs in meta
while finishing initialization of itself and hence, all steps will be completed
(no bypass of failures this time).
Please take a look at [https://github.com/apache/hbase/pull/3417]
Thanks
> 1.x to 2.3.x upgrade does not work; you must install an hbase2 that is
> earlier than hbase-2.3.0 first
> -----------------------------------------------------------------------------------------------------
>
> Key: HBASE-25902
> URL: https://issues.apache.org/jira/browse/HBASE-25902
> Project: HBase
> Issue Type: Bug
> Components: meta, Operability
> Affects Versions: 2.3.0, 2.4.0
> Reporter: Michael Stack
> Priority: Critical
> Attachments: NoSuchColumnFamilyException.png
>
>
> Making note of this issue in case others run into it. At my place of employ,
> we tried to upgrade a cluster that was an hbase-1.2.x version to an
> hbase-2.3.5 but it failed because meta didn't have the 'table' column family.
> Up to 2.3.0, hbase:meta was hardcoded. HBASE-12035 added the 'table' CF for
> hbase-2.0.0. HBASE-23782 (2.3.0) undid hardcoding of the hbase:meta schema;
> i.e. reading hbase:meta schema from the filesystem. The hbase:meta schema is
> only created on initial install. If an upgrade over existing data, the
> hbase-1 hbase:meta will not be suitable for hbase-2.3.x context as it will be
> missing columnfamilies needed to run (HBASE-23055 made it so hbase:meta could
> be altered (2.3.0) but probably of no use since Master won't come up).
> It would be a nice-to-have if a user could go from hbase1 to hbase.2.3.0 w/o
> having to first install an hbase2 that is earlier than 2.3.0 but needs to be
> demand before we would work on it; meantime, install an intermediate hbase2
> version before going to hbase-2.3.0+ if coming from hbase-1.x
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
