[
https://issues.apache.org/jira/browse/HBASE-23000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926862#comment-16926862
]
Rushabh S Shah commented on HBASE-23000:
----------------------------------------
> I had different test results when porting HBASE-22627. Not doubting the
> results here, but wondering if more is going on.
I am certain that HBASE-22627 caused these tests to fail. I checked out the
commit before and after this commit and the tests were succeeding before and
failing just after.
> Ok, well HBASE-22627 is a critical bug fix and cannot be reverted,
Agree with this statement.
I think I understand why these tests are failing.
The stack trace from logs:
{noformat}
WARNING! HBase file layout needs to be upgraded. You have version null and I
want version 8. Consult http:/
/hbase.apache.org/book.html for further information about upgrading HBase. Is
your hbase.rootdir valid? If
so, you may need to run 'hbase hbck -fixVersionFile'.
2019-09-10 12:03:30,031 FATAL [rushabhshah-ltm:58520.activeMasterManager]
master.HMaster$2(1891): Failed to become active master
org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout
needs to be upgraded. You have version null and I want version 8. Consult
http://hbase.apache.org/book.html for further information about upgrading
HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck
-fixVersionFile'.
at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:684)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:499)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:171)
at
org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:141)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:719)
at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:202)
at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1887)
at java.lang.Thread.run(Thread.java:748)
{noformat}
It is expecting to find hbase.version file under wal dir but it is unable to
find that file.
Before HBASE-22627, the code use to go into this method
[FSUtils.java#L667|https://github.com/apache/hbase/blob/branch-1.3/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java#L667]
since metaRegiondir didn't exist.
But after the change, metaRegionDir *does* exist because we write sequence file
to this directory while instantiating the HRegion. Refer to this
[FSUtils.java#L1227|https://github.com/apache/hbase/blob/branch-1.3/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java#L1227]
In short, the root cause the tests are failing
{code:title=FsUtils.java|borderStyle=solid}
public static Path getWALTableDir(final Configuration conf, final TableName
tableName)
throws IOException {
Path baseDir = new Path(getWALRootDir(conf),
HConstants.BASE_NAMESPACE_DIR); --- This line was introduce in HBASE-22627 that
is causing the issue.
return new Path(new Path(baseDir, tableName.getNamespaceAsString()),
tableName.getQualifierAsString());
{code}
Hope this explains the issue.
> Fix all consistently failing tests in branch-1.3
> ------------------------------------------------
>
> Key: HBASE-23000
> URL: https://issues.apache.org/jira/browse/HBASE-23000
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 1.3.6
> Reporter: Rushabh S Shah
> Assignee: Rushabh S Shah
> Priority: Major
>
> Flaky test report:
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1.3/Flaky_20Test_20Report/dashboard.html#job_2
> In last 30 builds this test failed all 30 times.
> Here is the stack trace:
> {noformat}
> Stacktrace
> java.io.IOException: Shutting down
> at
> org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428)
> Caused by: java.lang.RuntimeException: Master not initialized after 200000ms
> seconds
> at
> org.apache.hadoop.hbase.fs.TestBlockReorder.testBlockLocation(TestBlockReorder.java:428)
> {noformat}
> Link to latest jenkins build:
> https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-1.3/9351/testReport/org.apache.hadoop.hbase.fs/TestBlockReorder/testBlockLocation/
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
