[
https://issues.apache.org/jira/browse/HBASE-22826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-22826:
--------------------------------------
Description:
When WAL is attached to a separate file system, recovered.edits are going to
hbase root directory.
PROBLEM
* Customer environment
HBase root directory : On WASB
hbase.wal.dir : On HDFS
Customer is creating and HBase table and running VIEW DDL on top of the Hbase
table. The recovered.edits are going to hbase root directory in WASB and region
assignments getting failed.
Customer is on HBase 2.0.4.
The below stack trace is from local env reproduction:
{code:java}2019-08-05 22:07:31,940 ERROR
[RS_OPEN_META-regionserver/c47-node3:16020-0] handler.OpenRegionHandler: Failed
open of region=hbase:meta,,1.1588230740
java.lang.IllegalArgumentException: Wrong FS:
hdfs://c47-node2.squadron-labs.com:8020/hbasewal/hbase/meta/1588230740/recovered.edits,
expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:730)
at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86)
at
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:460)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)
at
org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:678)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:270)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)
at
org.apache.hadoop.hbase.wal.WALSplitter.getSequenceIdFiles(WALSplitter.java:647)
at
org.apache.hadoop.hbase.wal.WALSplitter.writeRegionSequenceIdFile(WALSplitter.java:680)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:984)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7149)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7108)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7080)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7038)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6989)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:283)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
was:
When WAL is attached to a separate file system, recovered.edits are going to
hbase root directory.
PROBLEM
* Customer environment
HBase root directory : On WASB
hbase.wal.dir : On HDFS
Customer is creating and HBase table and running VIEW DDL on top of the Hbase
table. The recovered.edits are going to hbase root directory in WASB and region
assignments getting failed.
Customer is on HBase 2.0.4.
{code:java}if (RegionReplicaUtil.isDefaultReplica(getRegionInfo())) {
LOG.debug("writing seq id for {}", this.getRegionInfo().getEncodedName());
WALSplitter.writeRegionSequenceIdFile(fs.getFileSystem(),
getWALRegionDir(), nextSeqId);
//WALSplitter.writeRegionSequenceIdFile(getWalFileSystem(),
getWALRegionDir(), nextSeqId - 1);{code}
{code:java}2019-08-05 22:07:31,940 ERROR
[RS_OPEN_META-regionserver/c47-node3:16020-0] handler.OpenRegionHandler: Failed
open of region=hbase:meta,,1.1588230740
java.lang.IllegalArgumentException: Wrong FS:
hdfs://c47-node2.squadron-labs.com:8020/hbasewal/hbase/meta/1588230740/recovered.edits,
expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:730)
at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86)
at
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:460)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)
at
org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:678)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:270)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)
at
org.apache.hadoop.hbase.wal.WALSplitter.getSequenceIdFiles(WALSplitter.java:647)
at
org.apache.hadoop.hbase.wal.WALSplitter.writeRegionSequenceIdFile(WALSplitter.java:680)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:984)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7149)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7108)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7080)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7038)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6989)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:283)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
> Wrong FS: recovered.edits goes to wrong file system
> ---------------------------------------------------
>
> Key: HBASE-22826
> URL: https://issues.apache.org/jira/browse/HBASE-22826
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 2.0.5
> Reporter: Vladimir Rodionov
> Assignee: Vladimir Rodionov
> Priority: Major
>
> When WAL is attached to a separate file system, recovered.edits are going to
> hbase root directory.
> PROBLEM
> * Customer environment
> HBase root directory : On WASB
> hbase.wal.dir : On HDFS
> Customer is creating and HBase table and running VIEW DDL on top of the Hbase
> table. The recovered.edits are going to hbase root directory in WASB and
> region assignments getting failed.
> Customer is on HBase 2.0.4.
> The below stack trace is from local env reproduction:
> {code:java}2019-08-05 22:07:31,940 ERROR
> [RS_OPEN_META-regionserver/c47-node3:16020-0] handler.OpenRegionHandler:
> Failed open of region=hbase:meta,,1.1588230740
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://c47-node2.squadron-labs.com:8020/hbasewal/hbase/meta/1588230740/recovered.edits,
> expected: file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:730)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:460)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:678)
> at
> org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:270)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.getSequenceIdFiles(WALSplitter.java:647)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.writeRegionSequenceIdFile(WALSplitter.java:680)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:984)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:881)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7149)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7108)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7080)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7038)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6989)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:283)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
