[
https://issues.apache.org/jira/browse/HBASE-23197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956193#comment-16956193
]
Josh Elser commented on HBASE-23197:
------------------------------------
Thanks for filing this, Wellington. Just to be crystal clear: not a data loss
scenario -- just limited to the code which tries to archive the recovered.edits
(which we have config for to help identify data loss issues)
Looked like an inadvertent problem that comes from us reusing HFile archiving
code to archive the recovered.edits which are on the WAL fs (not the "normal"
fs). This results in the helper methods using a normal FS path (e.g. s3a://..)
on the WAL fs object (e.g. hdfs://...).
> "IllegalArgumentException: Wrong FS" on edits replay when WALs on different
> file system and hbase.region.archive.recovered.edits is enabled.
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-23197
> URL: https://issues.apache.org/jira/browse/HBASE-23197
> Project: HBase
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Major
>
> This was caught by execution tests from our [~romil.choksi], when running
> hbase with WALs on hdfs, HFiles on S3. Upon initial analysis from [~elserj]
> of below stack trace, we concluded this is triggered by enabling
> *hbase.region.archive.recovered.edits* property.
> {noformat}
> ***** ABORTING region server my-rs-1.example.com,16020,1571335836583: Failed
> to open region hbase:meta,,1.1588230740 and can not recover *****
> Cause:
> java.lang.IllegalArgumentException: Wrong FS:
> s3a://qe-s3-bucket-mst/cc-hbase-191017-113225-e6x-dw-4/hbase/archive/data/hbase/meta/1588230740/recovered.edits,
> expected: hdfs://my-hdfs-1.example.com:8020
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:742)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:234)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1323)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1320)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1337)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1312)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2292)
> at
> org.apache.hadoop.hbase.backup.HFileArchiver.archiveStoreFiles(HFileArchiver.java:321)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.removeStoreFiles(HRegionFileSystem.java:505)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:4626)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:968)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:915)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7320)
> {noformat}
> This property is available mainly for debugging purposes, and it wraps
> recovered edits over store files for re-using the StoreFiles archiving logic.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
