wchevreuil commented on a change in pull request #740: HBASE-23197
'IllegalArgumentException: Wrong FS' on edits replay when…
URL: https://github.com/apache/hbase/pull/740#discussion_r347090421
##########
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
##########
@@ -314,9 +316,21 @@ public static void archiveStoreFiles(Configuration conf,
FileSystem fs, RegionIn
// build the archive path
if (regionInfo == null || family == null) throw new IOException(
"Need to have a region and a family to archive from.");
-
- Path storeArchiveDir = HFileArchiveUtil.getStoreArchivePath(conf,
regionInfo, tableDir, family);
-
+ //NOTE: This extra check is needed for the exceptional scenario where we
archive wal edits
+ // replayed, when hbase.region.archive.recovered.edits is on. Since WALs
may be configured to
+ // use different FS than root dir, we need to make sure to pick the proper
FS when deciding
+ // on the archiving path.
+ //1) If no wal dir setting, wals and root dir are on same FS, so good to
go with the root dir;
+ //2) When we have wal dir set it will only be on different FS if
"scheme://authority" is
+ // defined on wal path. In this case, if this is a proper store file
archiving call, the passed
+ // FS scheme will be different from the wal dir one, and you should pick
root dir as base.
+ String workingDir = conf.get(CommonFSUtils.HBASE_WAL_DIR);
+ if(workingDir == null || !workingDir.startsWith(fs.getScheme())){
Review comment:
So _archiveStoreFiles_ gets called by _HRegionFileSystem.removeStoreFiles_
and _MobUtils.removeMobFiles_. We are really interested on check
_HRegionFileSystem.removeStoreFiles_, since this is the one that eventually
get's passed a different FS from the actual store files one, when called from
_HRegion.replayRecoveredEditsIfAny_. There are then the possible combinations
of FS and file types to be archived:
1) We are archiving recovered edits, wals on different dir but same FS as
root dir:
this workingDir value is non null, but since it doesn't has an fs scheme
defined, we can assume it's on the same FS as the passed root dir. In this
case, we passed the WAL FS, back there on _HRegion.replayRecoveredEditsIfAny_,
but the WAL FS is same FS as the rootdir, so we are good to go with the normal
archive path computed from the root dir value.
2) We are archiving recovered edits, WALs configured on different FS than
the rootdir:
workinDir value is non null and it does start with the same scheme as the
passed FS (since we passed the wal FS here). This will create the path for
archiving on the WAL FS, as we are passing an schema + authority format.
3) Archiving store files with wal dir set to a separate dir on the same FS
as root dir:
Again workingDir is non null, but it will not have the scheme portion, so we
set workingDir back to the root dir value, which is what we want for store
files. This is the case being tested at
_TestHRegion.testArchiveRecoveredEditsReplay_, that's why we assert the files
are on a folder under original archive dir.
4) Archiving store files, wal dir set to a different FS:
Again workingDir is non null, but it's scheme is different from the passed
FS scheme, so we reset it to the rootdir value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
