[
https://issues.apache.org/jira/browse/HADOOP-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182586#comment-15182586
]
Liu Shaohui commented on HADOOP-12885:
--------------------------------------
[~madhuch-ms]
{quote}
Could you please share the logs, a cluster where you are able to repro this
issue and the storage account details of the cluster?
{quote}
The storage account is xiaomitst2 in Central India and the container is named
xiaomi. The state of container didn't be changed after this issue. The
RenamePending.json(/hbase/azurtst-xiaomi/data/default/YCSBTest-RenamePending.json)
is still there.
{quote}
And are you getting this error when the first rename happened, or after some
crash happened, and we are trying to redo the pending renames?
{quote}
The exception when the first rename happened
{code}
2016-02-29,05:58:57,780 ERROR
org.apache.hadoop.hbase.master.handler.TableEventHandler: Error manipulating
table YCSBTest
org.apache.hadoop.fs.azure.AzureException: Source blob
hbase/azurtst-xiaomi/data/default/YCSBTest/5f882f5492c90b4c03a26561a2ee0a96/.regioninfo
does not exist.
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2405)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:413)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1997)
at
org.apache.hadoop.hbase.master.MasterFileSystem.moveTableToTemp(MasterFileSystem.java:586)
at
org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:105)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:129)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
When we are trying to redo the pending renames, it encounter the bug mentioned
in HDFS-9897
{code}
java.io.IOException: Attempting to complete rename of file
hbase/azurtst-xiaomi/data/default/YCSBTest/.tabledesc during folder rename
redo, and file was not found in source or destination.
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.finishSingleFileRename(NativeAzureFileSystem.java:561)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.redo(NativeAzureFileSystem.java:511)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.conditionalRedoFolderRenames(NativeAzureFileSystem.java:1743)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.listStatus(NativeAzureFileSystem.java:1689)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555)
at
org.apache.hadoop.hbase.util.FSUtils.getLocalTableDirs(FSUtils.java:1333)
at org.apache.hadoop.hbase.util.FSUtils.getTableDirs(FSUtils.java:1318)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:206)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:920)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:711)
at java.lang.Thread.run(Thread.java:745)
{code}
> An operation of atomic fold rename crashed in Wasb FileSystem
> -------------------------------------------------------------
>
> Key: HADOOP-12885
> URL: https://issues.apache.org/jira/browse/HADOOP-12885
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Liu Shaohui
> Assignee: madhumita chakraborty
> Priority: Critical
>
> An operation of atomic fold rename crashed in Wasb FileSystem
> {code}
> org.apache.hadoop.fs.azure.AzureException: Source blob
> hbase/azurtst-xiaomi/data/default/YCSBTest/5f882f5492c90b4c03a26561a2ee0a96/.regioninfo
> does not exist.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2405)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:413)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1997)
> {code}
> The problem is that there are duplicated file is the RenamePending.json.
> {code}
> "5f882f5492c90b4c03a26561a2ee0a96",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/.regioninfo",
>
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/8a2c08db432d447d9e0ed5266940b25e",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/9425c621073e41df9430e88f0ef61c01",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/f9fc55a94fa34efbb2d26be77c76187c",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/.regioninfo",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/.tmp",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/8a2c08db432d447d9e0ed5266940b25e",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/9425c621073e41df9430e88f0ef61c01",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/f9fc55a94fa34efbb2d26be77c76187c",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/recovered.edits",
> {code}
> Maybe there is a bug in the Listing of all the files in the folder in Wasb.
> Any suggestions?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
