[
https://issues.apache.org/jira/browse/HADOOP-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Xu updated HADOOP-12239:
----------------------------
Status: Patch Available (was: In Progress)
> StorageException complaining " no lease ID" when updating
> FolderLastModifiedTime in WASB
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-12239
> URL: https://issues.apache.org/jira/browse/HADOOP-12239
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
> Affects Versions: 2.7.0
> Reporter: Duo Xu
> Assignee: Duo Xu
> Attachments: HADOOP-12239.01.patch
>
>
> This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a
> customer's HBase cluster logs, but the piece of code is in a different place.
> {code}
> 2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager:
> dead splitlog workers
> [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180]
> 2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler:
> Caught throwable while processing event M_SERVER_SHUTDOWN
> java.io.IOException: failed log splitting for
> workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374, will retry
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unable to write RenamePending file for folder
> rename from
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to
> hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288)
> at
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204)
> ... 4 more
> Caused by: org.apache.hadoop.fs.azure.AzureException:
> com.microsoft.azure.storage.StorageException: There is currently a lease on
> the blob and no lease ID was specified in the request.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255)
> ... 11 more
> Caused by: com.microsoft.azure.storage.StorageException: There is currently a
> lease on the blob and no lease ID was specified in the request.
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
> at
> com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:307)
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:182)
> at
> com.microsoft.azure.storage.blob.CloudBlob.uploadProperties(CloudBlob.java:2892)
> at
> org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.uploadProperties(StorageInterfaceImpl.java:372)
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2593)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
