[ https://issues.apache.org/jira/browse/HADOOP-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Duo Xu updated HADOOP-12239: ---------------------------- Description: This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a customer's HBase cluster logs, but the piece of code is in a different place. {code} 2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager: dead splitlog workers [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180] 2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN java.io.IOException: failed log splitting for workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374, will retry at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Unable to write RenamePending file for folder rename from hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998) at org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204) ... 4 more Caused by: org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request. at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598) at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775) at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255) ... 11 more Caused by: com.microsoft.azure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request. at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89) at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:307) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:182) at com.microsoft.azure.storage.blob.CloudBlob.uploadProperties(CloudBlob.java:2892) at org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.uploadProperties(StorageInterfaceImpl.java:372) at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2593) ... 19 more {code} was: This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a customer's HBase cluster logs, but the piece of code is in a different place. {code} 2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager: dead splitlog workers [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180] 2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN java.io.IOException: failed log splitting for workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374, will retry at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Unable to write RenamePending file for folder rename from hbase/WALs/workernode12.skypedataprdhbaeus00.b6.internal.cloudapp.net,60020,1436448566374 to hbase/WALs/workernode12.skypedataprdhbaeus00.b6.internal.cloudapp.net,60020,1436448566374-splitting at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998) at org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204) ... 4 more Caused by: org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request. at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598) at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775) at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255) ... 11 more Caused by: com.microsoft.azure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request. at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89) at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:307) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:182) at com.microsoft.azure.storage.blob.CloudBlob.uploadProperties(CloudBlob.java:2892) at org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.uploadProperties(StorageInterfaceImpl.java:372) at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2593) ... 19 more {code} > StorageException complaining " no lease ID" when updating > FolderLastModifiedTime in WASB > ---------------------------------------------------------------------------------------- > > Key: HADOOP-12239 > URL: https://issues.apache.org/jira/browse/HADOOP-12239 > Project: Hadoop Common > Issue Type: Bug > Components: tools > Affects Versions: 2.7.0 > Reporter: Duo Xu > Assignee: Duo Xu > > This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a > customer's HBase cluster logs, but the piece of code is in a different place. > {code} > 2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager: > dead splitlog workers > [workernode3.xxx.b6.internal.cloudapp.net,60020,1436448555180] > 2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler: > Caught throwable while processing event M_SERVER_SHUTDOWN > java.io.IOException: failed log splitting for > workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374, will retry > at > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343) > at > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: Unable to write RenamePending file for folder > rename from > hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374 to > hbase/WALs/workernode12.xxx.b6.internal.cloudapp.net,60020,1436448566374-splitting > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258) > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110) > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998) > at > org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325) > at > org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412) > at > org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390) > at > org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288) > at > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204) > ... 4 more > Caused by: org.apache.hadoop.fs.azure.AzureException: > com.microsoft.azure.storage.StorageException: There is currently a lease on > the blob and no lease ID was specified in the request. > at > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598) > at > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609) > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366) > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775) > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255) > ... 11 more > Caused by: com.microsoft.azure.storage.StorageException: There is currently a > lease on the blob and no lease ID was specified in the request. > at > com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89) > at > com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:307) > at > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:182) > at > com.microsoft.azure.storage.blob.CloudBlob.uploadProperties(CloudBlob.java:2892) > at > org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.uploadProperties(StorageInterfaceImpl.java:372) > at > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2593) > ... 19 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)