[jira] [Commented] (HADOOP-12885) An operation of atomic fold rename crashed in Wasb FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15182571#comment-15182571
]
madhumita chakraborty commented on HADOOP-12885:
[~liushaohui] Some more doubts
How often you are facing this? Are you seeing this frequently across many
clusters in production?
> An operation of atomic fold rename crashed in Wasb FileSystem
> -
>
> Key: HADOOP-12885
> URL: https://issues.apache.org/jira/browse/HADOOP-12885
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: madhumita chakraborty
>Priority: Critical
>
> An operation of atomic fold rename crashed in Wasb FileSystem
> {code}
> org.apache.hadoop.fs.azure.AzureException: Source blob
> hbase/azurtst-xiaomi/data/default/YCSBTest/5f882f5492c90b4c03a26561a2ee0a96/.regioninfo
> does not exist.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2405)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:413)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1997)
> {code}
> The problem is that there are duplicated file is the RenamePending.json.
> {code}
> "5f882f5492c90b4c03a26561a2ee0a96",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/.regioninfo",
>
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/8a2c08db432d447d9e0ed5266940b25e",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/9425c621073e41df9430e88f0ef61c01",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/f9fc55a94fa34efbb2d26be77c76187c",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/.regioninfo",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/.tmp",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/8a2c08db432d447d9e0ed5266940b25e",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/9425c621073e41df9430e88f0ef61c01",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/f9fc55a94fa34efbb2d26be77c76187c",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/recovered.edits",
> {code}
> Maybe there is a bug in the Listing of all the files in the folder in Wasb.
> Any suggestions?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12885) An operation of atomic fold rename crashed in Wasb FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15182536#comment-15182536
]
madhumita chakraborty commented on HADOOP-12885:
[~liushaohui] Could you please share the logs, a cluster where you are able to
repro this issue and the storage account details of the cluster? And are you
getting this error when the first rename happened, or after some crash
happened, and we are trying to redo the pending renames?
> An operation of atomic fold rename crashed in Wasb FileSystem
> -
>
> Key: HADOOP-12885
> URL: https://issues.apache.org/jira/browse/HADOOP-12885
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: madhumita chakraborty
>Priority: Critical
>
> An operation of atomic fold rename crashed in Wasb FileSystem
> {code}
> org.apache.hadoop.fs.azure.AzureException: Source blob
> hbase/azurtst-xiaomi/data/default/YCSBTest/5f882f5492c90b4c03a26561a2ee0a96/.regioninfo
> does not exist.
> at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2405)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:413)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1997)
> {code}
> The problem is that there are duplicated file is the RenamePending.json.
> {code}
> "5f882f5492c90b4c03a26561a2ee0a96",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/.regioninfo",
>
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/8a2c08db432d447d9e0ed5266940b25e",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/9425c621073e41df9430e88f0ef61c01",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/f9fc55a94fa34efbb2d26be77c76187c",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/.regioninfo",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/.tmp",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/8a2c08db432d447d9e0ed5266940b25e",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/9425c621073e41df9430e88f0ef61c01",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/C\/f9fc55a94fa34efbb2d26be77c76187c",
>
> "5f882f5492c90b4c03a26561a2ee0a96\/recovered.edits",
> {code}
> Maybe there is a bug in the Listing of all the files in the folder in Wasb.
> Any suggestions?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12717) NPE when trying to rename a directory in Windows Azure Storage FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175182#comment-15175182 ] madhumita chakraborty commented on HADOOP-12717: Looks good to me. Would it be possible to add one unit test for this scenario? > NPE when trying to rename a directory in Windows Azure Storage FileSystem > - > > Key: HADOOP-12717 > URL: https://issues.apache.org/jira/browse/HADOOP-12717 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Robert Yokota >Assignee: Robert Yokota > Attachments: HADOOP-12717.001.patch, diff.txt > > > Encountered an NPE when trying to use the HBase utility ExportSnapshot with > Azure as the target. > It turns out verifyAndConvertToStandardFormat is returning null when > determining the hbaseRoot, and this is being added to the atomicRenameDirs > set. > java.lang.NullPointerException > at > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.isKeyForDirectorySet(AzureNativeFileSystemStore.java:1059) > at > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.isAtomicRenameKey(AzureNativeFileSystemStore.java:1053) > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2098) > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1996) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:944) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > com.yammer.calmie.snapshot.AbstractSnapshotUtil.exportSnapshot(AbstractSnapshotUtil.java:210) > at > com.yammer.calmie.snapshot.AbstractSnapshotUtil.run(AbstractSnapshotUtil.java:79) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > com.yammer.calmie.snapshot.SnapshotAzureBlobUtil.main(SnapshotAzureBlobUtil.java:85) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12853) Change WASB documentation regarding page blob support
[ https://issues.apache.org/jira/browse/HADOOP-12853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12853: --- Attachment: HADOOP-12853.001.patch > Change WASB documentation regarding page blob support > - > > Key: HADOOP-12853 > URL: https://issues.apache.org/jira/browse/HADOOP-12853 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Minor > Attachments: HADOOP-12853.001.patch > > > WASB page blob support documentation in Features list at the top: > Supports both page blobs (suitable for most use cases, such as MapReduce) and > block blobs (suitable for continuous write use cases, such as an HBase > write-ahead log). > Which is actually the opposite. Block blobs are better for typical big data > use cases, and page blobs were implemented to support the HBase WAL. > It is also mentioned that > "Page blobs can be used for other purposes beyond just HBase log files > though." > This is not strong enough because page blob has been tested only in context > of HBase write ahead logs. They have not been broadly certified for use > across the HDP stack, hence is not recommended for use as a general purpose > file-system via Hadoop cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12853) Change WASB documentation regarding page blob support
[ https://issues.apache.org/jira/browse/HADOOP-12853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12853: --- Status: Patch Available (was: Open) > Change WASB documentation regarding page blob support > - > > Key: HADOOP-12853 > URL: https://issues.apache.org/jira/browse/HADOOP-12853 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Minor > Attachments: HADOOP-12853.001.patch > > > WASB page blob support documentation in Features list at the top: > Supports both page blobs (suitable for most use cases, such as MapReduce) and > block blobs (suitable for continuous write use cases, such as an HBase > write-ahead log). > Which is actually the opposite. Block blobs are better for typical big data > use cases, and page blobs were implemented to support the HBase WAL. > It is also mentioned that > "Page blobs can be used for other purposes beyond just HBase log files > though." > This is not strong enough because page blob has been tested only in context > of HBase write ahead logs. They have not been broadly certified for use > across the HDP stack, hence is not recommended for use as a general purpose > file-system via Hadoop cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12853) Change WASB documentation regarding page blob support
madhumita chakraborty created HADOOP-12853: -- Summary: Change WASB documentation regarding page blob support Key: HADOOP-12853 URL: https://issues.apache.org/jira/browse/HADOOP-12853 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Reporter: madhumita chakraborty Assignee: madhumita chakraborty Priority: Minor WASB page blob support documentation in Features list at the top: Supports both page blobs (suitable for most use cases, such as MapReduce) and block blobs (suitable for continuous write use cases, such as an HBase write-ahead log). Which is actually the opposite. Block blobs are better for typical big data use cases, and page blobs were implemented to support the HBase WAL. It is also mentioned that "Page blobs can be used for other purposes beyond just HBase log files though." This is not strong enough because page blob has been tested only in context of HBase write ahead logs. They have not been broadly certified for use across the HDP stack, hence is not recommended for use as a general purpose file-system via Hadoop cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Status: Patch Available (was: Open)
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch,
> HADOOP-12535.003.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Status: Open (was: Patch Available)
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch,
> HADOOP-12535.003.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159609#comment-15159609
]
madhumita chakraborty commented on HADOOP-12535:
Took care if the comments and uploaded new patch.
[~cnauroth] Some doubts
1. How do atomic rename and delete settings
(fs.contract.supports-atomic-directory-delete and
fs.contract.supports-atomic-directory-rename) get used? I dont see any
reference to SUPPORTS_ATOMIC_RENAME and SUPPORTS_ATOMIC_DIRECTORY_DELETE
defined in ContractOptions. If a filesystem supports atomic rename, do they
need to write new contract test?
2. I added contract test for append. testRenameFileBeingAppended if
failing. I have skipped this test for now. This test case appends a file and
then renames it.
FSDataOutputStream outputStream = getFileSystem().append(target);
outputStream.write(dataset);
Path renamed = new Path(testPath,”renamed”);
Rename(target, renamed);
outputStream.close();
So here we see that after renaming it closes the original file stream. Was this
test written in this way to maintain parity with unix file system? But WASB
does not support this semantic. If the file is renamed before closing the
original stream then we get 2 types of error.
a. In append we take SelfRenewingLease on the blob. In Close we see if the
lease has expired it will try to renew the lease and that will throw exception
as the blob has already been deleted as part of rename.
b. In close we commit the blocks of blockblob. So if we rename the file
before closing the stream, actually the content does not get committed. And
empty renamed file is created.
cc [~dchickabasapa]
3.Root folder access is not fully supported in WASB. So removed
TestAzureNativeContractRootDir.
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch,
> HADOOP-12535.003.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Attachment: HADOOP-12535.003.patch
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch,
> HADOOP-12535.003.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Status: Patch Available (was: Open)
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch,
> HADOOP-12535.003.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Status: Open (was: Patch Available)
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152308#comment-15152308
]
madhumita chakraborty commented on HADOOP-12535:
As we should not commit the WASB account credentials, so I disabled the tests
according to
https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/filesystem/testing.html
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Status: Patch Available (was: Open)
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Attachment: (was: HADOOP-12535.002.patch)
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Attachment: HADOOP-12535.002.patch
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Status: Open (was: Patch Available)
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Status: Patch Available (was: Open)
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Status: Open (was: Patch Available)
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12780) During atomic rename handle crash when one directory has been renamed but not file under it.
[
https://issues.apache.org/jira/browse/HADOOP-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15139632#comment-15139632
]
madhumita chakraborty commented on HADOOP-12780:
[~cnauroth] Could you please take a look at the patch?
> During atomic rename handle crash when one directory has been renamed but not
> file under it.
>
>
> Key: HADOOP-12780
> URL: https://issues.apache.org/jira/browse/HADOOP-12780
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: madhumita chakraborty
>Assignee: madhumita chakraborty
>Priority: Critical
> Attachments: HADOOP-12780.001.patch
>
>
> During atomic folder rename process preperaion we record the proposed change
> to a metadata file (-renamePending.json).
> Say we are renaming parent/folderToRename to parent/renamedFolder.
> folderToRename has an inner folder innerFolder and innerFolder has a file
> innerFile
> Content of the –renamePending.json file will be
> { OldFolderName: parent/ folderToRename",
> NewFolderName: "parent/renamedFolder",
> FileList: [ "innerFolder", "innerFolder/innerFile" ]
> }
> Atfirst we rename all files within the source directory and then rename the
> source directory at the last step
> The steps are
> 1.Atfirst we will rename innerFolder,
> 2.Then rename innerFolder/innerFile
> 3.Then rename source directory folderToRename
> Say the process crashes after step 1.
> So innerFolder has been renamed.
> Note that Azure storage does not natively support folder. So if a directory
> created by mkdir command, we create an empty placeholder blob with metadata
> for the directory.
> So after step 1, the empty blob corresponding to the directory innerFolder
> has been renamed.
> When the process comes up, in redo path it will go through the
> –renamePending.json file try to redo the renames.
> For each file in file list of renamePending file it checks if the source file
> exists, if source file exists then it renames the file. When it gets
> innerFolder, it calls filesystem.exists(innerFolder). Now
> filesystem.exists(innerFolder) will return true, because file under that
> folder exists even though the empty blob corresponding th that folder does
> not exist. So it will try to rename this folder, and as the empty blob has
> already been deleted so this fails with exception that “source blob does not
> exist”.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12780) During atomic rename handle crash when one directory has been renamed but not file under it.
madhumita chakraborty created HADOOP-12780:
--
Summary: During atomic rename handle crash when one directory has
been renamed but not file under it.
Key: HADOOP-12780
URL: https://issues.apache.org/jira/browse/HADOOP-12780
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Affects Versions: 2.8.0
Reporter: madhumita chakraborty
Assignee: madhumita chakraborty
Priority: Critical
During atomic folder rename process preperaion we record the proposed change to
a metadata file (-renamePending.json).
Say we are renaming parent/folderToRename to parent/renamedFolder.
folderToRename has an inner folder innerFolder and innerFolder has a file
innerFile
Content of the –renamePending.json file will be
{ OldFolderName: /parent/ folderToRename", NewFolderName:
"parent/renamedFolder", FileList: [ "innerFolder", "innerFolder/innerFile" ] }
Atfirst we rename all files within the source directory and then rename the
source directory at the last step
The steps are
1. Atfirst we will rename innerFolder,
2. Then rename innerFolder/innerFile
3. Then rename source directory folderToRename
Say the process crashes after step 1.
So innerFolder has been renamed.
Note that Azure storage does not natively support folder. So if a directory
created by mkdir command, we create an empty placeholder blob with metadata for
the directory.
So after step 1, the empty blob corresponding to the directory innerFolder has
been renamed.
When the process comes up, in redo path it will go through the
–renamePending.json file try to redo the renames.
For each file in file list of renamePending file it checks if the source file
exists, if source file exists then it renames the file. When it gets
innerFolder, it calls filesystem.exists(innerFolder). Now
filesystem.exists(innerFolder) will return true, because file under that folder
exists even though the empty blob corresponding th that folder does not exist.
So it will try to rename this folder, and as the empty blob has already been
deleted so this fails with exception that “source blob does not exist”.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12780) During atomic rename handle crash when one directory has been renamed but not file under it.
[
https://issues.apache.org/jira/browse/HADOOP-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12780:
---
Description:
During atomic folder rename process preperaion we record the proposed change to
a metadata file (-renamePending.json).
Say we are renaming parent/folderToRename to parent/renamedFolder.
folderToRename has an inner folder innerFolder and innerFolder has a file
innerFile
Content of the –renamePending.json file will be
{ OldFolderName: parent/ folderToRename",
NewFolderName: "parent/renamedFolder",
FileList: [ "innerFolder", "innerFolder/innerFile" ]
}
Atfirst we rename all files within the source directory and then rename the
source directory at the last step
The steps are
1. Atfirst we will rename innerFolder,
2. Then rename innerFolder/innerFile
3. Then rename source directory folderToRename
Say the process crashes after step 1.
So innerFolder has been renamed.
Note that Azure storage does not natively support folder. So if a directory
created by mkdir command, we create an empty placeholder blob with metadata for
the directory.
So after step 1, the empty blob corresponding to the directory innerFolder has
been renamed.
When the process comes up, in redo path it will go through the
–renamePending.json file try to redo the renames.
For each file in file list of renamePending file it checks if the source file
exists, if source file exists then it renames the file. When it gets
innerFolder, it calls filesystem.exists(innerFolder). Now
filesystem.exists(innerFolder) will return true, because file under that folder
exists even though the empty blob corresponding th that folder does not exist.
So it will try to rename this folder, and as the empty blob has already been
deleted so this fails with exception that “source blob does not exist”.
was:
During atomic folder rename process preperaion we record the proposed change to
a metadata file (-renamePending.json).
Say we are renaming parent/folderToRename to parent/renamedFolder.
folderToRename has an inner folder innerFolder and innerFolder has a file
innerFile
Content of the –renamePending.json file will be
{ OldFolderName: parent/ folderToRename", NewFolderName:
"parent/renamedFolder", FileList: [ "innerFolder", "innerFolder/innerFile" ] }
Atfirst we rename all files within the source directory and then rename the
source directory at the last step
The steps are
1. Atfirst we will rename innerFolder,
2. Then rename innerFolder/innerFile
3. Then rename source directory folderToRename
Say the process crashes after step 1.
So innerFolder has been renamed.
Note that Azure storage does not natively support folder. So if a directory
created by mkdir command, we create an empty placeholder blob with metadata for
the directory.
So after step 1, the empty blob corresponding to the directory innerFolder has
been renamed.
When the process comes up, in redo path it will go through the
–renamePending.json file try to redo the renames.
For each file in file list of renamePending file it checks if the source file
exists, if source file exists then it renames the file. When it gets
innerFolder, it calls filesystem.exists(innerFolder). Now
filesystem.exists(innerFolder) will return true, because file under that folder
exists even though the empty blob corresponding th that folder does not exist.
So it will try to rename this folder, and as the empty blob has already been
deleted so this fails with exception that “source blob does not exist”.
> During atomic rename handle crash when one directory has been renamed but not
> file under it.
>
>
> Key: HADOOP-12780
> URL: https://issues.apache.org/jira/browse/HADOOP-12780
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: madhumita chakraborty
>Assignee: madhumita chakraborty
>Priority: Critical
>
> During atomic folder rename process preperaion we record the proposed change
> to a metadata file (-renamePending.json).
> Say we are renaming parent/folderToRename to parent/renamedFolder.
> folderToRename has an inner folder innerFolder and innerFolder has a file
> innerFile
> Content of the –renamePending.json file will be
> { OldFolderName: parent/ folderToRename",
> NewFolderName: "parent/renamedFolder",
> FileList: [ "innerFolder", "innerFolder/innerFile" ]
> }
> Atfirst we rename all files within the source directory and then rename the
> source directory at the last step
> The steps are
> 1.Atfirst we will rename innerFolder,
> 2.Then rename innerFolder/innerFile
> 3.Then rename source directory folderToRename
> Say the process crashes after
[jira] [Updated] (HADOOP-12780) During atomic rename handle crash when one directory has been renamed but not file under it.
[
https://issues.apache.org/jira/browse/HADOOP-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12780:
---
Description:
During atomic folder rename process preperaion we record the proposed change to
a metadata file (-renamePending.json).
Say we are renaming parent/folderToRename to parent/renamedFolder.
folderToRename has an inner folder innerFolder and innerFolder has a file
innerFile
Content of the –renamePending.json file will be
{ OldFolderName: parent/ folderToRename", NewFolderName:
"parent/renamedFolder", FileList: [ "innerFolder", "innerFolder/innerFile" ] }
Atfirst we rename all files within the source directory and then rename the
source directory at the last step
The steps are
1. Atfirst we will rename innerFolder,
2. Then rename innerFolder/innerFile
3. Then rename source directory folderToRename
Say the process crashes after step 1.
So innerFolder has been renamed.
Note that Azure storage does not natively support folder. So if a directory
created by mkdir command, we create an empty placeholder blob with metadata for
the directory.
So after step 1, the empty blob corresponding to the directory innerFolder has
been renamed.
When the process comes up, in redo path it will go through the
–renamePending.json file try to redo the renames.
For each file in file list of renamePending file it checks if the source file
exists, if source file exists then it renames the file. When it gets
innerFolder, it calls filesystem.exists(innerFolder). Now
filesystem.exists(innerFolder) will return true, because file under that folder
exists even though the empty blob corresponding th that folder does not exist.
So it will try to rename this folder, and as the empty blob has already been
deleted so this fails with exception that “source blob does not exist”.
was:
During atomic folder rename process preperaion we record the proposed change to
a metadata file (-renamePending.json).
Say we are renaming parent/folderToRename to parent/renamedFolder.
folderToRename has an inner folder innerFolder and innerFolder has a file
innerFile
Content of the –renamePending.json file will be
{ OldFolderName: /parent/ folderToRename", NewFolderName:
"parent/renamedFolder", FileList: [ "innerFolder", "innerFolder/innerFile" ] }
Atfirst we rename all files within the source directory and then rename the
source directory at the last step
The steps are
1. Atfirst we will rename innerFolder,
2. Then rename innerFolder/innerFile
3. Then rename source directory folderToRename
Say the process crashes after step 1.
So innerFolder has been renamed.
Note that Azure storage does not natively support folder. So if a directory
created by mkdir command, we create an empty placeholder blob with metadata for
the directory.
So after step 1, the empty blob corresponding to the directory innerFolder has
been renamed.
When the process comes up, in redo path it will go through the
–renamePending.json file try to redo the renames.
For each file in file list of renamePending file it checks if the source file
exists, if source file exists then it renames the file. When it gets
innerFolder, it calls filesystem.exists(innerFolder). Now
filesystem.exists(innerFolder) will return true, because file under that folder
exists even though the empty blob corresponding th that folder does not exist.
So it will try to rename this folder, and as the empty blob has already been
deleted so this fails with exception that “source blob does not exist”.
> During atomic rename handle crash when one directory has been renamed but not
> file under it.
>
>
> Key: HADOOP-12780
> URL: https://issues.apache.org/jira/browse/HADOOP-12780
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: madhumita chakraborty
>Assignee: madhumita chakraborty
>Priority: Critical
>
> During atomic folder rename process preperaion we record the proposed change
> to a metadata file (-renamePending.json).
> Say we are renaming parent/folderToRename to parent/renamedFolder.
> folderToRename has an inner folder innerFolder and innerFolder has a file
> innerFile
> Content of the –renamePending.json file will be
> { OldFolderName: parent/ folderToRename", NewFolderName:
> "parent/renamedFolder", FileList: [ "innerFolder", "innerFolder/innerFile" ] }
> Atfirst we rename all files within the source directory and then rename the
> source directory at the last step
> The steps are
> 1.Atfirst we will rename innerFolder,
> 2.Then rename innerFolder/innerFile
> 3.Then rename source directory folderToRename
> Say the process crashes after step 1.
> So innerFolder has
[jira] [Updated] (HADOOP-12780) During atomic rename handle crash when one directory has been renamed but not file under it.
[
https://issues.apache.org/jira/browse/HADOOP-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12780:
---
Status: Patch Available (was: Open)
> During atomic rename handle crash when one directory has been renamed but not
> file under it.
>
>
> Key: HADOOP-12780
> URL: https://issues.apache.org/jira/browse/HADOOP-12780
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: madhumita chakraborty
>Assignee: madhumita chakraborty
>Priority: Critical
> Attachments: HADOOP-12780.001.patch
>
>
> During atomic folder rename process preperaion we record the proposed change
> to a metadata file (-renamePending.json).
> Say we are renaming parent/folderToRename to parent/renamedFolder.
> folderToRename has an inner folder innerFolder and innerFolder has a file
> innerFile
> Content of the –renamePending.json file will be
> { OldFolderName: parent/ folderToRename",
> NewFolderName: "parent/renamedFolder",
> FileList: [ "innerFolder", "innerFolder/innerFile" ]
> }
> Atfirst we rename all files within the source directory and then rename the
> source directory at the last step
> The steps are
> 1.Atfirst we will rename innerFolder,
> 2.Then rename innerFolder/innerFile
> 3.Then rename source directory folderToRename
> Say the process crashes after step 1.
> So innerFolder has been renamed.
> Note that Azure storage does not natively support folder. So if a directory
> created by mkdir command, we create an empty placeholder blob with metadata
> for the directory.
> So after step 1, the empty blob corresponding to the directory innerFolder
> has been renamed.
> When the process comes up, in redo path it will go through the
> –renamePending.json file try to redo the renames.
> For each file in file list of renamePending file it checks if the source file
> exists, if source file exists then it renames the file. When it gets
> innerFolder, it calls filesystem.exists(innerFolder). Now
> filesystem.exists(innerFolder) will return true, because file under that
> folder exists even though the empty blob corresponding th that folder does
> not exist. So it will try to rename this folder, and as the empty blob has
> already been deleted so this fails with exception that “source blob does not
> exist”.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12780) During atomic rename handle crash when one directory has been renamed but not file under it.
[
https://issues.apache.org/jira/browse/HADOOP-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12780:
---
Status: Open (was: Patch Available)
> During atomic rename handle crash when one directory has been renamed but not
> file under it.
>
>
> Key: HADOOP-12780
> URL: https://issues.apache.org/jira/browse/HADOOP-12780
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: madhumita chakraborty
>Assignee: madhumita chakraborty
>Priority: Critical
> Attachments: HADOOP-12780.001.patch
>
>
> During atomic folder rename process preperaion we record the proposed change
> to a metadata file (-renamePending.json).
> Say we are renaming parent/folderToRename to parent/renamedFolder.
> folderToRename has an inner folder innerFolder and innerFolder has a file
> innerFile
> Content of the –renamePending.json file will be
> { OldFolderName: parent/ folderToRename",
> NewFolderName: "parent/renamedFolder",
> FileList: [ "innerFolder", "innerFolder/innerFile" ]
> }
> Atfirst we rename all files within the source directory and then rename the
> source directory at the last step
> The steps are
> 1.Atfirst we will rename innerFolder,
> 2.Then rename innerFolder/innerFile
> 3.Then rename source directory folderToRename
> Say the process crashes after step 1.
> So innerFolder has been renamed.
> Note that Azure storage does not natively support folder. So if a directory
> created by mkdir command, we create an empty placeholder blob with metadata
> for the directory.
> So after step 1, the empty blob corresponding to the directory innerFolder
> has been renamed.
> When the process comes up, in redo path it will go through the
> –renamePending.json file try to redo the renames.
> For each file in file list of renamePending file it checks if the source file
> exists, if source file exists then it renames the file. When it gets
> innerFolder, it calls filesystem.exists(innerFolder). Now
> filesystem.exists(innerFolder) will return true, because file under that
> folder exists even though the empty blob corresponding th that folder does
> not exist. So it will try to rename this folder, and as the empty blob has
> already been deleted so this fails with exception that “source blob does not
> exist”.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12780) During atomic rename handle crash when one directory has been renamed but not file under it.
[
https://issues.apache.org/jira/browse/HADOOP-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12780:
---
Attachment: (was: HADOOP-12780.001.patch)
> During atomic rename handle crash when one directory has been renamed but not
> file under it.
>
>
> Key: HADOOP-12780
> URL: https://issues.apache.org/jira/browse/HADOOP-12780
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: madhumita chakraborty
>Assignee: madhumita chakraborty
>Priority: Critical
> Attachments: HADOOP-12780.001.patch
>
>
> During atomic folder rename process preperaion we record the proposed change
> to a metadata file (-renamePending.json).
> Say we are renaming parent/folderToRename to parent/renamedFolder.
> folderToRename has an inner folder innerFolder and innerFolder has a file
> innerFile
> Content of the –renamePending.json file will be
> { OldFolderName: parent/ folderToRename",
> NewFolderName: "parent/renamedFolder",
> FileList: [ "innerFolder", "innerFolder/innerFile" ]
> }
> Atfirst we rename all files within the source directory and then rename the
> source directory at the last step
> The steps are
> 1.Atfirst we will rename innerFolder,
> 2.Then rename innerFolder/innerFile
> 3.Then rename source directory folderToRename
> Say the process crashes after step 1.
> So innerFolder has been renamed.
> Note that Azure storage does not natively support folder. So if a directory
> created by mkdir command, we create an empty placeholder blob with metadata
> for the directory.
> So after step 1, the empty blob corresponding to the directory innerFolder
> has been renamed.
> When the process comes up, in redo path it will go through the
> –renamePending.json file try to redo the renames.
> For each file in file list of renamePending file it checks if the source file
> exists, if source file exists then it renames the file. When it gets
> innerFolder, it calls filesystem.exists(innerFolder). Now
> filesystem.exists(innerFolder) will return true, because file under that
> folder exists even though the empty blob corresponding th that folder does
> not exist. So it will try to rename this folder, and as the empty blob has
> already been deleted so this fails with exception that “source blob does not
> exist”.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12780) During atomic rename handle crash when one directory has been renamed but not file under it.
[
https://issues.apache.org/jira/browse/HADOOP-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12780:
---
Attachment: HADOOP-12780.001.patch
> During atomic rename handle crash when one directory has been renamed but not
> file under it.
>
>
> Key: HADOOP-12780
> URL: https://issues.apache.org/jira/browse/HADOOP-12780
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: madhumita chakraborty
>Assignee: madhumita chakraborty
>Priority: Critical
> Attachments: HADOOP-12780.001.patch
>
>
> During atomic folder rename process preperaion we record the proposed change
> to a metadata file (-renamePending.json).
> Say we are renaming parent/folderToRename to parent/renamedFolder.
> folderToRename has an inner folder innerFolder and innerFolder has a file
> innerFile
> Content of the –renamePending.json file will be
> { OldFolderName: parent/ folderToRename",
> NewFolderName: "parent/renamedFolder",
> FileList: [ "innerFolder", "innerFolder/innerFile" ]
> }
> Atfirst we rename all files within the source directory and then rename the
> source directory at the last step
> The steps are
> 1.Atfirst we will rename innerFolder,
> 2.Then rename innerFolder/innerFile
> 3.Then rename source directory folderToRename
> Say the process crashes after step 1.
> So innerFolder has been renamed.
> Note that Azure storage does not natively support folder. So if a directory
> created by mkdir command, we create an empty placeholder blob with metadata
> for the directory.
> So after step 1, the empty blob corresponding to the directory innerFolder
> has been renamed.
> When the process comes up, in redo path it will go through the
> –renamePending.json file try to redo the renames.
> For each file in file list of renamePending file it checks if the source file
> exists, if source file exists then it renames the file. When it gets
> innerFolder, it calls filesystem.exists(innerFolder). Now
> filesystem.exists(innerFolder) will return true, because file under that
> folder exists even though the empty blob corresponding th that folder does
> not exist. So it will try to rename this folder, and as the empty blob has
> already been deleted so this fails with exception that “source blob does not
> exist”.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12780) During atomic rename handle crash when one directory has been renamed but not file under it.
[
https://issues.apache.org/jira/browse/HADOOP-12780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12780:
---
Status: Patch Available (was: Open)
> During atomic rename handle crash when one directory has been renamed but not
> file under it.
>
>
> Key: HADOOP-12780
> URL: https://issues.apache.org/jira/browse/HADOOP-12780
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: madhumita chakraborty
>Assignee: madhumita chakraborty
>Priority: Critical
> Attachments: HADOOP-12780.001.patch
>
>
> During atomic folder rename process preperaion we record the proposed change
> to a metadata file (-renamePending.json).
> Say we are renaming parent/folderToRename to parent/renamedFolder.
> folderToRename has an inner folder innerFolder and innerFolder has a file
> innerFile
> Content of the –renamePending.json file will be
> { OldFolderName: parent/ folderToRename",
> NewFolderName: "parent/renamedFolder",
> FileList: [ "innerFolder", "innerFolder/innerFile" ]
> }
> Atfirst we rename all files within the source directory and then rename the
> source directory at the last step
> The steps are
> 1.Atfirst we will rename innerFolder,
> 2.Then rename innerFolder/innerFile
> 3.Then rename source directory folderToRename
> Say the process crashes after step 1.
> So innerFolder has been renamed.
> Note that Azure storage does not natively support folder. So if a directory
> created by mkdir command, we create an empty placeholder blob with metadata
> for the directory.
> So after step 1, the empty blob corresponding to the directory innerFolder
> has been renamed.
> When the process comes up, in redo path it will go through the
> –renamePending.json file try to redo the renames.
> For each file in file list of renamePending file it checks if the source file
> exists, if source file exists then it renames the file. When it gets
> innerFolder, it calls filesystem.exists(innerFolder). Now
> filesystem.exists(innerFolder) will return true, because file under that
> folder exists even though the empty blob corresponding th that folder does
> not exist. So it will try to rename this folder, and as the empty blob has
> already been deleted so this fails with exception that “source blob does not
> exist”.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Attachment: HADOOP-12535.001.patch
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty updated HADOOP-12535:
---
Status: Patch Available (was: Open)
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
madhumita chakraborty reassigned HADOOP-12535:
--
Assignee: madhumita chakraborty (was: Dushyanth)
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.004.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.005.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085902#comment-15085902 ] madhumita chakraborty commented on HADOOP-12678: [~cnauroth] I have addressed your comments. Could you please take a look? > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.006.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch, > HADOOP-12678.006.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch, > HADOOP-12678.006.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: (was: HADOOP-12678.006.patch) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.006.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch, > HADOOP-12678.006.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch, > HADOOP-12678.006.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch, HADOOP-12678.004.patch, HADOOP-12678.005.patch, > HADOOP-12678.006.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.001.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.002.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15073513#comment-15073513 ] madhumita chakraborty commented on HADOOP-12678: [~cnauroth],[~gouravk],[~pravinmittal],[~dchickabasapa] could you guys please take a look at the patch? > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Open (was: Patch Available) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Status: Patch Available (was: Open) > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[ https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhumita chakraborty updated HADOOP-12678: --- Attachment: HADOOP-12678.003.patch > Handle empty rename pending metadata file during atomic rename in redo path > --- > > Key: HADOOP-12678 > URL: https://issues.apache.org/jira/browse/HADOOP-12678 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: madhumita chakraborty >Assignee: madhumita chakraborty >Priority: Critical > Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch, > HADOOP-12678.003.patch > > > Handle empty rename pending metadata file during atomic rename in redo path > During atomic rename we create metadata file for rename(-renamePending.json). > We create that in 2 steps > 1. We create an empty blob corresponding to the .json file in its real > location > 2. We create a scratch file to which we write the contents of the rename > pending which is then copied over into the blob described in 1 > If process crash occurs after step 1 and before step 2 is complete - we will > be left with a zero size blob corresponding to the pending rename metadata > file. > This kind of scenario can happen in the /hbase/.tmp folder because it is > considered a candidate folder for atomic rename. Now when HMaster starts up > it executes listStatus on the .tmp folder to clean up pending data. At this > stage due to the lazy pending rename complete process we look for these json > files. On seeing an empty file the process simply throws a fatal exception > assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
madhumita chakraborty created HADOOP-12678: -- Summary: Handle empty rename pending metadata file during atomic rename in redo path Key: HADOOP-12678 URL: https://issues.apache.org/jira/browse/HADOOP-12678 Project: Hadoop Common Issue Type: Bug Components: fs/azure Reporter: madhumita chakraborty Assignee: madhumita chakraborty Priority: Critical Handle empty rename pending metadata file during atomic rename in redo path During atomic rename we create metadata file for rename(-renamePending.json). We create that in 2 steps 1. We create an empty blob corresponding to the .json file in its real location 2. We create a scratch file to which we write the contents of the rename pending which is then copied over into the blob described in 1 If process crash occurs after step 1 and before step 2 is complete - we will be left with a zero size blob corresponding to the pending rename metadata file. This kind of scenario can happen in the /hbase/.tmp folder because it is considered a candidate folder for atomic rename. Now when HMaster starts up it executes listStatus on the .tmp folder to clean up pending data. At this stage due to the lazy pending rename complete process we look for these json files. On seeing an empty file the process simply throws a fatal exception assuming something went wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
