[jira] [Updated] (HDFS-9308) Add truncateMeta() and deleteMeta() to MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-9308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-9308:
--
Component/s: (was: HDFS)
> Add truncateMeta() and deleteMeta() to MiniDFSCluster
> -
>
> Key: HDFS-9308
> URL: https://issues.apache.org/jira/browse/HDFS-9308
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
>Priority: Minor
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9308.001.patch, HDFS-9308.002.patch,
> HDFS-9308.003.patch
>
>
> HDFS-9188 introduced {{corruptMeta()}} method to make corrupting the metadata
> file filesystem agnostic. There should also be a {{truncateMeta()}} and
> {{deleteMeta()}} method in MiniDFSCluster to allow truncation of metadata
> files on DataNodes without writing code that's specific to underling file
> system. {{FsDatasetTestUtils#truncateMeta()}} is already implemented by
> HDFS-9188 and cam be exposed easily in {{MiniDFSCluster}}.
> This will be useful for tests such as
> {{TestLeaseRecovery#testBlockRecoveryWithLessMetafile}} and
> {{TestCrcCorruption#testCrcCorruption}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9308) Add truncateMeta() and deleteMeta() to MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-9308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-9308:
Resolution: Fixed
Fix Version/s: 2.8.0
3.0.0
Status: Resolved (was: Patch Available)
Thanks a lot for woking on this, [~twu]
Committed to {{trunk}} and {{branch-2}}.
> Add truncateMeta() and deleteMeta() to MiniDFSCluster
> -
>
> Key: HDFS-9308
> URL: https://issues.apache.org/jira/browse/HDFS-9308
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS, test
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
>Priority: Minor
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9308.001.patch, HDFS-9308.002.patch,
> HDFS-9308.003.patch
>
>
> HDFS-9188 introduced {{corruptMeta()}} method to make corrupting the metadata
> file filesystem agnostic. There should also be a {{truncateMeta()}} and
> {{deleteMeta()}} method in MiniDFSCluster to allow truncation of metadata
> files on DataNodes without writing code that's specific to underling file
> system. {{FsDatasetTestUtils#truncateMeta()}} is already implemented by
> HDFS-9188 and cam be exposed easily in {{MiniDFSCluster}}.
> This will be useful for tests such as
> {{TestLeaseRecovery#testBlockRecoveryWithLessMetafile}} and
> {{TestCrcCorruption#testCrcCorruption}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9308) Add truncateMeta() and deleteMeta() to MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-9308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tony Wu updated HDFS-9308:
--
Attachment: HDFS-9308.003.patch
In v3 patch:
* Addressed [~eddyxu]'s review comments.
*Please note:*
{{TestCrcCorruption#testCrcCorruption}} seems to be broken by
43539b5ff4ac0874a8a454dc93a2a782b0e0ea8f for HDFS-4937 (did a quick binary
search to find this change). I will open a separate JIRA for the test failure.
Manually applied the patch to before the change and verified it works:
{code}
---
T E S T S
---
Running org.apache.hadoop.hdfs.TestCrcCorruption
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.56 sec - in
org.apache.hadoop.hdfs.TestCrcCorruption
Running org.apache.hadoop.hdfs.TestLeaseRecovery
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.752 sec - in
org.apache.hadoop.hdfs.TestLeaseRecovery
Results :
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0
{code}
> Add truncateMeta() and deleteMeta() to MiniDFSCluster
> -
>
> Key: HDFS-9308
> URL: https://issues.apache.org/jira/browse/HDFS-9308
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS, test
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
>Priority: Minor
> Attachments: HDFS-9308.001.patch, HDFS-9308.002.patch,
> HDFS-9308.003.patch
>
>
> HDFS-9188 introduced {{corruptMeta()}} method to make corrupting the metadata
> file filesystem agnostic. There should also be a {{truncateMeta()}} and
> {{deleteMeta()}} method in MiniDFSCluster to allow truncation of metadata
> files on DataNodes without writing code that's specific to underling file
> system. {{FsDatasetTestUtils#truncateMeta()}} is already implemented by
> HDFS-9188 and cam be exposed easily in {{MiniDFSCluster}}.
> This will be useful for tests such as
> {{TestLeaseRecovery#testBlockRecoveryWithLessMetafile}} and
> {{TestCrcCorruption#testCrcCorruption}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9308) Add truncateMeta() and deleteMeta() to MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-9308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tony Wu updated HDFS-9308:
--
Attachment: HDFS-9308.002.patch
* Add {{truncateMeta()}} and {{deleteMeta()}} method to truncate/delete meta
data files on DataNodes.
* Modify {{TestLeaseRecovery#testBlockRecoveryWithLessMetafile}} to use the new
APIs.
* Enhance {{TestLeaseRecovery#testBlockRecoveryWithLessMetafile}} to check the
file size after lease recovery is complete. Truncating the metadata file on
DataNodes will effectively reduce the file size after lease recovery. The test
should verify the new file size as well.
* Modify {{TestCrcCorruption#thistest}} to use the new APIs.
> Add truncateMeta() and deleteMeta() to MiniDFSCluster
> -
>
> Key: HDFS-9308
> URL: https://issues.apache.org/jira/browse/HDFS-9308
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS, test
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
>Priority: Minor
> Attachments: HDFS-9308.001.patch, HDFS-9308.002.patch
>
>
> HDFS-9188 introduced {{corruptMeta()}} method to make corrupting the metadata
> file filesystem agnostic. There should also be a {{truncateMeta()}} and
> {{deleteMeta()}} method in MiniDFSCluster to allow truncation of metadata
> files on DataNodes without writing code that's specific to underling file
> system. {{FsDatasetTestUtils#truncateMeta()}} is already implemented by
> HDFS-9188 and cam be exposed easily in {{MiniDFSCluster}}.
> This will be useful for tests such as
> {{TestLeaseRecovery#testBlockRecoveryWithLessMetafile}} and
> {{TestCrcCorruption#testCrcCorruption}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9308) Add truncateMeta() and deleteMeta() to MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-9308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tony Wu updated HDFS-9308:
--
Summary: Add truncateMeta() and deleteMeta() to MiniDFSCluster (was: Add
truncateMeta() to MiniDFSCluster)
> Add truncateMeta() and deleteMeta() to MiniDFSCluster
> -
>
> Key: HDFS-9308
> URL: https://issues.apache.org/jira/browse/HDFS-9308
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS, test
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
>Priority: Minor
> Attachments: HDFS-9308.001.patch
>
>
> HDFS-9188 introduced {{corruptMeta()}} method to make corrupting the metadata
> file filesystem agnostic. There should also be a {{truncateMeta()}} method in
> MiniDFSCluster to allow truncation of metadata files on DataNodes without
> writing code that's specific to underling file system.
> {{FsDatasetTestUtils#truncateMeta()}} is already implemented by HDFS-9188 and
> cam be exposed easily in {{MiniDFSCluster}}.
> This will be useful for tests such as
> {{TestLeaseRecovery#testBlockRecoveryWithLessMetafile}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9308) Add truncateMeta() and deleteMeta() to MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-9308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tony Wu updated HDFS-9308:
--
Description:
HDFS-9188 introduced {{corruptMeta()}} method to make corrupting the metadata
file filesystem agnostic. There should also be a {{truncateMeta()}} and
{{deleteMeta()}} method in MiniDFSCluster to allow truncation of metadata files
on DataNodes without writing code that's specific to underling file system.
{{FsDatasetTestUtils#truncateMeta()}} is already implemented by HDFS-9188 and
cam be exposed easily in {{MiniDFSCluster}}.
This will be useful for tests such as
{{TestLeaseRecovery#testBlockRecoveryWithLessMetafile}} and
{{TestCrcCorruption#testCrcCorruption}}.
was:
HDFS-9188 introduced {{corruptMeta()}} method to make corrupting the metadata
file filesystem agnostic. There should also be a {{truncateMeta()}} method in
MiniDFSCluster to allow truncation of metadata files on DataNodes without
writing code that's specific to underling file system.
{{FsDatasetTestUtils#truncateMeta()}} is already implemented by HDFS-9188 and
cam be exposed easily in {{MiniDFSCluster}}.
This will be useful for tests such as
{{TestLeaseRecovery#testBlockRecoveryWithLessMetafile}}.
> Add truncateMeta() and deleteMeta() to MiniDFSCluster
> -
>
> Key: HDFS-9308
> URL: https://issues.apache.org/jira/browse/HDFS-9308
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS, test
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
>Priority: Minor
> Attachments: HDFS-9308.001.patch
>
>
> HDFS-9188 introduced {{corruptMeta()}} method to make corrupting the metadata
> file filesystem agnostic. There should also be a {{truncateMeta()}} and
> {{deleteMeta()}} method in MiniDFSCluster to allow truncation of metadata
> files on DataNodes without writing code that's specific to underling file
> system. {{FsDatasetTestUtils#truncateMeta()}} is already implemented by
> HDFS-9188 and cam be exposed easily in {{MiniDFSCluster}}.
> This will be useful for tests such as
> {{TestLeaseRecovery#testBlockRecoveryWithLessMetafile}} and
> {{TestCrcCorruption#testCrcCorruption}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
