[jira] [Updated] (HDFS-12336) Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12336:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.3
2.9.0
Status: Resolved (was: Patch Available)
Committed to branch-2 and branch-2.8. Thank you, [~wchevreuil]!
> Listing encryption zones still fails when deleted EZ is not a direct child of
> snapshottable directory
> -
>
> Key: HDFS-12336
> URL: https://issues.apache.org/jira/browse/HDFS-12336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.3
>
> Attachments: HDFS-12336.001.patch, HDFS-12336.002.patch,
> HDFS-12336.003.patch, HDFS-12336.004.patch, HDFS-12336-branch-2.001.patch
>
>
> The fix proposed on HDFS-11197 didn't cover the scenario where the EZ deleted
> but still under a snapshot is not a direct child of the snapshottable
> directory.
> Here the code snippet proposed on HDFS-11197 that would avoid the error
> reported by *hdfs crypto -listZones* when a deleted EZ is still under a given
> snapshot:
> {noformat}
> INode lastINode = null;
> if (inode.getParent() != null || inode.isRoot()) {
> INodesInPath iip = dir.getINodesInPath(pathName, DirOp.READ_LINK);
> lastINode = iip.getLastINode();
> }
> if (lastINode == null || lastINode.getId() != ezi.getINodeId()) {
> continue;
> }
> {noformat}
> It will ignore EZs when it's a direct child of a snapshot, because its parent
> inode will be null, and it isn't the root inode. However, if the EZ is not
> directly under snapshottable directory, its parent will not be null, and it
> will pass this check, so it will fail further due *absolute path required*
> validation error.
> I would like to work on a fix that would also cover this scenario.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12336) Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-12336:
Attachment: HDFS-12336-branch-2.001.patch
Thanks a lot [~xiaochen] for the help here. Attached a patch for branch-2.
> Listing encryption zones still fails when deleted EZ is not a direct child of
> snapshottable directory
> -
>
> Key: HDFS-12336
> URL: https://issues.apache.org/jira/browse/HDFS-12336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12336.001.patch, HDFS-12336.002.patch,
> HDFS-12336.003.patch, HDFS-12336.004.patch, HDFS-12336-branch-2.001.patch
>
>
> The fix proposed on HDFS-11197 didn't cover the scenario where the EZ deleted
> but still under a snapshot is not a direct child of the snapshottable
> directory.
> Here the code snippet proposed on HDFS-11197 that would avoid the error
> reported by *hdfs crypto -listZones* when a deleted EZ is still under a given
> snapshot:
> {noformat}
> INode lastINode = null;
> if (inode.getParent() != null || inode.isRoot()) {
> INodesInPath iip = dir.getINodesInPath(pathName, DirOp.READ_LINK);
> lastINode = iip.getLastINode();
> }
> if (lastINode == null || lastINode.getId() != ezi.getINodeId()) {
> continue;
> }
> {noformat}
> It will ignore EZs when it's a direct child of a snapshot, because its parent
> inode will be null, and it isn't the root inode. However, if the EZ is not
> directly under snapshottable directory, its parent will not be null, and it
> will pass this check, so it will fail further due *absolute path required*
> validation error.
> I would like to work on a fix that would also cover this scenario.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12336) Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12336:
-
Fix Version/s: 3.0.0-beta1
> Listing encryption zones still fails when deleted EZ is not a direct child of
> snapshottable directory
> -
>
> Key: HDFS-12336
> URL: https://issues.apache.org/jira/browse/HDFS-12336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12336.001.patch, HDFS-12336.002.patch,
> HDFS-12336.003.patch, HDFS-12336.004.patch
>
>
> The fix proposed on HDFS-11197 didn't cover the scenario where the EZ deleted
> but still under a snapshot is not a direct child of the snapshottable
> directory.
> Here the code snippet proposed on HDFS-11197 that would avoid the error
> reported by *hdfs crypto -listZones* when a deleted EZ is still under a given
> snapshot:
> {noformat}
> INode lastINode = null;
> if (inode.getParent() != null || inode.isRoot()) {
> INodesInPath iip = dir.getINodesInPath(pathName, DirOp.READ_LINK);
> lastINode = iip.getLastINode();
> }
> if (lastINode == null || lastINode.getId() != ezi.getINodeId()) {
> continue;
> }
> {noformat}
> It will ignore EZs when it's a direct child of a snapshot, because its parent
> inode will be null, and it isn't the root inode. However, if the EZ is not
> directly under snapshottable directory, its parent will not be null, and it
> will pass this check, so it will fail further due *absolute path required*
> validation error.
> I would like to work on a fix that would also cover this scenario.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12336) Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12336:
-
Component/s: encryption
> Listing encryption zones still fails when deleted EZ is not a direct child of
> snapshottable directory
> -
>
> Key: HDFS-12336
> URL: https://issues.apache.org/jira/browse/HDFS-12336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12336.001.patch, HDFS-12336.002.patch,
> HDFS-12336.003.patch, HDFS-12336.004.patch
>
>
> The fix proposed on HDFS-11197 didn't cover the scenario where the EZ deleted
> but still under a snapshot is not a direct child of the snapshottable
> directory.
> Here the code snippet proposed on HDFS-11197 that would avoid the error
> reported by *hdfs crypto -listZones* when a deleted EZ is still under a given
> snapshot:
> {noformat}
> INode lastINode = null;
> if (inode.getParent() != null || inode.isRoot()) {
> INodesInPath iip = dir.getINodesInPath(pathName, DirOp.READ_LINK);
> lastINode = iip.getLastINode();
> }
> if (lastINode == null || lastINode.getId() != ezi.getINodeId()) {
> continue;
> }
> {noformat}
> It will ignore EZs when it's a direct child of a snapshot, because its parent
> inode will be null, and it isn't the root inode. However, if the EZ is not
> directly under snapshottable directory, its parent will not be null, and it
> will pass this check, so it will fail further due *absolute path required*
> validation error.
> I would like to work on a fix that would also cover this scenario.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12336) Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-12336:
Attachment: HDFS-12336.004.patch
Thanks for the latest suggestions [~xiaochen]! All make sense to me, am
attaching a new patch with those applied.
If these are ok, let me know if you want me to create a patch for branch-2 as
well.
> Listing encryption zones still fails when deleted EZ is not a direct child of
> snapshottable directory
> -
>
> Key: HDFS-12336
> URL: https://issues.apache.org/jira/browse/HDFS-12336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-12336.001.patch, HDFS-12336.002.patch,
> HDFS-12336.003.patch, HDFS-12336.004.patch
>
>
> The fix proposed on HDFS-11197 didn't cover the scenario where the EZ deleted
> but still under a snapshot is not a direct child of the snapshottable
> directory.
> Here the code snippet proposed on HDFS-11197 that would avoid the error
> reported by *hdfs crypto -listZones* when a deleted EZ is still under a given
> snapshot:
> {noformat}
> INode lastINode = null;
> if (inode.getParent() != null || inode.isRoot()) {
> INodesInPath iip = dir.getINodesInPath(pathName, DirOp.READ_LINK);
> lastINode = iip.getLastINode();
> }
> if (lastINode == null || lastINode.getId() != ezi.getINodeId()) {
> continue;
> }
> {noformat}
> It will ignore EZs when it's a direct child of a snapshot, because its parent
> inode will be null, and it isn't the root inode. However, if the EZ is not
> directly under snapshottable directory, its parent will not be null, and it
> will pass this check, so it will fail further due *absolute path required*
> validation error.
> I would like to work on a fix that would also cover this scenario.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12336) Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-12336:
Attachment: HDFS-12336.003.patch
Thanks for the suggestions [~xiaochen]! Indeed, checking the absolute path is
much simpler. I thought of defining a new helper method, *isValidAbsolutePath*
on *INode* class that does the check previously done on *checkAbsolutePath*,
then using this new one on *EncryptionZoneManager.listEncryptionZones* to
determine if the inode should be filtered or not.
Also had applied the suggestions to *EncryptionZonesTest*, had added javadoc
explaining why we need to move to Trash first, instead of simply deleting it,
and also removed the calls to assertZonePresent.
I also noticed latest patch was out of sync with trunk. I had rebased it to
latest trunk version and had resolved some conflicts with HDFS-10899, which has
doe a large refactoring in EncryptionZoneManager. These scenario was still
failing, though.
> Listing encryption zones still fails when deleted EZ is not a direct child of
> snapshottable directory
> -
>
> Key: HDFS-12336
> URL: https://issues.apache.org/jira/browse/HDFS-12336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-12336.001.patch, HDFS-12336.002.patch,
> HDFS-12336.003.patch
>
>
> The fix proposed on HDFS-11197 didn't cover the scenario where the EZ deleted
> but still under a snapshot is not a direct child of the snapshottable
> directory.
> Here the code snippet proposed on HDFS-11197 that would avoid the error
> reported by *hdfs crypto -listZones* when a deleted EZ is still under a given
> snapshot:
> {noformat}
> INode lastINode = null;
> if (inode.getParent() != null || inode.isRoot()) {
> INodesInPath iip = dir.getINodesInPath(pathName, DirOp.READ_LINK);
> lastINode = iip.getLastINode();
> }
> if (lastINode == null || lastINode.getId() != ezi.getINodeId()) {
> continue;
> }
> {noformat}
> It will ignore EZs when it's a direct child of a snapshot, because its parent
> inode will be null, and it isn't the root inode. However, if the EZ is not
> directly under snapshottable directory, its parent will not be null, and it
> will pass this check, so it will fail further due *absolute path required*
> validation error.
> I would like to work on a fix that would also cover this scenario.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12336) Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-12336:
Attachment: HDFS-12336.002.patch
Hi [~xiaochen], thanks for reviewing this. I realised I made a mistake on the
test added to TestCryptoAdminCLI, which was failing to capture the error when
the fix is not applied.
The problem manifests when an EZ is under a snapshottable directory. If a
snapshot is taken, then the EZ (or a parent directory that contains the EZ) is
moved to user's Trash folder, and later, it's completely removed from user's
Trash, the EZ's inode path becomes somehow incomplete in the image. Here's an
example of what *inode.getFullPathName()* returns in this case:
{noformat}
INFO org.apache.hadoop.hdfs.server.namenode.EncryptionZoneManager: >>>inode:
dir10/dir11/ezz
INFO org.apache.hadoop.hdfs.server.namenode.EncryptionZoneManager: >>>inode:
dir2/ez2
{noformat}
I believe this happens because inode that got first moved to Trash, then
deleted, ends up with no parent inode in the image, as if it was child of
snaphots only.
I agree it makes sense adding test to TestEncryptionZones. I'm including a
patch with the additional test to TestEncryptionZones and fix to
TestCryptoAdminCLI.
> Listing encryption zones still fails when deleted EZ is not a direct child of
> snapshottable directory
> -
>
> Key: HDFS-12336
> URL: https://issues.apache.org/jira/browse/HDFS-12336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-12336.001.patch, HDFS-12336.002.patch
>
>
> The fix proposed on HDFS-11197 didn't cover the scenario where the EZ deleted
> but still under a snapshot is not a direct child of the snapshottable
> directory.
> Here the code snippet proposed on HDFS-11197 that would avoid the error
> reported by *hdfs crypto -listZones* when a deleted EZ is still under a given
> snapshot:
> {noformat}
> INode lastINode = null;
> if (inode.getParent() != null || inode.isRoot()) {
> INodesInPath iip = dir.getINodesInPath(pathName, DirOp.READ_LINK);
> lastINode = iip.getLastINode();
> }
> if (lastINode == null || lastINode.getId() != ezi.getINodeId()) {
> continue;
> }
> {noformat}
> It will ignore EZs when it's a direct child of a snapshot, because its parent
> inode will be null, and it isn't the root inode. However, if the EZ is not
> directly under snapshottable directory, its parent will not be null, and it
> will pass this check, so it will fail further due *absolute path required*
> validation error.
> I would like to work on a fix that would also cover this scenario.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12336) Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-12336:
Status: Patch Available (was: Open)
> Listing encryption zones still fails when deleted EZ is not a direct child of
> snapshottable directory
> -
>
> Key: HDFS-12336
> URL: https://issues.apache.org/jira/browse/HDFS-12336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-12336.001.patch
>
>
> The fix proposed on HDFS-11197 didn't cover the scenario where the EZ deleted
> but still under a snapshot is not a direct child of the snapshottable
> directory.
> Here the code snippet proposed on HDFS-11197 that would avoid the error
> reported by *hdfs crypto -listZones* when a deleted EZ is still under a given
> snapshot:
> {noformat}
> INode lastINode = null;
> if (inode.getParent() != null || inode.isRoot()) {
> INodesInPath iip = dir.getINodesInPath(pathName, DirOp.READ_LINK);
> lastINode = iip.getLastINode();
> }
> if (lastINode == null || lastINode.getId() != ezi.getINodeId()) {
> continue;
> }
> {noformat}
> It will ignore EZs when it's a direct child of a snapshot, because its parent
> inode will be null, and it isn't the root inode. However, if the EZ is not
> directly under snapshottable directory, its parent will not be null, and it
> will pass this check, so it will fail further due *absolute path required*
> validation error.
> I would like to work on a fix that would also cover this scenario.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12336) Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-12336:
Attachment: HDFS-12336.001.patch
Proposing initial patch with changes to also cover condition when EZ is not a
direct child of snapshottable dir, together with tests to emulate such
condition.
> Listing encryption zones still fails when deleted EZ is not a direct child of
> snapshottable directory
> -
>
> Key: HDFS-12336
> URL: https://issues.apache.org/jira/browse/HDFS-12336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-12336.001.patch
>
>
> The fix proposed on HDFS-11197 didn't cover the scenario where the EZ deleted
> but still under a snapshot is not a direct child of the snapshottable
> directory.
> Here the code snippet proposed on HDFS-11197 that would avoid the error
> reported by *hdfs crypto -listZones* when a deleted EZ is still under a given
> snapshot:
> {noformat}
> INode lastINode = null;
> if (inode.getParent() != null || inode.isRoot()) {
> INodesInPath iip = dir.getINodesInPath(pathName, DirOp.READ_LINK);
> lastINode = iip.getLastINode();
> }
> if (lastINode == null || lastINode.getId() != ezi.getINodeId()) {
> continue;
> }
> {noformat}
> It will ignore EZs when it's a direct child of a snapshot, because its parent
> inode will be null, and it isn't the root inode. However, if the EZ is not
> directly under snapshottable directory, its parent will not be null, and it
> will pass this check, so it will fail further due *absolute path required*
> validation error.
> I would like to work on a fix that would also cover this scenario.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
