[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-11197: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.8.0 Status: Resolved (was: Patch Available) Committed to trunk, branch-2 and branch-2.8. Thanks a lot [~wchevreuil] for reporting and fixing the issue, and [~jojochuang] for reviewing! > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch, > HDFS-11197-6.patch, HDFS-11197-7.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Status: Patch Available (was: In Progress) > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch, > HDFS-11197-6.patch, HDFS-11197-7.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Status: In Progress (was: Patch Available) > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch, > HDFS-11197-6.patch, HDFS-11197-7.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Attachment: HDFS-11197-7.patch Hi [~jojochuang], am proposing a new patch, adding end-to-end tests for this specific issue on {{hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCryptoConf.xml and TestCryptoAdminCLI}}. Few changes performed on {{TestCryptoAdminCLI}} for these tests: 1) Since the issue involved snapshots, I had to make {{TestConfigFileParserCryptoAdmin}} to extend {{CLITestHelperDFS.TestConfigFileParserDFS}}, instead of {{CLITestHelper.TestConfigFileParser}} directly, so that it could parse *dfs-admin-command* commands from {{testCryptoConf.xml}}. This was needed because the test had to run a *dfsadmin -allowSnapshot* command; 2) The issue also involved files being moved to trash, prior to permanent deletion. For that purpose, I had to enable "move to Trash" behaviour on {{TestCryptoAdminCLI.setup}} method, by adding {{conf.setLong(CommonConfigurationKeysPublic.FS_TRASH_INTERVAL_KEY, 10);}} The additional tests check for listZones output under the following cases: 1) Creates a single EZ then call listZones and check if only the created EZ was returned; 2) Creates two EZs, then delete one of those, the call listZones and check if the two are listed. The deleted EZ should be listed "/user/*/.Trash/Current" directory, since Trash is enabled; 3) Creates two EZs, then remove one of the EZ, then remove the deleted one from Trash, then call listZones and checks if only the non-deleted EZ was returned; 4) Creates two EZs under a snapshottable directory, takes a snapshot for the directory, remove one of the EZs, then remove this EZ from Trash, calls listZones and check if only the non-deleted EZ was returned. Please let me know on your thoughts. > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch, > HDFS-11197-6.patch, HDFS-11197-7.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail:
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Status: Patch Available (was: In Progress) > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch, HDFS-11197-6.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Attachment: HDFS-11197-6.patch Thanks again, [~xiaochen]! Sending another patch attempt with your last suggestions. Apologies for the white space previously. I actually tried to run the {{git apply --whitespace=fix <>}} on that patch to fix white spaces, but I guess it didn't work (or maybe I'm doing something wrong). Anyway, I believe it has no white spaces now. Let me know if you find any additional enhancements. > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch, HDFS-11197-6.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Status: In Progress (was: Patch Available) > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch, HDFS-11197-6.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Status: Patch Available (was: In Progress) > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Status: In Progress (was: Patch Available) > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Attachment: HDFS-11197-5.patch Thanks for confirming on this, [~xiaochen]! I'm submitting another patch with the whitespace fix, plus the code conventions changes on the test class. Please let me know if you have any additional comments on this. > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Status: Patch Available (was: In Progress) > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Attachment: HDFS-11197-4.patch Just removing the extra logging message from "EncryptionZoneManager.listEncryptionZones" I had left on the previous patch. > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch, HDFS-11197-4.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Attachment: HDFS-11197-3.patch For some reason, the changes I mentioned on my previous comment were not included in the last patch. Including a 3rd patch with the fix for the test and the additional unit test for checking this condition on TestEncryptionZoneManager. The findbugs warning does not seem related to any of the changes from this patch. > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Status: In Progress (was: Patch Available) > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch, > HDFS-11197-3.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Attachment: HDFS-11197-2.patch Second patch with the checkstyle issues fixed for TestEncryptionZoneManager class. Apologies for not had ran checkstyles locally previously, had succesfully ran it now. Also changed to comply the situation from the failed tests *org.apache.hadoop.hdfs.TestEncryptionZones* and *org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS*, where the EZ is actually the root folder. Added a 3rd unit test to cover this case. Tests are passing locally, now. > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Status: In Progress (was: Patch Available) > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Status: Patch Available (was: In Progress) > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Priority: Minor (was: Major) > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Attachments: HDFS-11197-1.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Status: Patch Available (was: Open) First patch proposal. > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil > Attachments: HDFS-11197-1.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones* does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Description: If a EZ directory is under a snapshotable directory, and a snapshot has been taking, then if this EZ is permanently deleted, it causes *hdfs crypto listZones* command to fail without showing any of the still available zones. This happens only after the EZ is removed from Trash folder. For example, considering */test-snap* folder is snapshotable and there is already an snapshot for it: {noformat} $ hdfs crypto -listZones /user/systest my-key /test-snap/EZ-1 my-key $ hdfs dfs -rmr /test-snap/EZ-1 INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 $ hdfs crypto -listZones /user/systest my-key /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 $ hdfs crypto -listZones RemoteException: Absolute path required {noformat} Once this error happens, *hdfs crypto -listZones* only works again if we remove the snapshot: {noformat} $ hdfs dfs -deleteSnapshot /test-snap snap1 $ hdfs crypto -listZones /user/systest my-key {noformat} If we instead delete the EZ using *skipTrash* option, *hdfs crypto -listZones* does not break: {noformat} $ hdfs crypto -listZones /user/systest my-key /test-snap/EZ-2 my-key $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 Deleted /test-snap/EZ-2 $ hdfs crypto -listZones /user/systest my-key {noformat} The different behaviour seems to be because when removing the EZ trash folder, it's related INode is left with no parent INode. This causes *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when trying to resolve the inodes in the given path. Am proposing a patch that fixes this issue by simply performing an additional check on *EncryptionZoneManager.listEncryptionZones* for the case an inode has no parent, so that it would be skipped on the list without trying to resolve it. Feedback on the proposal is appreciated. was: If a EZ directory is under a snapshotable directory, and a snapshot has been taking, then if this EZ is permanently deleted, it causes *hdfs crypto listZones* command to fail without showing any of the still available zones. This happens only after the EZ is removed from Trash folder. For example, considering */test-snap* folder is snapshotable and there is already an snapshot for it: {noformat} $ hdfs crypto -listZones /user/systest my-key /test-snap/EZ-1 my-key $ hdfs dfs -rmr /test-snap/EZ-1 INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 $ hdfs crypto -listZones /user/systest my-key /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 $ hdfs crypto -listZones RemoteException: Absolute path required {noformat} Once this error happens, *hdfs crypto -listZones* only works again if we remove the snapshot: {noformat} $ hdfs dfs -deleteSnapshot /test-snap snap1 $ hdfs crypto -listZones /user/systest my-key {noformat} If we instead delete the EZ using *skipTrash* option, *hdfs crypto -listZones" does not break: {noformat} $ hdfs crypto -listZones /user/systest my-key /test-snap/EZ-2 my-key $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 Deleted /test-snap/EZ-2 $ hdfs crypto -listZones /user/systest my-key {noformat} The different behaviour seems to be because when removing the EZ trash folder, it's related INode is left with no parent INode. This causes *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when trying to resolve the inodes in the given path. Am proposing a patch that fixes this issue by simply performing an additional check on *EncryptionZoneManager.listEncryptionZones* for the case an inode has no parent, so that it would be skipped on the list without trying to resolve it. Feedback on the proposal is appreciated. > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil > Attachments: HDFS-11197-1.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[ https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-11197: Attachment: HDFS-11197-1.patch > Listing encryption zones fails when deleting a EZ that is on a snapshotted > directory > > > Key: HDFS-11197 > URL: https://issues.apache.org/jira/browse/HDFS-11197 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil > Attachments: HDFS-11197-1.patch > > > If a EZ directory is under a snapshotable directory, and a snapshot has been > taking, then if this EZ is permanently deleted, it causes *hdfs crypto > listZones* command to fail without showing any of the still available zones. > This happens only after the EZ is removed from Trash folder. For example, > considering */test-snap* folder is snapshotable and there is already an > snapshot for it: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-1 my-key > $ hdfs dfs -rmr /test-snap/EZ-1 > INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at: > hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > /user/systest my-key > /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key > $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1 > Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1 > $ hdfs crypto -listZones > RemoteException: Absolute path required > {noformat} > Once this error happens, *hdfs crypto -listZones* only works again if we > remove the snapshot: > {noformat} > $ hdfs dfs -deleteSnapshot /test-snap snap1 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > If we instead delete the EZ using *skipTrash* option, *hdfs crypto > -listZones" does not break: > {noformat} > $ hdfs crypto -listZones > /user/systest my-key > /test-snap/EZ-2 my-key > $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2 > Deleted /test-snap/EZ-2 > $ hdfs crypto -listZones > /user/systest my-key > {noformat} > The different behaviour seems to be because when removing the EZ trash > folder, it's related INode is left with no parent INode. This causes > *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when > trying to resolve the inodes in the given path. > Am proposing a patch that fixes this issue by simply performing an additional > check on *EncryptionZoneManager.listEncryptionZones* for the case an inode > has no parent, so that it would be skipped on the list without trying to > resolve it. Feedback on the proposal is appreciated. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org