[jira] [Work logged] (HDFS-15611) Add list Snapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?focusedWorklogId=493826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-493826 ] ASF GitHub Bot logged work on HDFS-15611: - Author: ASF GitHub Bot Created on: 02/Oct/20 05:43 Start Date: 02/Oct/20 05:43 Worklog Time Spent: 10m Work Description: bshashikant commented on pull request #2355: URL: https://github.com/apache/hadoop/pull/2355#issuecomment-702536750 > Thanks @bshashikant . Should we add `SnapshotStatus[] getSnapshotListing(Path snapshotRoot)` > to FileSystem? If yes, we should move SnapshotStatus to org.apache.hadoop.fs and declare dirStatus using FileStatus. Thanks @szetszwo . I checked the code and saw that "getSnapshottableDirectoryList()" is also not moved to Filesystem class yet. I would prefer no to move any of these to Filesystem class here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 493826) Time Spent: 0.5h (was: 20m) > Add list Snapshot command in WebHDFS > > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15611) Add list Snapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?focusedWorklogId=493809=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-493809 ] ASF GitHub Bot logged work on HDFS-15611: - Author: ASF GitHub Bot Created on: 02/Oct/20 03:43 Start Date: 02/Oct/20 03:43 Worklog Time Spent: 10m Work Description: szetszwo commented on a change in pull request #2355: URL: https://github.com/apache/hadoop/pull/2355#discussion_r498605120 ## File path: hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java ## @@ -1459,6 +1460,19 @@ SnapshotDiffReport decodeResponse(Map json) { }.run(); } + public SnapshotStatus[] getSnapshotList(final Path snapshotDir) Review comment: In DistributedFileSystem, it is called ` public SnapshotStatus[] getSnapshotListing(Path snapshotRoot)` Let's use the same name? ## File path: hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshotStatus.java ## @@ -25,6 +25,7 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.permission.FsPermission; import org.apache.hadoop.hdfs.DFSUtilClient; +import org.apache.hadoop.util.StringUtils; Review comment: Unused import. ## File path: hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java ## @@ -1582,6 +1584,18 @@ public SnapshotDiffReport getSnapshotDiffReport(Path path, return JsonUtilClient.toSnapshottableDirectoryList(json); } + public SnapshotStatus[] getSnapshotList(Path snapshotRoot) Review comment: It should call getSnapshotListing. ## File path: hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java ## @@ -872,4 +873,39 @@ private static SnapshottableDirectoryStatus toSnapshottableDirectoryStatus( snapshotQuota, parentFullPath); return snapshottableDirectoryStatus; } + + public static SnapshotStatus[] toSnapshotList(final Map json) { +if (json == null) { + return null; +} +List list = (List) json.get("SnapshotList"); +if (list == null) { + return null; +} +SnapshotStatus[] statuses = +new SnapshotStatus[list.size()]; +for (int i = 0; i < list.size(); i++) { + statuses[i] = toSnapshotStatus((Map) list.get(i)); +} +return statuses; + } + + private static SnapshotStatus toSnapshotStatus( + Map json) { +if (json == null) { + return null; +} +int snapshotID = getInt(json, "snapshotID", 0); +boolean isDeleted = ((String)json.get("deletionStatus")). +contentEquals("DELETED"); Review comment: Use "DELETED".equal(..) in order to avoid NPE? ` final boolean isDeleted = "DELETED".equals(json.get("deletionStatus"));` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 493809) Time Spent: 20m (was: 10m) > Add list Snapshot command in WebHDFS > > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15567) [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicetly.
[ https://issues.apache.org/jira/browse/HDFS-15567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-15567: --- Affects Version/s: 2.10.0 Assignee: Konstantin Shvachko Status: Patch Available (was: Open) Decided to go with option 1 exposing {{msync}} via {{FileSystem}} and {{FileContext}}. See patch attached. We can further implement {{msync}} for {{ViewFileSystem}}, which would try to call it on all mount points, whichever supports it. {{WebHDFS}} is not built for that as it doesn't have {{AlignmentContext}} and therefore cannot pass state ids between NameNode and clients. > [SBN Read] HDFS should expose msync() API to allow downstream applications > call it explicetly. > -- > > Key: HDFS-15567 > URL: https://issues.apache.org/jira/browse/HDFS-15567 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, hdfs-client >Affects Versions: 2.10.0 >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko >Priority: Major > Attachments: HDFS-15567.001.patch > > > Consistent reads from Standby introduced {{msync()}} API HDFS-13688, which > updates client's state ID with current state of the Active NameNode to > guarantee consistency of subsequent calls to an ObserverNode. Currently this > API is exposed via {{DFSClient}} only, which makes it hard for applications > to access {{msync()}}. One way is to use something like this: > {code} > if(fs instanceof DistributedFileSystem) { > ((DistributedFileSystem)fs).getClient().msync(); > } > {code} > This should be exposed both for {{FileSystem}} and {{FileContext}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15567) [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicetly.
[ https://issues.apache.org/jira/browse/HDFS-15567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-15567: --- Attachment: HDFS-15567.001.patch > [SBN Read] HDFS should expose msync() API to allow downstream applications > call it explicetly. > -- > > Key: HDFS-15567 > URL: https://issues.apache.org/jira/browse/HDFS-15567 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, hdfs-client >Reporter: Konstantin Shvachko >Priority: Major > Attachments: HDFS-15567.001.patch > > > Consistent reads from Standby introduced {{msync()}} API HDFS-13688, which > updates client's state ID with current state of the Active NameNode to > guarantee consistency of subsequent calls to an ObserverNode. Currently this > API is exposed via {{DFSClient}} only, which makes it hard for applications > to access {{msync()}}. One way is to use something like this: > {code} > if(fs instanceof DistributedFileSystem) { > ((DistributedFileSystem)fs).getClient().msync(); > } > {code} > This should be exposed both for {{FileSystem}} and {{FileContext}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15612) WebHDFS: Support provisionSnapshotTrash
[ https://issues.apache.org/jira/browse/HDFS-15612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15612: -- Description: In HDFS-15607 we are adding snapshot trash root provision command to help create the trash root with the right permission. But in order to add this feature to WebHDFS (and HttpFS), we might have to copy or move the client-side logic to the server since the logic for directly accessing WebHDFS by REST API is on the NameNode {{NamenodeWebHdfsMethods}}. was: In HDFS-15607 we are adding snapshot trash root provision command to help create the trash root with the right permission. But in order to add this feature to WebHDFS, we might have to copy or move the client-side logic to the server since the logic for directly accessing WebHDFS by REST API is on the NameNode {{NamenodeWebHdfsMethods}}. > WebHDFS: Support provisionSnapshotTrash > --- > > Key: HDFS-15612 > URL: https://issues.apache.org/jira/browse/HDFS-15612 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Siyao Meng >Priority: Minor > > In HDFS-15607 we are adding snapshot trash root provision command to help > create the trash root with the right permission. But in order to add this > feature to WebHDFS (and HttpFS), we might have to copy or move the > client-side logic to the server since the logic for directly accessing > WebHDFS by REST API is on the NameNode {{NamenodeWebHdfsMethods}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15607) Create trash dir when allowing snapshottable dir
[ https://issues.apache.org/jira/browse/HDFS-15607?focusedWorklogId=493611=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-493611 ] ASF GitHub Bot logged work on HDFS-15607: - Author: ASF GitHub Bot Created on: 01/Oct/20 17:46 Start Date: 01/Oct/20 17:46 Worklog Time Spent: 10m Work Description: smengcl edited a comment on pull request #2352: URL: https://github.com/apache/hadoop/pull/2352#issuecomment-701586546 > Do we need to add provisionTrash command for WebHdfs as well? Good point. I think so, yes. Update: As I was attempting to add `PROVISIONSNAPSHOTTRASH` to WebHDFS, I realized `NamenodeWebHdfsMethods` is already server code. But the whole provision trash logic are on the **client** side (same as provision EZ trash). This would also imply that WebHDFS `ALLOWSNAPSHOT` won't trigger provision snapshot trash at the moment. Note WebHDFS doesn't support encryption zone commands (create, list, etc.). I have opened another jira [HDFS-15612](https://issues.apache.org/jira/browse/HDFS-15612) for discussion on WebHDFS support for provision snapshot trash. Let's rule WebHDFS out in this jira for now. @bshashikant This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 493611) Time Spent: 1h 40m (was: 1.5h) > Create trash dir when allowing snapshottable dir > > > Key: HDFS-15607 > URL: https://issues.apache.org/jira/browse/HDFS-15607 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Affects Versions: 3.4.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with > permission 700 (and without sticky bit) by the first user that moves a file > to the trash. This is an issue when other users try to move files to that > trash because they may not have the permission to move to that trash if the > trash root is shared. -- in this case, snapshottable directories. > This only affects users when trash is enabled inside snapshottable > directories ({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and > when a user performing move to trash operations doesn't have admin > permissions. > Solution: Create a {{.Trash}} directory with 777 permission and sticky bits > enabled (similar solution as HDFS-10324). > Also need to deal with some corner cases: > 1. even when the snapshottable directory trash root config is not enabled > ({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the > {{.Trash}} directory anyway? Or should we ask the admin to provision trash > manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an > existing cluster? > - If the cluster is just upgraded, we need to provision trash manually anyway. > 2. When immediately disallowing trash, it shouldn't fail. just remove the > .Trash directory when disallowing snapshot on a dir if it is empty? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15612) WebHDFS:
Siyao Meng created HDFS-15612: - Summary: WebHDFS: Key: HDFS-15612 URL: https://issues.apache.org/jira/browse/HDFS-15612 Project: Hadoop HDFS Issue Type: Improvement Reporter: Siyao Meng In HDFS-15607 we are adding snapshot trash root provision command to help create the trash root with the right permission. But in order to add this feature to WebHDFS, we might have to copy or move the client-side logic to the server since the logic for directly accessing WebHDFS by REST API is on the NameNode {{NamenodeWebHdfsMethods}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15612) WebHDFS: Support provisionSnapshotTrash
[ https://issues.apache.org/jira/browse/HDFS-15612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15612: -- Summary: WebHDFS: Support provisionSnapshotTrash (was: WebHDFS: ) > WebHDFS: Support provisionSnapshotTrash > --- > > Key: HDFS-15612 > URL: https://issues.apache.org/jira/browse/HDFS-15612 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Siyao Meng >Priority: Minor > > In HDFS-15607 we are adding snapshot trash root provision command to help > create the trash root with the right permission. But in order to add this > feature to WebHDFS, we might have to copy or move the client-side logic to > the server since the logic for directly accessing WebHDFS by REST API is on > the NameNode {{NamenodeWebHdfsMethods}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15611) Add list Snapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15611: -- Labels: pull-request-available (was: ) > Add list Snapshot command in WebHDFS > > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15611) Add list Snapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?focusedWorklogId=493482=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-493482 ] ASF GitHub Bot logged work on HDFS-15611: - Author: ASF GitHub Bot Created on: 01/Oct/20 13:12 Start Date: 01/Oct/20 13:12 Worklog Time Spent: 10m Work Description: bshashikant opened a new pull request #2355: URL: https://github.com/apache/hadoop/pull/2355 please check https://issues.apache.org/jira/browse/HDFS-15611 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 493482) Remaining Estimate: 0h Time Spent: 10m > Add list Snapshot command in WebHDFS > > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.4.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15611) Add list Snapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee reassigned HDFS-15611: -- Assignee: Shashikant Banerjee > Add list Snapshot command in WebHDFS > > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.4.0 > > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15611) Add list Snapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-15611: --- Summary: Add list Snapshot command in WebHDFS (was: Add lsSnapshot command in WebHDFS) > Add list Snapshot command in WebHDFS > > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Priority: Major > Fix For: 3.4.0 > > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15611) Add lsSnapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-15611: --- Summary: Add lsSnapshot command in WebHDFS (was: Add lsSnapshot command in HDFS) > Add lsSnapshot command in WebHDFS > - > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Priority: Major > Fix For: 3.4.0 > > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15611) Add lsSnapshot command in HDFS
Shashikant Banerjee created HDFS-15611: -- Summary: Add lsSnapshot command in HDFS Key: HDFS-15611 URL: https://issues.apache.org/jira/browse/HDFS-15611 Project: Hadoop HDFS Issue Type: Sub-task Components: snapshots Reporter: Shashikant Banerjee Fix For: 3.4.0 Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org