[jira] [Updated] (HDFS-6509) create a /.reserved/raw filesystem namespace
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6509: --- Attachment: HDFS-6509.003.patch .003 addresses the issues you mention. I also removed the fsd arg to fsn#resolvePath since it was always receiving the FSN#dir member var. create a /.reserved/raw filesystem namespace Key: HDFS-6509 URL: https://issues.apache.org/jira/browse/HDFS-6509 Project: Hadoop HDFS Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6509.001.patch, HDFS-6509.002.patch, HDFS-6509.003.patch, HDFS-6509distcpandDataatRestEncryption-2.pdf, HDFS-6509distcpandDataatRestEncryption-3.pdf, HDFS-6509distcpandDataatRestEncryption.pdf This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6509) create a /.reserved/raw filesystem namespace
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6509: --- Attachment: HDFS-6509.002.patch [~andrew.wang], Thanks for your review. I've attached a .002 version which I believe addresses your comments. FSN and FSDir: bq. You can just do isReservedRawName(src) in place rather than declaring a new variable if it's not being reused. Very cheap function too. But since resolvePathname changes src to remove /.reserved/raw, I needed to call iiRRN prior to the resolvePathname call. It's a moot point now that I've fixed the logAudit calls to log the /.reserved/raw pathnames. bq. Lines longer than 80 chars I fixed the one in constructRemainingPath (I left that in the first diffs since the diffs didn't show that particular line as a change). Were there others besides that one? The comments for resolvePath ran right up to 80, but I don't think they were over 80. bq. Need p/ to line break in a javadoc comment. Fixed. bq. I shy away from the term namespace for raw, can we just say directory? It is just a directory after all. Fixed. bq. FSDirectory, not a fan of even mildly complicated ternary statements. Can we turn these into ifs, or at least not inline them into method calls? Fixed. There was only one inside a method call. I moved it outside. The other ternary statements didn't look too complicated so I didn't change them, but if there still some that you don't like, let me know. bq. Calling up to FSNamesystem from FSDirectory is a no-no, since we want to separate FSD and FSN locks and promote a lock hierarchy. Can we do the superuser check in FSNamesystem? I've turned all of the FSD.resolvePath() calls in FSN into a new FSN#resolvePath. This looked better than adding a call to to checkAccessForReservedRaw(src) before every call to FSD#resolvePath. bq. Stripping the prefix off is scary because we lose context. At the very least, we should log the original path in the audit log. Various log messages will be more confusing. I think using the stripped path is okay in the edit log. Agree. I've made all of the relevant logAuditEvent calls use the pre-resolvePathname path rather than the non-/.reserved/raw pathname. bq. Some javadocs needs updating for the new isRawPath parameter. I added @param isRawPath to FSD#getFileInfo. I think that was the only one. Test: bq. Helper methods look duplicated from TestEncryptionZones. Let's extract the common code somewhere like DFSTestUtil or GenericTestUtils. Done. bq. Why is the special casing necessary in TestDFSUtil for mkdirs? If mkdirs behaves incorrectly on a raw path, then we should fix mkdirs. A normal user could call mkdirs on a raw path after all. Artifact. Removed. bq. Can we name the test TestReservedRawPaths? Done. bq. The .. tests, let's do some tests where the .. is in between .reserved and raw. Done. bq. There's a dfsadmin defined for the class, why create a new one in each test? The dfsAdmin in testRelativePathnames also isn't used. Fixed. bq. Why can't we list /.reserved? Seems like it should work. That feels a bit ugly. While I agree that it would be nice to be able to list /.reserved and have it show raw and .inodes, we'd have to create and return a DirectoryListing with a pair of HdfsFileStatus instances, one for raw, one for .inodes. There would be a bunch of special case code. If you feel strongly about this, then I'd rather do this work in a followon Jira. bq. Should also assert the entry that we get back is the entry we expect, not just the length. Fixed. create a /.reserved/raw filesystem namespace Key: HDFS-6509 URL: https://issues.apache.org/jira/browse/HDFS-6509 Project: Hadoop HDFS Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6509.001.patch, HDFS-6509.002.patch, HDFS-6509distcpandDataatRestEncryption-2.pdf, HDFS-6509distcpandDataatRestEncryption-3.pdf, HDFS-6509distcpandDataatRestEncryption.pdf This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6509) create a /.reserved/raw filesystem namespace
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6509: --- Attachment: (was: HDFS-6509.001.patch) create a /.reserved/raw filesystem namespace Key: HDFS-6509 URL: https://issues.apache.org/jira/browse/HDFS-6509 Project: Hadoop HDFS Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6509distcpandDataatRestEncryption-2.pdf, HDFS-6509distcpandDataatRestEncryption.pdf This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6509) create a /.reserved/raw filesystem namespace
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6509: --- Attachment: HDFS-6509.001.patch create a /.reserved/raw filesystem namespace Key: HDFS-6509 URL: https://issues.apache.org/jira/browse/HDFS-6509 Project: Hadoop HDFS Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6509.001.patch, HDFS-6509distcpandDataatRestEncryption-2.pdf, HDFS-6509distcpandDataatRestEncryption.pdf This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6509) create a /.reserved/raw filesystem namespace
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6509: --- Attachment: HDFS-6509distcpandDataatRestEncryption-3.pdf The design doc has been updated to reflect choice of raw.* extended attribute namespace and the restriction that only the admin can create/access files in /.reserved/raw. create a /.reserved/raw filesystem namespace Key: HDFS-6509 URL: https://issues.apache.org/jira/browse/HDFS-6509 Project: Hadoop HDFS Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6509.001.patch, HDFS-6509distcpandDataatRestEncryption-2.pdf, HDFS-6509distcpandDataatRestEncryption-3.pdf, HDFS-6509distcpandDataatRestEncryption.pdf This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6509) create a /.reserved/raw filesystem namespace
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6509: --- Attachment: (was: HDFS-6509.001.patch) create a /.reserved/raw filesystem namespace Key: HDFS-6509 URL: https://issues.apache.org/jira/browse/HDFS-6509 Project: Hadoop HDFS Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6509distcpandDataatRestEncryption-2.pdf, HDFS-6509distcpandDataatRestEncryption-3.pdf, HDFS-6509distcpandDataatRestEncryption.pdf This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6509) create a /.reserved/raw filesystem namespace
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6509: --- Attachment: HDFS-6509.001.patch create a /.reserved/raw filesystem namespace Key: HDFS-6509 URL: https://issues.apache.org/jira/browse/HDFS-6509 Project: Hadoop HDFS Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6509.001.patch, HDFS-6509distcpandDataatRestEncryption-2.pdf, HDFS-6509distcpandDataatRestEncryption-3.pdf, HDFS-6509distcpandDataatRestEncryption.pdf This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6509) create a /.reserved/raw filesystem namespace
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6509: --- Description: This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. (was: distcp needs to work with Data At Rest Encryption) Summary: create a /.reserved/raw filesystem namespace (was: distcp vs Data At Rest Encryption) create a /.reserved/raw filesystem namespace Key: HDFS-6509 URL: https://issues.apache.org/jira/browse/HDFS-6509 Project: Hadoop HDFS Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6509distcpandDataatRestEncryption-2.pdf, HDFS-6509distcpandDataatRestEncryption.pdf This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6509) create a /.reserved/raw filesystem namespace
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6509: --- Attachment: HDFS-6509.001.patch create a /.reserved/raw filesystem namespace Key: HDFS-6509 URL: https://issues.apache.org/jira/browse/HDFS-6509 Project: Hadoop HDFS Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6509.001.patch, HDFS-6509distcpandDataatRestEncryption-2.pdf, HDFS-6509distcpandDataatRestEncryption.pdf This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6509) create a /.reserved/raw filesystem namespace
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6509: --- Attachment: (was: HDFS-6509.001.patch) create a /.reserved/raw filesystem namespace Key: HDFS-6509 URL: https://issues.apache.org/jira/browse/HDFS-6509 Project: Hadoop HDFS Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6509.001.patch, HDFS-6509distcpandDataatRestEncryption-2.pdf, HDFS-6509distcpandDataatRestEncryption.pdf This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6509) create a /.reserved/raw filesystem namespace
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6509: --- Attachment: HDFS-6509.001.patch create a /.reserved/raw filesystem namespace Key: HDFS-6509 URL: https://issues.apache.org/jira/browse/HDFS-6509 Project: Hadoop HDFS Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6509.001.patch, HDFS-6509distcpandDataatRestEncryption-2.pdf, HDFS-6509distcpandDataatRestEncryption.pdf This is part of the work for making distcp work with Data at Rest Encryption. Per the attached document, create a /.reserved/raw HDFS filesystem namespace that allows access to the encrypted bytes of a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
