HDFS-9843. Document distcp options required for copying between encrypted locations. Contributed by Xiaoyu Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/dbbfc58c Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/dbbfc58c Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/dbbfc58c Branch: refs/heads/HDFS-1312 Commit: dbbfc58c33fd1d2f7abae1784c2d78b7438825e2 Parents: 47b92f2 Author: Chris Nauroth <[email protected]> Authored: Wed Feb 24 15:16:05 2016 -0800 Committer: Chris Nauroth <[email protected]> Committed: Wed Feb 24 15:16:05 2016 -0800 ---------------------------------------------------------------------- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++ .../hadoop-hdfs/src/site/markdown/TransparentEncryption.md | 8 ++++---- 2 files changed, 7 insertions(+), 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/dbbfc58c/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index c31e768..08a270d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -1965,6 +1965,9 @@ Release 2.8.0 - UNRELEASED HDFS-9854. Log cipher suite negotiation more verbosely (Wei-Chiu Chuang via cnauroth) + HDFS-9843. Document distcp options required for copying between encrypted + locations. (Xiaoyu Yao via cnauroth) + OPTIMIZATIONS HDFS-8026. Trace FSOutputSummer#writeChecksumChunks rather than http://git-wip-us.apache.org/repos/asf/hadoop/blob/dbbfc58c/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md index 8314ed5..3b04255 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md @@ -32,7 +32,7 @@ Transparent Encryption in HDFS * [Example usage](#Example_usage) * [Distcp considerations](#Distcp_considerations) * [Running as the superuser](#Running_as_the_superuser) - * [Copying between encrypted and unencrypted locations](#Copying_between_encrypted_and_unencrypted_locations) + * [Copying into encrypted locations](#Copying_into_encrypted_locations) * [Rename and Trash considerations](#Rename_and_Trash_considerations) * [Attack vectors](#Attack_vectors) * [Hardware access exploits](#Hardware_access_exploits) @@ -207,11 +207,11 @@ One common usecase for distcp is to replicate data between clusters for backup a To enable this same workflow when using HDFS encryption, we introduced a new virtual path prefix, `/.reserved/raw/`, that gives superusers direct access to the underlying block data in the filesystem. This allows superusers to distcp data without needing having access to encryption keys, and also avoids the overhead of decrypting and re-encrypting data. It also means the source and destination data will be byte-for-byte identical, which would not be true if the data was being re-encrypted with a new EDEK. -When using `/.reserved/raw` to distcp encrypted data, it's important to preserve extended attributes with the [-px](#a-px) flag. This is because encrypted file attributes (such as the EDEK) are exposed through extended attributes within `/.reserved/raw`, and must be preserved to be able to decrypt the file. This means that if the distcp is initiated at or above the encryption zone root, it will automatically create an encryption zone at the destination if it does not already exist. However, it's still recommended that the admin first create identical encryption zones on the destination cluster to avoid any potential mishaps. +When using `/.reserved/raw` to distcp encrypted data, it's important to preserve extended attributes with the [-px](../../hadoop-distcp/DistCp.html#Command_Line_Options) flag. This is because encrypted file attributes (such as the EDEK) are exposed through extended attributes within `/.reserved/raw`, and must be preserved to be able to decrypt the file. This means that if the distcp is initiated at or above the encryption zone root, it will automatically create an encryption zone at the destination if it does not already exist. However, it's still recommended that the admin first create identical encryption zones on the destination cluster to avoid any potential mishaps. -### <a name="Copying_between_encrypted_and_unencrypted_locations"></a>Copying between encrypted and unencrypted locations +### <a name="Copying_into_encrypted_locations"></a>Copying into encrypted locations -By default, distcp compares checksums provided by the filesystem to verify that the data was successfully copied to the destination. When copying between an unencrypted and encrypted location, the filesystem checksums will not match since the underlying block data is different. In this case, specify the [-skipcrccheck](#a-skipcrccheck) and [-update](#a-update) distcp flags to avoid verifying checksums. +By default, distcp compares checksums provided by the filesystem to verify that the data was successfully copied to the destination. When copying from unencrypted or encrypted location into an encrypted location, the filesystem checksums will not match since the underlying block data is different because a new EDEK will be used to encrypt at destination. In this case, specify the [-skipcrccheck](../../hadoop-distcp/DistCp.html#Command_Line_Options) and [-update](../../hadoop-distcp/DistCp.html#Command_Line_Options) distcp flags to avoid verifying checksums. <a name="Rename_and_Trash_considerations"></a>Rename and Trash considerations ---------------------
