Attila Bukor has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18025 )
Change subject: [security] KUDU-3316 Add encrypted file keys ...................................................................... [security] KUDU-3316 Add encrypted file keys The previous patches in the data at rest encryption saga add the ability to encrypt data at rest, but files were encrypted using the same hard-coded key. This patch adds an extra header to encrypted files to store the encryption algorithm used and the encrypted file key. For now, the file keys are encrypted with the same dummy encryption key which was previously used to encrypt the files. The header is a bit different from the one described in the design doc: The encryption algorithm and key length was changed to be stored in 1 byte instead of 2 for easier handling and the magic string is "kuduenc" instead of "kuduen". This patch also introduces a new flag which is hidden for now: --encryption_key_length. This can be set to any valid AES key length as per its specification (128, 192, or 256 bits), as only AES encryption is supported for now, and there are no plans to support anything else in the foreseeable future. As we add a 64-byte header to encrypted files, some changes had to be made to code that handles files and relies on sizes and offsets, including in tests. One of these changes is in the LogBlockManager, which expects blocks to be aligned to file system block boundaries, which is necessary for hole punching. With encryption enabled, extra steps are necessary to align the first block, and aligning blocks makes it impractical to hide encryption header size within Env and use logical file sizes outside of it. This commit also changes the PBC tool to check if a file is encrypted based on the encryption header instead of the file name. I ran the full test suite manually locally and on dist-test with encryption enabled to make sure turning on encryption doesn't break anything: http://dist-test.cloudera.org/job?job_id=abukor.1643215963.60435 To make running dist-test with encryption enabled possible, this commit also adds forwarding the KUDU_ENCRYPT_DATA_IN_TESTS environment variable to dist_test.py. Change-Id: Idb1282c117271fda63a8cc54c00add7cc96dcffd Reviewed-on: http://gerrit.cloudera.org:8080/18025 Reviewed-by: Alexey Serbin <[email protected]> Tested-by: Kudu Jenkins Reviewed-by: Andrew Wong <[email protected]> --- M build-support/dist_test.py M src/kudu/consensus/consensus_meta-test.cc M src/kudu/consensus/log-test.cc M src/kudu/consensus/log_index.cc M src/kudu/consensus/log_util.cc M src/kudu/fs/dir_util.cc M src/kudu/fs/file_block_manager.cc M src/kudu/fs/log_block_manager-test-util.cc M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/mini-cluster/external_mini_cluster.cc M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_pbc.cc M src/kudu/tserver/tablet_copy_client-test.cc M src/kudu/tserver/tablet_copy_service-test.cc M src/kudu/tserver/tablet_copy_source_session-test.cc M src/kudu/tserver/tablet_copy_source_session.cc M src/kudu/util/env-test.cc M src/kudu/util/env.cc M src/kudu/util/env.h M src/kudu/util/env_posix.cc M src/kudu/util/env_util.cc M src/kudu/util/env_util.h M src/kudu/util/file_cache-stress-test.cc M src/kudu/util/file_cache-test.cc M src/kudu/util/file_cache.cc M src/kudu/util/pb_util-test.cc M src/kudu/util/pb_util.cc 28 files changed, 715 insertions(+), 255 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved Kudu Jenkins: Verified Andrew Wong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18025 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idb1282c117271fda63a8cc54c00add7cc96dcffd Gerrit-Change-Number: 18025 Gerrit-PatchSet: 19 Gerrit-Owner: Attila Bukor <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Attila Bukor <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Zoltan Chovan <[email protected]>
