[ https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136469#comment-16136469 ]
Xiao Chen edited comment on HDFS-10899 at 8/23/17 4:26 AM: ----------------------------------------------------------- bq. ReencryptionUpdater#throttle(): updater would keep contending for namenode lock {{batchService.take();}} is a blocking call, so hangs there if 'nothing to do', so NN lock untouched. 1.0 means no throttling, so would be touch on locking - that's because this is intended to be run in a maintenance window. Same reason why renames are disabled during this time. Throttler also considers how many tasks are pending, to prevent piling up tasks on NN heap. bq. ... ZoneSubmissionTracker#tasks.... If there will always be just one ReencryptionHandler, then this is okay. Good analysis. Yes, one handler. bq. the edit log is written only when all tasks are successful. Edit: I think I misread your comment. Yes, in case of failures, we best effort by telling admin 'there are failures, please examine and rerun'. was (Author: xiaochen): bq. ReencryptionUpdater#throttle(): updater would keep contending for namenode lock {{batchService.take();}} is a blocking call, so hangs there if 'nothing to do', so NN lock untouched. 1.0 means no throttling, so would be touch on locking - that's because this is intended to be run in a maintenance window. Same reason why renames are disabled during this time. Throttler also considers how many tasks are pending, to prevent piling up tasks on NN heap. bq. ... ZoneSubmissionTracker#tasks.... If there will always be just one ReencryptionHandler, then this is okay. Good analysis. Yes, one handler. bq. the edit log is written only when all tasks are successful. That {{updateReencryptionProgress}} call is to update the zone node with the progress. The actual file xattrs (aka. new edeks) are logged during the processing of each batch, via {{FSDirEncryptionZoneOp.setFileEncryptionInfo}} > Add functionality to re-encrypt EDEKs > ------------------------------------- > > Key: HDFS-10899 > URL: https://issues.apache.org/jira/browse/HDFS-10899 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption, kms > Reporter: Xiao Chen > Assignee: Xiao Chen > Attachments: editsStored, HDFS-10899.01.patch, HDFS-10899.02.patch, > HDFS-10899.03.patch, HDFS-10899.04.patch, HDFS-10899.05.patch, > HDFS-10899.06.patch, HDFS-10899.07.patch, HDFS-10899.08.patch, > HDFS-10899.09.patch, HDFS-10899.10.patch, HDFS-10899.10.wip.patch, > HDFS-10899.11.patch, HDFS-10899.12.patch, HDFS-10899.13.patch, > HDFS-10899.14.patch, HDFS-10899.15.patch, HDFS-10899.wip.2.patch, > HDFS-10899.wip.patch, Re-encrypt edek design doc.pdf, Re-encrypt edek design > doc V2.pdf > > > Currently when an encryption zone (EZ) key is rotated, it only takes effect > on new EDEKs. We should provide a way to re-encrypt EDEKs after the EZ key > rotation, for improved security. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org