[jira] [Commented] (HDFS-11203) Rename support during re-encrypt EDEK
[ https://issues.apache.org/jira/browse/HDFS-11203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17802525#comment-17802525 ] Shilun Fan commented on HDFS-11203: --- Bulk update: moved all 3.4.0 non-blocker issues, please move back if it is a blocker. Retarget 3.5.0. > Rename support during re-encrypt EDEK > - > > Key: HDFS-11203 > URL: https://issues.apache.org/jira/browse/HDFS-11203 > Project: Hadoop HDFS > Issue Type: Task > Components: encryption >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > > Currently HDFS-10899 disables renames within the EZ if it's under > re-encryption. (similar to current cross-zone rename checks). > We'd like to support rename in the long run, so cluster is fully functioning > during re-encryption. > The reason rename is particularly difficult is: > - We want to re-encrypt all files under an EZ in one pass, without missing any > - We want to iterate through the files and keep track of where we are (i.e. a > cursor), so in case of NN failover/crash, we can resume from fsimage/edits. > - We cannot guarantee namespace is not changed during re-encryption. Newly > created files automatically has new edek, deleted files we don't care. But if > a file is renamed from behind the cursor to before, it may be missed in the > re-encryption. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11203) Rename support during re-encrypt EDEK
[ https://issues.apache.org/jira/browse/HDFS-11203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104409#comment-16104409 ] Xiao Chen commented on HDFS-11203: -- Hi [~daryn], I discussed with [~andrew.wang] today, and our speculation is you're proposing only about re-resolution, not how we iterate the EZ. I have edited the description of this jira to state the problem clearer. Could you check and see if we're on the same page? Thanks. > Rename support during re-encrypt EDEK > - > > Key: HDFS-11203 > URL: https://issues.apache.org/jira/browse/HDFS-11203 > Project: Hadoop HDFS > Issue Type: Task > Components: encryption >Reporter: Xiao Chen >Assignee: Xiao Chen > > Currently HDFS-10899 disables renames within the EZ if it's under > re-encryption. (similar to current cross-zone rename checks). > We'd like to support rename in the long run, so cluster is fully functioning > during re-encryption. > The reason rename is particularly difficult is: > - We want to re-encrypt all files under an EZ in one pass, without missing any > - We want to iterate through the files and keep track of where we are (i.e. a > cursor), so in case of NN failover/crash, we can resume from fsimage/edits. > - We cannot guarantee namespace is not changed during re-encryption. Newly > created files automatically has new edek, deleted files we don't care. But if > a file is renamed from behind the cursor to before, it may be missed in the > re-encryption. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11203) Rename support during re-encrypt EDEK
[ https://issues.apache.org/jira/browse/HDFS-11203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102191#comment-16102191 ] Xiao Chen commented on HDFS-11203: -- Thanks for the comment [~daryn]! You're correct, the problem is: - Re-encryption should happen to all files inside an EZ - Currently we try to walk the EZ's path, and since locks can't be hold for the whole process, we need a way to be aware of renames. HDFS-10899 currently just disables renames within the EZ if it's under re-encryption. (similar to current cross-zone rename checks) Could you elaborate your thoughts on inodeid? IIUC, you mean instead of using current file's path as a cursor, we use the current inodeid, and walk through the inodemap? That should solve the rename problem (I'll need more research to confirm), but also we may have to walk the entire file system even if the admin just want to re-encrypt a small zone. Or did I miss something? > Rename support during re-encrypt EDEK > - > > Key: HDFS-11203 > URL: https://issues.apache.org/jira/browse/HDFS-11203 > Project: Hadoop HDFS > Issue Type: Task > Components: encryption >Reporter: Xiao Chen >Assignee: Xiao Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11203) Rename support during re-encrypt EDEK
[ https://issues.apache.org/jira/browse/HDFS-11203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16102158#comment-16102158 ] Daryn Sharp commented on HDFS-11203: What is the actual problem statement? Presumably it relates to releasing and reacquiring the lock during the reencrypt? If yes, that is generally only a problem if using path based re-resolution. It becomes less of an exercise if it's inodeId based. It makes it easier to detect deletes too. > Rename support during re-encrypt EDEK > - > > Key: HDFS-11203 > URL: https://issues.apache.org/jira/browse/HDFS-11203 > Project: Hadoop HDFS > Issue Type: Task > Components: encryption >Reporter: Xiao Chen >Assignee: Xiao Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
