[ https://issues.apache.org/jira/browse/HDFS-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319807#comment-15319807 ]
Uma Maheswara Rao G edited comment on HDFS-10473 at 6/8/16 1:00 AM: -------------------------------------------------------------------- Thanks a lot Jing for taking look. {quote} My understanding is policies like "WARM" and "ONE_SSD" are mainly targeting replication (since they're mainly setting specific storage type for the first replica) thus are not suitable. Could you please confirm it? {quote} Yes. You are right. {quote} For the patch, storage policies are mainly set on directories (in fact to set storage policies on files is not recommended), and we allow moving EC files across EC directory boundaries. Therefore it is not possible to disallow setting storage policies on striped file in O(1) time complexity. Looks like the changes on the NN side may be unnecessary here. We only need to let Mover ignore striped files for now. {quote} In reality yes. In current patch, we just disabled for files only if some one sets. Yes you are right we can not disable for each level of file here in dir case. That handled while running mover only. Actual plan is to find the suitable policies and enable only for them. At first step we thought we will disable and then think more carefully what policies suitable. Yes, we can think now itself and do full changes. Here is how I am thinking : Mover is the key tool here who moves the file blocks. So, lets define EC allowed policies. i.e, ALL_SSD, ARCHIVE, etc Since policies are static, lets keep allowed list statically in Code. When mover attempt move striped files, if the targeted policy is either of the them, then we will just proceed for that file, otherwise we will just skip. For the files specifically if someone attempting to set other than above policy, then we don't allow straightway by rejecting call. We can not do on directory case because it applies for many files under it. some of them may be non ec file directories. May be when listing policies for EC files, we should ignore if inherited one other than above list? What do you say? was (Author: umamaheswararao): Thanks a lot Jing for taking look. {quote} My understanding is policies like "WARM" and "ONE_SSD" are mainly targeting replication (since they're mainly setting specific storage type for the first replica) thus are not suitable. Could you please confirm it? {quote} Yes. You are right. {quote} For the patch, storage policies are mainly set on directories (in fact to set storage policies on files is not recommended), and we allow moving EC files across EC directory boundaries. Therefore it is not possible to disallow setting storage policies on striped file in O(1) time complexity. Looks like the changes on the NN side may be unnecessary here. We only need to let Mover ignore striped files for now. {quote} In reality yes. Currently in NN we just disable for files only if some one sets. Yes you are right we can not disable for each level of file here. This handle while running mover only. Actual plan is to find the suitable policies and enable only for them. At first step we thought we will disable and then think more carefully what policies suitable. Yes, we can think now itself and do full changes. Here is how I am thinking : Mover is the key tool here who moves the file blocks. So, lets define EC allowed policies. i.e, ALL_SSD, ARCHIVE, etc Since policies are static, lets keep allowed list statically in Code. When mover attempt move striped files, if the targeted policy is either of the them, then we will just proceed for that file, otherwise we will just skip. For the files specifically if someone attempting to set other than above policy, then we don't allow straightway by rejecting call. We can not do on directory case because it applies for many files under it. some of them may be non ec file directories. May be when listing policies for EC files, we should ignore if inherited one other than above list? What do you say? > Allow only suitable storage policies to be set on striped files > --------------------------------------------------------------- > > Key: HDFS-10473 > URL: https://issues.apache.org/jira/browse/HDFS-10473 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode > Reporter: Uma Maheswara Rao G > Assignee: Uma Maheswara Rao G > Attachments: HDFS-10473-01.patch > > > Currently existing storage policies are not suitable for striped layout files. > This JIRA proposes to reject setting storage policy on striped files. > Another thought is to allow only suitable storage polices like ALL_SSD. > Since the major use case of EC is for cold data, this may not be at high > importance. So, I am ok to reject setting storage policy on striped files at > this stage. Please suggest if others have some thoughts on this. > Thanks [~zhz] for offline discussion on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org