[
https://issues.apache.org/jira/browse/HDFS-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319807#comment-15319807
]
Uma Maheswara Rao G commented on HDFS-10473:
--------------------------------------------
Thanks a lot Jing for taking look.
{quote}
My understanding is policies like "WARM" and "ONE_SSD" are mainly targeting
replication (since they're mainly setting specific storage type for the first
replica) thus are not suitable. Could you please confirm it?
{quote}
Yes. You are right.
{quote}
For the patch, storage policies are mainly set on directories (in fact to set
storage policies on files is not recommended), and we allow moving EC files
across EC directory boundaries. Therefore it is not possible to disallow
setting storage policies on striped file in O(1) time complexity. Looks like
the changes on the NN side may be unnecessary here. We only need to let Mover
ignore striped files for now.
{quote}
In reality yes. Currently in NN we just disable for files only if some one
sets. Yes you are right we can not disable for each level of file here. This
handle while running mover only.
Actual plan is to find the suitable policies and enable only for them. At first
step we thought we will disable and then think more carefully what policies
suitable. Yes, we can think now itself and do full changes.
Here is how I am thinking :
Mover is the key tool here who moves the file blocks. So, lets define EC
allowed policies. i.e, ALL_SSD, ARCHIVE, etc
Since policies are static, lets keep allowed list statically in Code. When
mover attempt move striped files, if the targeted policy is either of the them,
then we will just proceed for that file, otherwise we will just skip.
For the files specifically if someone attempting to set other than above
policy, then we don't allow straightway by rejecting call. We can not do on
directory case because it applies for many files under it. some of them may be
non ec file directories. May be when listing policies for EC files, we should
ignore if inherited one other than above list?
What do you say?
> Allow only suitable storage policies to be set on striped files
> ---------------------------------------------------------------
>
> Key: HDFS-10473
> URL: https://issues.apache.org/jira/browse/HDFS-10473
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-10473-01.patch
>
>
> Currently existing storage policies are not suitable for striped layout files.
> This JIRA proposes to reject setting storage policy on striped files.
> Another thought is to allow only suitable storage polices like ALL_SSD.
> Since the major use case of EC is for cold data, this may not be at high
> importance. So, I am ok to reject setting storage policy on striped files at
> this stage. Please suggest if others have some thoughts on this.
> Thanks [~zhz] for offline discussion on this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
