[
https://issues.apache.org/jira/browse/HDFS-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319754#comment-15319754
]
Jing Zhao commented on HDFS-10473:
----------------------------------
Thanks for working on this, Uma! Could you please explain more on why "existing
storage policies are not suitable for striped layout files" ? My understanding
is policies like "WARM" and "ONE_SSD" are mainly targeting replication (since
they're mainly setting specific storage type for the first replica) thus are
not suitable. Could you please confirm it?
For the patch, storage policies are mainly set on directories (in fact to set
storage policies on files is not recommended), and we allow moving EC files
across EC directory boundaries. Therefore it is not possible to disallow
setting storage policies on striped file in O(1) time complexity. Looks like
the changes on the NN side may be unnecessary here. We only need to let Mover
ignore striped files for now.
However, this change may cause other issue. Since currently the main use case
for EC is cold data, it is very natural for a customer to set a directory as
EC, and set COLD storage policy on the directory. In this way all the EC files
created later under this directory will be placed on Archival storages. We
should keep this semantic since this is a very strong use case, but in the
meanwhile, disabling Mover for EC files will conflict with this semantic: i.e.,
we recognize storage policies during file creation but not afterwards.
Therefore, currently I think we can either 1) make no changes at all and depend
on admin to make the correct decision while setting EC and storage policies, or
2) have a long term plan to fix the issue completely. For #2 maybe the best way
is to bring in Volume concept, since if we have different settings on nested
directories we will have to scan the subtree for validation.
> Allow only suitable storage policies to be set on striped files
> ---------------------------------------------------------------
>
> Key: HDFS-10473
> URL: https://issues.apache.org/jira/browse/HDFS-10473
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-10473-01.patch
>
>
> Currently existing storage policies are not suitable for striped layout files.
> This JIRA proposes to reject setting storage policy on striped files.
> Another thought is to allow only suitable storage polices like ALL_SSD.
> Since the major use case of EC is for cold data, this may not be at high
> importance. So, I am ok to reject setting storage policy on striped files at
> this stage. Please suggest if others have some thoughts on this.
> Thanks [~zhz] for offline discussion on this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
