[
https://issues.apache.org/jira/browse/HDFS-14591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869380#comment-16869380
]
Jinglun commented on HDFS-14591:
--------------------------------
Hi [~jojochuang] and [~ayushtkn], thanks for your references, those are very
good works and really inspired. Actually my external service is very like the
SPS. The external service automatically downloads HDFS's image file and
analyzes the image to find out the cold files, then it set xattrs to these
INodes. Another service find the INodes with the special xattrs and start
movers to move them. If I use the SPS, the progress would be: find cold files,
change the policies of them, let external-SPS handle the files. I would wait
SPS be stable and consider changing to it.
SSM is very ambitious and it handles all the replica placement jobs like
balance, mover and storage policy mismatches. It would simplify my maintenance
work. Looking forward to play with it.
I'm considering start a new type of NameNode that only handles the replica
placement. It loads images and edits and receives block reports, there won't be
rpcs to the active so it won't be a burden to my heavy NameNode. It's job is to
scan all blocks and datanodes to find the mismatched blocks and over-used dns,
then move replicas to resolve the problems. Detecting cold files could also be
added to it.
Since there already has issues including storage mismatch problem, I'll close
this issue. Again thanks Wei-Chiu and Ayush for your kindly comments.:)
> NameNode should move the replicas to the correct storages after the storage
> policy is changed.
> ----------------------------------------------------------------------------------------------
>
> Key: HDFS-14591
> URL: https://issues.apache.org/jira/browse/HDFS-14591
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
>
> Our Xiaomi HDFS has a cluster storaging both HOT and COLD data. We have a
> backgroud process searching all the files to find those that are not accessed
> for a period of time. Then we set them to COLD and start a mover to move the
> replicas. After moving, all the replicas are consistent with the storage
> policy.
> It's a natural idea to let the NameNode handle the move.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
