[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939969#comment-15939969
]
Kai Zheng commented on HDFS-7343:
---------------------------------
Thanks [~zhouwei] for updating the docs and addressing the questions.
[~andrew.wang] has a good asking:
bq. How often does the SSM need to poll the NN to get information? How much
information each time? Some triggers might require listing a lot of the
namespace.
I had an offline discussion about this with Wei, [~HuafengWang] and [~Sammi].
To avoid listing a lot of the namespace, one way is to let SSM server have a
copy of namespace. This would allow to evolve SSM towards really powerful and
doing a lot of things without affecting NameNode very much. So the questions
now comes to:
* What kinds of file meta info are needed to make SSM function;
* How to fetch the namespace to prepare for the base;
* Given the namespace base, how to incrementally get updated time by time.
h4. What file meta info are needed
Roughly file meta info like filename, xattributes, permissions, atime/mtime,
length and etc. but not including block list info. So it's part of the
namespace, very compacted, not the whole, should be much smaller than the whole.
h4. How to fetch the namespace to prepare for the base
There're two ways for SSM to do this:
* Fetch the fsimage and load the needed info by itself.
* Get file status list from NameNode page by page at a NN affordable speed.
There doesn't urgent need that requires SSM to complete the work in short time.
It can take some days in a large cluster and the work can happen in idle time.
In either way, SSM can take quite much time to finish the work so a {{safe
mode}} would be needed to allow the period. During safe mode time, SSM won't
allow relevant operations.
h4. Given the namespace base, how to incrementally get updated time by time
I thought both [~andrew.wang] and [~eddyxu] (by offline) suggested we can use
the inotify support. It's a good chance to exercise the function.
Note this will consider NameNode federation. And the maintained files info will
be checkpointed to HDFS ensure it's reliability. When SSM is restarted on other
nodes, the data can be reloaded from HDFS other than from NameNodes again.
Any comment? Thanks!
> HDFS smart storage management
> -----------------------------
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Kai Zheng
> Assignee: Wei Zhou
> Attachments: HDFSSmartStorageManagement-General-20170315.pdf,
> HDFS-Smart-Storage-Management.pdf,
> HDFSSmartStorageManagement-Phase1-20170315.pdf,
> HDFS-Smart-Storage-Management-update.pdf, move.jpg
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
