[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945026#comment-15945026
]
Kai Zheng commented on HDFS-7343:
---------------------------------
Thanks [~eddyxu] for your time reviewing this and reading the docs! Very good
questions. :)
bq. SSM needs to maintain stats of each file / block in NN ...
This needs to clarify some bit. SSM server needs to maintain states of each
file in SSM side. In NN side, it just maintains access count records happened
for some files recently. For example, for a time period if 100 files are read,
then in the NN AccessCounterMap there're only 100 entries; then when SSM
retrieves and polls to SSM, NN will remove the 100 entries if necessary (given
a threshold). It's SSM side to maintain and aggregate all the access counts for
all files.
We want SSM to maintain states and access count aggregates for all files, but
in a compact and concise approach, avoiding the memory consumption like O(n).
It's possible because SSM not like NN, the stored states info can be grouped
and shared based on common folders and access patterns.
bq. when SSM pulling these stats from NN, what kind of overhead we are
expecting?
An approach mentioned above is to let SSM fetch file meta lists by many
iterations across the inode tree, at a speed that NN can affords.
bq. I think that in the general design, we should work on define a good
interface time series store for metrics, instead of specifying RocksDB. RocksDB
might be a good implementation for now.
I thought a generic interface to abstract away concrete database impl is a good
idea. RocksDB may be not a good fit, you're right we're still thinking about a
better one. Initially we want to use a embedded SQL database to store the
collected file states for flexible query, but looks like no good one for Apache
space. The needed time series feature (like continuous aggregation/compaction,
window sliding count) would be easy to come up for our purpose given a SQL
database. Do you have any SQL or time series database to recommend? Thanks!
bq. it is not clear to me why both SSM and NN need to persist metrics in two
separated rocksdb? ...
We know that current HDFS NameNode now supports to collect many kinds of
metrics from DNs and we also want NN to record certain file access counts. Such
metrics can suffer from loss during NN switch or restart. To minimize the
memory consumption on this we suggest enhancing NN to allow it to spill metircs
to a local RocksDB. But I think we can consider this separately to avoid the
confusion here in SSM design. Sounds good?
bq. How stale or extensible the syntax will be? would the syntax be easy for
other applications to generate / parse / validate and etc?
We want to make the rule extensible, concise and flexible. The Json and YAML
are good as data formats, but could be very verbose to describe a rule, which
is why I haven't yet seen a DSL well written in them. You have a good
suggestion here, to support applications to generate/validate rules, to do
that, a Java library would be needed. Sounds good?
bq. How to update the accessCount ? how many samples of accessCount to be
maintained the accessCount during an interval? What if SSM failed to pull for
the the accessCount?
Regarding how to implement {{accessCount}} is worth to be addressed separately
and we'll follow here later. One thing to be noted now is, SSM or its
applications should be able to suffer from access count loss. In case some
files lose some read counts, the impact would be that SSM may not adjust
storage options for them in first time, which should be fine and no data hurts.
Anyhow we need good trade-off between access count accuracy and system
complexity/cost.
bq. Maybe we can define SSM rules as soft rules, while make HSM/EC and etc
rules as hard rules?
I might catch you so agree SSM rules are soft rules that may not have to
happen, but not very well to understand HSM/EC rules as hard rules. I thought
SSM aims to ease the deployment and admin operations for HSM/EC facilities.
> HDFS smart storage management
> -----------------------------
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Kai Zheng
> Assignee: Wei Zhou
> Attachments: HDFSSmartStorageManagement-General-20170315.pdf,
> HDFS-Smart-Storage-Management.pdf,
> HDFSSmartStorageManagement-Phase1-20170315.pdf,
> HDFS-Smart-Storage-Management-update.pdf, move.jpg
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
