[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287001#comment-14287001
]
Xiaoyu Yao commented on HDFS-7584:
----------------------------------
Thanks [~zhz] for the feedback.
bq. 1. This is an interesting scenario and is worth more discussion. It is a
conservative and safe policy to deduct from both SSD and DISK quotas. However
it doesn't fully comply with the principle of quota based on intended usage,
which might make it appear counter-intuitive to users (e.g. why am I double
charged?). As an extreme example, what if the user doesn't have any DISK quota?
Agree. "the cumulative disk space quota" in the spec means traditional space
quota not quota of DISK type. I will update it to avoid confusion.
We are actually calculating quota based on intended usage by not deducting DISK
quotas due to policy fallback.
For example, we have a directory \ssd1 with ONE_SSD policy enabled. The
directory currently has 2 blocks of SSD Quota and 3 blocks of DISK Quota
remaining.
If we want to create a file of 1 block with a replication factor of 3 under
\ssd1 but the actual available SSD storage is 0, the creation fallback to
DISK.
After that, the remaining SSD and DISK Quota should be 1 block and 1 block
instead of 2 block and 0 block, respectively.
If the fallback still can't be satisfied due to DISK quota unavailable, the
user will get QuotaByStorageTypeExceeded exception.
bq. 2. How about calculating quota truly based on intended usage? The charged
quota might be different than the usage, but so is the case with existing quota
logic. What are other disadvantages?
Quota calculation based on intended usage saves the replication monitor from
updating traditional space/namespace quota usage for under/over replicated
blocks. For quota by storage type, it can similarly save the Mover from
updating quota when the blocks are moved across storage tiers to meet the
policy requirement.
bq. 3. If we do want to charge by actual usage (5.2), maybe we should allow
different "quota currencies" to be exchanged? Something like 1GB of SSD = 2GB
of DISK = 4GB of ARCHIVAL. Or at least allow a user with only 1GB SSD quota to
use 1GB DISK space.
As mentioned above, we prefer charging by intended usage for its simplicity and
consistency. Correlation quota of different storage types looks interesting, it
may requires additional tuning to get appropriate currency rates between
storage types for different user scenarios.
> Enable Quota Support for Storage Types (SSD)
> ---------------------------------------------
>
> Key: HDFS-7584
> URL: https://issues.apache.org/jira/browse/HDFS-7584
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Reporter: Xiaoyu Yao
> Assignee: Xiaoyu Yao
> Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
> HDFS-7584.0.patch
>
>
> Phase II of the Heterogeneous storage features have completed by HDFS-6584.
> This JIRA is opened to enable Quota support of different storage types in
> terms of storage space usage. This is more important for certain storage
> types such as SSD as it is precious and more performant.
> As described in the design doc of HDFS-5682, we plan to add new
> quotaByStorageType command and new name node RPC protocol for it. The quota
> by storage type feature is applied to HDFS directory level similar to
> traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
