[
https://issues.apache.org/jira/browse/HDDS-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17141477#comment-17141477
]
Istvan Fajth commented on HDDS-3721:
------------------------------------
As I see the call, the use cases where it is important to have the value
correctly are:
- define and validate quota
- discover how much data a cluster have in certain places and how much it grows
over time
- operators to review and report data usage of different tenants in order to
help negotiate how to optimize space between them
For quota we need exact values at any point in time, hiwever for that we need
it on the server side.
For other use cases we might be fine semi accurate data, but still I guess we
need some kind of summary anyway but I would definitely avoid the way how it
works in HDFS.
I think this might be best if we hold the data with the keys in rocksdb, we
might not need immediate updates but probably after any writes we might update
the space occupied in a bg thread outside of the locks, and provide a
re-calculation or validation after restart or on demand. I am unsure how much
extra load it puts to the db, but with the prefix table we can aggregate the
data for all the directories on write that way making it a single rpc call...
Is it sounds viable? This means an extra long value so 8 bytes for all keys in
the db, and a calculation overhead outside locks for all directories, if we do
not require strong consistency between the real and stored data.
> Implement getContentSummary to provide replicated size properly to dfs -du
> command
> ----------------------------------------------------------------------------------
>
> Key: HDDS-3721
> URL: https://issues.apache.org/jira/browse/HDDS-3721
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Reporter: Istvan Fajth
> Assignee: Istvan Fajth
> Priority: Major
> Labels: Triaged
>
> Currently when you run hdfs dfs -du command against a path on Ozone, it uses
> the default implementation from FileSystem class in the Hadoop project, and
> that does not care to calculate with replication factor by default. In
> DistributedFileSystem and in a couple of FileSystem implementation there is
> an override to calculate the full replicated size properly.
> Currently the output is something like this for a folder which has file with
> replication factor of 3:
> {code}
> hdfs dfs -du -s -h o3fs://perfbucket.volume.ozone1/terasort/datagen
> 931.3 G 931.3 G o3fs://perfbucket.volume.ozone1/terasort/datagen
> {code}
> The command in Ozone's case as well should report the replicated size az the
> second number so something around 2.7TB in this case.
> In order to do so, we should implement getContentSummary and calculate the
> replicated size in the response properly in order to get there.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
