[
https://issues.apache.org/jira/browse/HDDS-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140694#comment-17140694
]
Arpit Agarwal commented on HDDS-3721:
-------------------------------------
bq. Arpit Agarwal, can you give some insight why you would like to avoid
implementing this on OM side, perhaps why at the end was it implemented on the
server side for HDFS?
IMO getContentSummary on NameNode is the single biggest mistake in HDFS. :) In
general we should have no recursive calls on the server which can run for a
long time while holding locks and tying up RPC handler threads.
getContentSummary is the largest contributor to HDFS performance issues among
our customers since they naively assume it is a simple RPC call and issue it
very frequently. I would like to avoid giving Ozone users such a powerful tool
to shoot themselves with.
bq. The problem of approaching this from the client side, is that it leads to 4
calls per subdirectory (14k calls in this case)... 1 READ_BUCKET, then 1
GET_FILE_STATUS (to see if this is a file or a dir), then if it is a directory
1 READ_BUCKET again, and finally a LIST_STATUS, which then can not be
controlled or throttled by the server side much as these are coming from the
client side and from possibly multiple clients at some times.
Can we fix the client to avoid this multiplication of calls?
Also I have a more high level question - why do we want to optimize
getContentSummary? I want to get people to move away from using this call on
arbitrary directories. Perhaps we can provide an efficient implementation at
the bucket level by tracking the usage per bucket and updating it on key
commit/key delete.
> Implement getContentSummary to provide replicated size properly to dfs -du
> command
> ----------------------------------------------------------------------------------
>
> Key: HDDS-3721
> URL: https://issues.apache.org/jira/browse/HDDS-3721
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Reporter: Istvan Fajth
> Assignee: Istvan Fajth
> Priority: Major
> Labels: Triaged
>
> Currently when you run hdfs dfs -du command against a path on Ozone, it uses
> the default implementation from FileSystem class in the Hadoop project, and
> that does not care to calculate with replication factor by default. In
> DistributedFileSystem and in a couple of FileSystem implementation there is
> an override to calculate the full replicated size properly.
> Currently the output is something like this for a folder which has file with
> replication factor of 3:
> {code}
> hdfs dfs -du -s -h o3fs://perfbucket.volume.ozone1/terasort/datagen
> 931.3 G 931.3 G o3fs://perfbucket.volume.ozone1/terasort/datagen
> {code}
> The command in Ozone's case as well should report the replicated size az the
> second number so something around 2.7TB in this case.
> In order to do so, we should implement getContentSummary and calculate the
> replicated size in the response properly in order to get there.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
