[
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-12502:
-----------------------------
Attachment: HDFS-12502.01.patch
v00 patch actually doesn't work, because {{logAuditEvent}} for {{getListing}}
doesn't pass the FileStatus of the directory as a parameter.
I see two options here:
# Extend the format of audit logs to add an optional field representing "number
of files in recursive operations". This applies to listing and deletes. Adding
the info to audit logs would help more accurately identify abusive users
issuing large listings. E.g. we will be able to tell which large dirs were
listed. But considering the field is only applicable to a small number of
operations, it's a little awkward to add to the schema.
# Add a method in {{TopMetrics}} to report {{filesInGetListing}} and directly
call it from {{getListing}}. The attached v01 patch uses this approach. I
verified with {{MiniDFSClusterManager}} that it does report the correct
metrics. If we decide to use this approach I'll add a unit test.
Pinging a few folks based on previous discussions on RPC scalability:
[~andrew.wang] [~drankye] [~ywskycn] [~shv]. Any thoughts?
> nntop should support a category based on FilesInGetListingOps
> -------------------------------------------------------------
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: metrics
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode
> slowness. The aggregate cost of listing ops is proportional to the
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore
> it'd be very useful for nntop to support this category.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
