[
https://issues.apache.org/jira/browse/HDFS-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580755#comment-16580755
]
Todd Lipcon commented on HDFS-13826:
------------------------------------
Thanks for pointing me at Dynamometer. That's quite interesting as it's
high-fidelity and simulates block reports, NN memory usage more accurately,
etc. I'll take a look at whether we can use that for our use case. If not, then
yea, the scope you described is just about it -- just a check if this setting
is configured and calling out to some other function to generate fake results
if so. Only a couple line hook into existing code and maybe 100 new lines
elsewhere to do the generation.
> Add a hidden configuration for NameNode to generate fake block locations
> ------------------------------------------------------------------------
>
> Key: HDFS-13826
> URL: https://issues.apache.org/jira/browse/HDFS-13826
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
>
> In doing testing and benchmarking of the NameNode and dependent systems, it's
> often useful to be able to use an fsimage provided by some production system
> in a controlled environment without actually having access to any of the
> data. For example, while doing some recent work on Apache Impala I was trying
> to optimize the transmission and storage of block locations and tokens and
> measure the results based on metadata from a production user. In order to
> achieve this, it would be useful for the NN to expose a developer-only
> (undocumented) configuration to generate fake block locations and return them
> to callers. The "fake" locations should be randomly distributed across a
> fixed set of fake datanodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
