[
https://issues.apache.org/jira/browse/HDFS-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365858#comment-16365858
]
Erik Krogen commented on HDFS-12345:
------------------------------------
Thanks for the feedback [~anu] and [~elgoiri]!
> The most important reason is that we will be able to make sure that this tool
> stays current with the code base.
Agreed, this will be a big win.
> It might lead to adding some simple "sanity check" type of unit tests though,
> so that we can detect it we have accidently broken this tool
I will be the first to admit that the testing present in Dynamometer currently
is lacking; basically just one big integration test. Definitely agreed that it
could use more.
> I'm personally interested on pushing HDFS as a YARN job into Hadoop itself.
> ....
> To summarize, my opinion is that we should go over on how to split the work
> but I think this most of Dynanometer (if not all) should go into Hadoop.
Probably if/when Dynamometer is added to Hadoop itself, it should be as two
separate tools, "infrastructure" and "workload" (to use Dynamometer
terminology), since each are useful on their own. This reflects the current
subproject structure of Dynamometer as well.
[~anu], I am curious if you can comment at all on one thing - I remember seeing
a HDP blog post about using some of the containerization support in newer
versions of YARN to run HDFS on YARN. Are you aware of progress in that
direction / overlap with Dynamometer in that sense?
> Scale testing HDFS NameNode with real metadata and workloads
> ------------------------------------------------------------
>
> Key: HDFS-12345
> URL: https://issues.apache.org/jira/browse/HDFS-12345
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode, test
> Reporter: Zhe Zhang
> Assignee: Erik Krogen
> Priority: Major
>
> Dynamometer has now been open sourced on our [GitHub
> page|https://github.com/linkedin/dynamometer]. Read more at our [recent blog
> post|https://engineering.linkedin.com/blog/2018/02/dynamometer--scale-testing-hdfs-on-minimal-hardware-with-maximum].
> To encourage getting the tool into the open for others to use as quickly as
> possible, we went through our standard open sourcing process of releasing on
> GitHub. However we are interested in the possibility of donating this to
> Apache as part of Hadoop itself and would appreciate feedback on whether or
> not this is something that would be supported by the community.
> Also of note, previous [discussions on the dev mail
> lists|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201707.mbox/%[email protected]%3e]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
