[
https://issues.apache.org/jira/browse/HDFS-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809299#comment-16809299
]
Erik Krogen commented on HDFS-12345:
------------------------------------
I'm attaching a v006 patch which clears up the remaining javac warnings, fixes
the shelldocs and javadocs issues, fixes most of the remaining checkstyle
warnings, fixes most of the shellcheck issues, fixes whitespace issues, removes
a few usages of {{commons.logging}} in favor of {{slf4j}}, removes use of
private YARN APIs within {{ApplicationMaster}}, and puts a timeout on the main
integration test.
I considered trying to fold {{SimulatedDataNodes}} into {{DataNodeCluster}},
but I think I would prefer to leave it as a follow-on item. For now, this patch
is completely isolated to new code within the
{{hadoop-tools/hadoop-dynamometer}} submodule (besides a few changes to build
and assembly files), and I would prefer to keep it that way. Merging SDN into
DNC should remove the last of the burden of private API usage.
Two things I noticed while going through this pass that I would like to discuss:
* This currently includes a class to allow Dynamometer to be run via Azkaban.
This is very useful for us internally, but may not make much sense to have
within Hadoop. Unless I hear another opinion, I will probably remove this in a
subsequent patch.
* Currently, {{TestDynamometerInfra}} requires {{bash}} and {{tar}} to be
present on the PATH, so it will fail on Windows. I'd like to remove the
reliance on {{tar}} completely, but I think it will be a bit more tricky, so I
think it would be reasonable to simply disable building Dynamometer on Windows
for now. I'm curious if anyone knows how to achieve this -- [~smeng] ? It seems
it should be possible through some POM magic.
> Scale testing HDFS NameNode with real metadata and workloads (Dynamometer)
> --------------------------------------------------------------------------
>
> Key: HDFS-12345
> URL: https://issues.apache.org/jira/browse/HDFS-12345
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode, test
> Reporter: Zhe Zhang
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HDFS-12345.000.patch, HDFS-12345.001.patch,
> HDFS-12345.002.patch, HDFS-12345.003.patch, HDFS-12345.004.patch,
> HDFS-12345.005.patch, HDFS-12345.006.patch
>
>
> Dynamometer has now been open sourced on our [GitHub
> page|https://github.com/linkedin/dynamometer]. Read more at our [recent blog
> post|https://engineering.linkedin.com/blog/2018/02/dynamometer--scale-testing-hdfs-on-minimal-hardware-with-maximum].
> To encourage getting the tool into the open for others to use as quickly as
> possible, we went through our standard open sourcing process of releasing on
> GitHub. However we are interested in the possibility of donating this to
> Apache as part of Hadoop itself and would appreciate feedback on whether or
> not this is something that would be supported by the community.
> Also of note, previous [discussions on the dev mail
> lists|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201707.mbox/%3c98fceffa-faff-4cf1-a14d-4faab6567...@gmail.com%3e]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
