[ https://issues.apache.org/jira/browse/HDFS-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803428#comment-16803428 ]
Erik Krogen commented on HDFS-12345: ------------------------------------ Hey [~daryn], thanks for raising some valid concerns. Regarding the use of private APIs, they are pretty limited. On the YARN side, there's a few now within the ApplicationMaster simply because it is copied from the example of DistributedShell which also makes use of such private APIs; I'm happy to fix those before pushing. On the HDFS side, I believe the only use of private APIs is within the {{SimulatedDataNodes}} class, which does some manipulation to inject blocks into a {{MiniDFSCluster}}. Given that {{DataNodeCluster}} does something very similar, I would expect any changes in this area to be handled very similarly and not be much extra maintenance burden. If there is a desire, I can probably combine {{SimulatedDataNodes}} and {{DataNodeCluster}} so that any maintenance would only be in one place; IIRC, the main reason I did not do so already is that besides the actual block injection, most code within {{DataNodeCluster}} is devoted to deciding how to generate all of the blocks, whereas {{SimulatedDataNodes}} reads its blocks from an already-generated listing. But I can look into combining them. Regarding maintenance burden, I guess most of the tools within {{hadoop-tools}} are "non-essential" maintenance burdens. I think [~smeng]'s idea of disabling compilation by default is interesting, but I worry that this would increase the chance that it falls out of sync with the main project and becomes even harder to maintain. I would definitely be open to feedback in this area. > Scale testing HDFS NameNode with real metadata and workloads (Dynamometer) > -------------------------------------------------------------------------- > > Key: HDFS-12345 > URL: https://issues.apache.org/jira/browse/HDFS-12345 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode, test > Reporter: Zhe Zhang > Assignee: Siyao Meng > Priority: Major > Attachments: HDFS-12345.000.patch, HDFS-12345.001.patch, > HDFS-12345.002.patch, HDFS-12345.003.patch, HDFS-12345.004.patch, > HDFS-12345.005.patch > > > Dynamometer has now been open sourced on our [GitHub > page|https://github.com/linkedin/dynamometer]. Read more at our [recent blog > post|https://engineering.linkedin.com/blog/2018/02/dynamometer--scale-testing-hdfs-on-minimal-hardware-with-maximum]. > To encourage getting the tool into the open for others to use as quickly as > possible, we went through our standard open sourcing process of releasing on > GitHub. However we are interested in the possibility of donating this to > Apache as part of Hadoop itself and would appreciate feedback on whether or > not this is something that would be supported by the community. > Also of note, previous [discussions on the dev mail > lists|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201707.mbox/%3c98fceffa-faff-4cf1-a14d-4faab6567...@gmail.com%3e] -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org