[jira] [Commented] (HDDS-2097) Add TeraSort to acceptance test
[ https://issues.apache.org/jira/browse/HDDS-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926679#comment-16926679 ] Steve Loughran commented on HDDS-2097: -- OK, HADOOP-16058 is the patch you need; it is in the forthcoming 3.2.1 release > Add TeraSort to acceptance test > --- > > Key: HDDS-2097 > URL: https://issues.apache.org/jira/browse/HDDS-2097 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Xiaoyu Yao >Priority: Major > > We may begin with 1GB teragen/terasort/teravalidate. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2097) Add TeraSort to acceptance test
[ https://issues.apache.org/jira/browse/HDDS-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926678#comment-16926678 ] Steve Loughran commented on HDDS-2097: -- The terasort stuff is there, though I did do some changes to the Terasort classes themselves so they'd support support and destination paths which were not the default FS (so you can have an hdfs cluster terasorting to an object store). Nothing major and easy to backport if needed > Add TeraSort to acceptance test > --- > > Key: HDDS-2097 > URL: https://issues.apache.org/jira/browse/HDDS-2097 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Xiaoyu Yao >Priority: Major > > We may begin with 1GB teragen/terasort/teravalidate. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2097) Add TeraSort to acceptance test
[ https://issues.apache.org/jira/browse/HDDS-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926094#comment-16926094 ] Xiaoyu Yao commented on HDDS-2097: -- Thanks [~ste...@apache.org] for the heads up. I will play with it and see if the existing one for s3a fits requirement on ozone. Also, ozone as a submodule depends on Hadoop 3.2.0, is this available in Hadoop 3.2.0? > Add TeraSort to acceptance test > --- > > Key: HDDS-2097 > URL: https://issues.apache.org/jira/browse/HDDS-2097 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Xiaoyu Yao >Priority: Major > > We may begin with 1GB teragen/terasort/teravalidate. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2097) Add TeraSort to acceptance test
[ https://issues.apache.org/jira/browse/HDDS-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924876#comment-16924876 ] Steve Loughran commented on HDDS-2097: -- Take a look at org.apache.hadoop.fs.s3a.commit.terasort.AbstractCommitTerasortIT to see how we do it in s3a. Which of course you can already point at hdds * its driven off a -Dscale option so only runs on scale suites * and by breaking up each stage into their own ordered test case, you get better reporting of where problems surface. * HADOOP-16207 fixes some issues you should also look at * and you can configure the size of the data for smaller and larger tests * you need to make sure the test timeout is big enough for large test runs Why not have a play with those tests against HDDS to see how well they work as a starting point? > Add TeraSort to acceptance test > --- > > Key: HDDS-2097 > URL: https://issues.apache.org/jira/browse/HDDS-2097 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Xiaoyu Yao >Priority: Major > > We may begin with 1GB teragen/terasort/teravalidate. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
