[ https://issues.apache.org/jira/browse/HADOOP-16058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748028#comment-16748028 ]
Steve Loughran commented on HADOOP-16058: ----------------------------------------- Patch 002 Pulls up the code to set up MR clusters for committer tests into a new intermediate base class, one which does not contain the code to actually set up those base clusters. instead the cluster setup/teradown is done in the @BeforeClass/@AfterClass operations of the subclasses, so guaranteeing isolation and a lifecycle which matches those child classes. Having done this, it hasn't made the terasort conflict go away; I've concluded now that that's due to some code in Terasort which uses LocalFS to save a partition list. Rather than do dramatic things to Terasort (e.g. add the ability to declare new local paths), I've just serialized the Terasorrt tests -after shrinking down their test size I haven't reverted the design which pushes cluster setup/teardown into the child classes, even though I'm not sure it is needed, just because it does make clear the lifecycle of class-level data types. +address checkstyle warnings from the previous patch, where possible. +address checkstyle warnings from the previous patch, where possible. Testing: S3A ireland, S3guard, ddb, auth, scale The scale test runs now take 17 minutes, which is long enough to become inconvenient, especially because that's with 12 VMs: the laptop isn't usable for anything else. {code:java} bin/hadoop fs -cat s3a://hwdev-steve-ireland-new/terasort-ITestTerasortMagicCommitter/results.csv "Operation" "Duration" "Generate" "0:28.596s" "Terasort" "0:32.456s" "Validate" "0:30.000s" "Completed" "1:33.824s" {code} {code:java} fs -cat s3a://hwdev-steve-ireland-new/terasort-ITestTerasortDirectoryCommitter/results.csv "Operation" "Duration" "Generate" "0:17.602s" "Terasort" "0:25.151s" "Validate" "0:26.132s" "Completed" "1:11.496s" {code} One test failure: HADOOP-16057 {code} ERROR] testDestroyNoArgs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal) Time elapsed: 1.167 s <<< ERROR! java.lang.IndexOutOfBoundsException: toIndex = 1 at java.util.ArrayList.subListRangeCheck(ArrayList.java:1004) at java.util.ArrayList.subList(ArrayList.java:996) at org.apache.hadoop.fs.shell.CommandFormat.parse(CommandFormat.java:89) {code} > S3A tests to include Terasort > ----------------------------- > > Key: HADOOP-16058 > URL: https://issues.apache.org/jira/browse/HADOOP-16058 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test > Affects Versions: 3.3.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Priority: Major > Attachments: HADOOP-16058-001.patch, HADOOP-16058-002.patch > > > Add S3A tests to run terasort for the magic and directory committers. > MAPREDUCE-7091 is a requirement for this > Bonus feature: print the results to see which committers are faster in the > specific test setup. As that's a function of latency to the store, bandwidth > and size of jobs, it's not at all meaningful, just interesting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org