[
https://issues.apache.org/jira/browse/HADOOP-16058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748028#comment-16748028
]
Steve Loughran commented on HADOOP-16058:
-----------------------------------------
Patch 002
Pulls up the code to set up MR clusters for committer tests into a new
intermediate base class, one which does not contain the code to actually set up
those base clusters. instead the cluster setup/teradown is done in the
@BeforeClass/@AfterClass operations of the subclasses, so guaranteeing
isolation and a lifecycle which matches those child classes.
Having done this, it hasn't made the terasort conflict go away; I've concluded
now that that's due to some code in Terasort which uses LocalFS to save a
partition list. Rather than do dramatic things to Terasort (e.g. add the
ability to declare new local paths), I've just serialized the Terasorrt tests
-after shrinking down their test size
I haven't reverted the design which pushes cluster setup/teardown into the
child classes, even though I'm not sure it is needed, just because it does make
clear the lifecycle of class-level data types.
+address checkstyle warnings from the previous patch, where possible.
+address checkstyle warnings from the previous patch, where possible.
Testing: S3A ireland, S3guard, ddb, auth, scale
The scale test runs now take 17 minutes, which is long enough to become
inconvenient, especially because that's with 12 VMs: the laptop isn't usable
for anything else.
{code:java}
bin/hadoop fs -cat
s3a://hwdev-steve-ireland-new/terasort-ITestTerasortMagicCommitter/results.csv
"Operation" "Duration"
"Generate" "0:28.596s"
"Terasort" "0:32.456s"
"Validate" "0:30.000s"
"Completed" "1:33.824s"
{code}
{code:java}
fs -cat
s3a://hwdev-steve-ireland-new/terasort-ITestTerasortDirectoryCommitter/results.csv
"Operation" "Duration"
"Generate" "0:17.602s"
"Terasort" "0:25.151s"
"Validate" "0:26.132s"
"Completed" "1:11.496s"
{code}
One test failure: HADOOP-16057
{code}
ERROR]
testDestroyNoArgs(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal) Time
elapsed: 1.167 s <<< ERROR!
java.lang.IndexOutOfBoundsException: toIndex = 1
at java.util.ArrayList.subListRangeCheck(ArrayList.java:1004)
at java.util.ArrayList.subList(ArrayList.java:996)
at org.apache.hadoop.fs.shell.CommandFormat.parse(CommandFormat.java:89)
{code}
> S3A tests to include Terasort
> -----------------------------
>
> Key: HADOOP-16058
> URL: https://issues.apache.org/jira/browse/HADOOP-16058
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-16058-001.patch, HADOOP-16058-002.patch
>
>
> Add S3A tests to run terasort for the magic and directory committers.
> MAPREDUCE-7091 is a requirement for this
> Bonus feature: print the results to see which committers are faster in the
> specific test setup. As that's a function of latency to the store, bandwidth
> and size of jobs, it's not at all meaningful, just interesting.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
