[
https://issues.apache.org/jira/browse/MAPREDUCE-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated MAPREDUCE-6054:
-----------------------------------
Attachment: mapreduce-6054.txt
*Parameterization:*
- Changed the parameterization to not test all {K,V} pairs, but instead test
all {K,LongWritable} and all {LongWritable,V}.
This reduced the number of parameterizations of the test by a factor of 12 or
so, and now the test runs in only a couple of minutes instead of timing out
after 15+.
*Test input generation:*
- Improved BytesUtil.toStringBinary to avoid the String.format call. The new
method is still pretty simple, but much faster
- Improved BytesFactory's API so it can reuse Writable objects. This reduces a
bunch of pressure on the GC during the test input generation phase.
The above two changes sped up the test file generation by a large margin. For
example:
before:
{code}
2014-08-26 21:18:06,326 INFO [main] kvtest.KVJob (KVJob.java:<init>(92)) -
Created test file
./kvtest/input/org.apache.hadoop.io.LongWritable/org.apache.hadoop.io.Text in
4573ms
{code}
after:
{code}
2014-08-26 21:25:44,855 INFO [main] kvtest.KVJob (KVJob.java:<init>(92)) -
Created test file
./kvtest/input/org.apache.hadoop.io.LongWritable/org.apache.hadoop.io.Text in
608ms
{code}
With these changes in place, all the tests (including native ones) ran in 3m40s
on my laptop, which I think is pretty reasonable for a module of this size.
> native-task: speed up test runs
> -------------------------------
>
> Key: MAPREDUCE-6054
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6054
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
> Reporter: Todd Lipcon
> Attachments: mapreduce-6054.txt
>
>
> Currently the KVTest compatibility test takes so long on my machine that it
> regularly times out maven. We should speed it up.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
