[
https://issues.apache.org/jira/browse/HADOOP-14407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022336#comment-16022336
]
Yongjun Zhang commented on HADOOP-14407:
----------------------------------------
Welcome [~omkarksa].
My bad, did not catch an issue in your branch-2 patch in time.
{code}
------------------------------------------------------
Running org.apache.hadoop.tools.TestDistCpOptions
Tests run: 24, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.266 sec <<<
FAILURE! - in org.apache.hadoop.tools.TestDistCpOptions
testToString(org.apache.hadoop.tools.TestDistCpOptions) Time elapsed: 0.018
sec <<< FAILURE!
org.junit.ComparisonFailure: expected:<..., filtersFile='null'[]}> but
was:<..., filtersFile='null'[, blocksPerChunk=0, copyBufferSize=8192]}>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.tools.TestDistCpOptions.testToString(TestDistCpOptions.java:317)
{code}
TestDistCpOptions.java is somehow missing from the branch-2 patch, would you
please create a new jira for branch-2 only and submit the patch asap?
Thanks.
> DistCp - Introduce a configurable copy buffer size
> --------------------------------------------------
>
> Key: HADOOP-14407
> URL: https://issues.apache.org/jira/browse/HADOOP-14407
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
> Affects Versions: 2.9.0
> Reporter: Omkar Aradhya K S
> Assignee: Omkar Aradhya K S
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-14407.001.patch, HADOOP-14407.002.patch,
> HADOOP-14407.002.patch, HADOOP-14407.003.patch,
> HADOOP-14407.004.branch2.patch, HADOOP-14407.004.patch,
> HADOOP-14407.004.patch, TotalTime-vs-CopyBufferSize.jpg
>
>
> Currently, the RetriableFileCopyCommand has a fixed copy buffer size of just
> 8KB. We have noticed in our performance tests that with bigger buffer sizes
> we saw upto ~3x performance boost. Hence, making the copy buffer size a
> configurable setting via the new parameter <copybuffersize>.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
