[
https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997166#comment-15997166
]
Jonathan Eagles commented on TEZ-3709:
--------------------------------------
[~rajesh.balamohan], can you have a detailed look at the v2 patch. I added some
complexity to gain a more scalable solution. I switched to static methods to
make the changes more testable. I put a snippet of how I was measuring
performance. I have some jobs that have more than 10,000,000 shuffled segments
so this something that can really make a difference for those jobs.
{code}
public static class TestSegment extends TezMerger.Segment {
final long length;
public TestSegment(long length) {
super(null, null);
this.length = length;
}
@Override
public long getLength() {return length;}
}
@Test
public void testComputeBytesInMergesPerformance() throws Exception {
final int numSegments = 10000000;
long start = System.currentTimeMillis();
ArrayList<TezMerger.Segment> segments = new ArrayList<>(numSegments);
for (int i = 0; i < numSegments; i++) {
segments.add(i, new TestSegment(i));
}
long duration = System.currentTimeMillis() - start;
System.out.println("init:" + duration);
start = System.currentTimeMillis();
long bytes = TezMerger.MergeQueue.computeBytesInMerges(true, segments, 100,
0);
duration = System.currentTimeMillis() - start;
System.out.println("computeTime:" + duration);
System.out.println("bytes:" + bytes);
}
{code}
> TezMerger is slow for high number of segments
> ---------------------------------------------
>
> Key: TEZ-3709
> URL: https://issues.apache.org/jira/browse/TEZ-3709
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Labels: performance
> Attachments: TEZ-3709.1.patch, TEZ-3709.2.patch
>
>
> The below code is a bad performer at scale since it has to memcpy the whole
> list of segments for each item in the batch instead of of just once per batch.
> This is true for both computeBytesInMerges and getSegmentDescriptors.
> {code}
> for (int i = 0; i < batch; i++) {
> ArrayList#remove(0)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
