[
https://issues.apache.org/jira/browse/TEZ-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-2473:
----------------------------------
Attachment: TEZ-2473.1.patch
{noformat}
With out patch:
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000191_0,
NUM_MEM_TO_DISK_MERGES count: 2: CPU=225520
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000192_0,
NUM_MEM_TO_DISK_MERGES count: 2: CPU=198550
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000193_0,
NUM_MEM_TO_DISK_MERGES count: 2: CPU=207220
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0334_1_01_000194_0,
NUM_MEM_TO_DISK_MERGES count: 2: CPU=216870
With patch:
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000191_0,
NUM_MEM_TO_DISK_MERGES count: 2: CPU=199620
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000192_0,
NUM_MEM_TO_DISK_MERGES count: 2: CPU=203500
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000193_0,
NUM_MEM_TO_DISK_MERGES count: 2: CPU=203730
VertexName: Reducer 2, TaskAttemptId: attempt_1431919257083_0335_1_01_000194_0,
NUM_MEM_TO_DISK_MERGES count: 2: CPU=208410
{noformat}
> Consider using RawLocalFileSystem in MapOutput.createDiskMapOutput
> ------------------------------------------------------------------
>
> Key: TEZ-2473
> URL: https://issues.apache.org/jira/browse/TEZ-2473
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Attachments: TEZ-2473.1.patch
>
>
> Currently it makes use of LocalFileSystem which would go through
> checksumming. This can save some CPU cycles in tasks involving disk merges.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
