[
https://issues.apache.org/jira/browse/MAPREDUCE-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594819#comment-14594819
]
Ray Chiang commented on MAPREDUCE-6376:
---------------------------------------
Just a quick follow up. I wrote up a quick program to convert old .jhist files
into the new binary form and loaded them into a new VM. The speedups are
decent at a high # of tasks, but closer to 2x (versus the 3x at <50k tasks).
751k Mappers/10 Reducers (json)
- Read Time: 163.8 seconds
- Event Time: 16.6 seconds
751k Mappers/10 Reducers (binary)
- Read Time: 86.7 seconds
- Event Time: 17.4 seconds
404k Mappers/4 Reducers (json)
- Read Time: 93 seconds
- Event Time: 8.9 seconds
404k Mappers/4 Reducers (binary)
- Read Time: 49.1 seconds
- Event Time: 9.2 seconds
I'm probably not going to get to a random event generator this week, but this
gives us at least two more data points.
> Fix long load times of .jhist file in JobHistoryServer
> ------------------------------------------------------
>
> Key: MAPREDUCE-6376
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6376
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: jobhistoryserver
> Affects Versions: 2.7.0
> Reporter: Ray Chiang
> Assignee: Ray Chiang
> Labels: supportability
> Attachments: MAPREDUCE-6376.001.patch, MAPREDUCE-6376.002.patch,
> MAPREDUCE-6376.003.patch
>
>
> When you click on a Job link in the JHS Web UI, it loads the .jhist file.
> For jobs which have a large number of tasks, the load time can break UI
> responsiveness.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
