[
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741890#comment-14741890
]
Bikas Saha edited comment on TEZ-2774 at 9/14/15 2:13 AM:
----------------------------------------------------------
Attaching v4 patch with some modification. Added and removed logs. Since its
all in the AM, just experimented using MockApp code on dev box for the AM log.
This does not account for scheduler logging. The simulation AM log was 1.5MB in
size for a 200x200 scatter-gather job. A hive query with 1600 task had an AM
size of 14.5MB which would be about 3.6MB scaled down to a 400 task job.
Biggest removal is counter logging in attempt finished event since it
duplicates counter logging in task finished event. That should save a lot of
space and make up for some of the logs that I added back. Counter logging
removal does not reduce simulation logs since they dont have counters.
Looking ahead we could stop logging counters completely in the history logs and
that should save a ton of space.
was (Author: bikassaha):
Attaching v4 patch with some modification. Added and removed logs. Since its
all in the AM, just experimented using MockApp code on dev box for the AM log.
This does not account for scheduler logging. Will try running a real job later
to check on that. The simulation AM log was 1.5MB in size for a 200x200
scatter-gather job.
Biggest removal is counter logging in attempt finished event since it
duplicates counter logging in task finished event. That should save a lot of
space and make up for some of the logs that I added back. Counter logging
removal does not reduce simulation logs since they dont have counters.
Looking ahead we could stop logging counters completely in the history logs and
that should save a ton of space.
> Reduce logging in the AM, and parts of the runtime
> --------------------------------------------------
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt,
> TEZ-2774.4.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
