[
https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741890#comment-14741890
]
Bikas Saha edited comment on TEZ-2774 at 9/12/15 4:01 AM:
----------------------------------------------------------
Attaching v4 patch with some modification. Added and removed logs. Since its
all in the AM, just experimented using MockApp code on dev box for the AM log.
This does not account for scheduler logging. Will try running a real job later
to check on that. The simulation AM log was 1.5MB in size for a 200x200
scatter-gather job.
Biggest removal is counter logging in attempt finished event since it
duplicates counter logging in task finished event. That should save a lot of
space and make up for some of the logs that I added back. Counter logging
removal does not reduce simulation logs since they dont have counters.
Looking ahead we could stop logging counters completely in the history logs and
that should save a ton of space.
was (Author: bikassaha):
Attaching v4 patch with some modification. Added and removed logs. Since its
all in the AM, just experimented using MockApp code on dev box for the AM log.
This does not account for scheduler logging. Will try running a real job later
to check on that. The simulation AM log was 1.5MB in size for a 200x200
scatter-gather job.
Biggest removal is counter logging in attempt finished event since it
duplicates counter logging in task finished event. That should save a lot of
space and make up for some of the logs that I added back. Counter logging
removal does not reduce simulation logs since they have not counters.
Looking ahead we could stop logging counters completely in the history logs and
that should save a ton of space.
> Reduce logging in the AM, and parts of the runtime
> --------------------------------------------------
>
> Key: TEZ-2774
> URL: https://issues.apache.org/jira/browse/TEZ-2774
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-2774.1.txt, TEZ-2774.2.txt, TEZ-2774.3.txt,
> TEZ-2774.4.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
