[ 
https://issues.apache.org/jira/browse/YARN-221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697314#comment-14697314
 ] 

Ming Ma commented on YARN-221:
------------------------------

The unit test failures aren't related. The tests pass on the local machine.

Another thing Xuan and I discussed is how other frameworks on YARN such as MR, 
Tez can use this feature; for example if they need to make config and/or code 
change to allow framework applications specify the policy at per application 
basis. There are several approaches.

* Have MR define its own configurations to config these policies. Make code 
change at YarnRunner to retrieve these configurations and set the values at 
ASC. That means Tez needs to do the same thing.
* Define some common YARN configurations such as 
yarn.logaggregation.policy.class. YarnRunner still needs to retrieve these 
configurations and set the values at ASC. But at least MR and Tez can share the 
same configuration names.
* Define some common YARN configurations such as 
yarn.logaggregation.policy.class. YarnClientImpl take care of fixing up ASC 
based on the configurations. In that way, no code change is required at the MR 
or Tez layer.

Eventually, we prefer to go with the first approach, which is used by other 
existing MR properties. If we want to define some common YARN properties used 
by different YARN applications, we can have a separate jira for it.

> NM should provide a way for AM to tell it not to aggregate logs.
> ----------------------------------------------------------------
>
>                 Key: YARN-221
>                 URL: https://issues.apache.org/jira/browse/YARN-221
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: log-aggregation, nodemanager
>            Reporter: Robert Joseph Evans
>            Assignee: Ming Ma
>         Attachments: YARN-221-6.patch, YARN-221-7.patch, YARN-221-8.patch, 
> YARN-221-9.patch, YARN-221-trunk-v1.patch, YARN-221-trunk-v2.patch, 
> YARN-221-trunk-v3.patch, YARN-221-trunk-v4.patch, YARN-221-trunk-v5.patch
>
>
> The NodeManager should provide a way for an AM to tell it that either the 
> logs should not be aggregated, that they should be aggregated with a high 
> priority, or that they should be aggregated but with a lower priority.  The 
> AM should be able to do this in the ContainerLaunch context to provide a 
> default value, but should also be able to update the value when the container 
> is released.
> This would allow for the NM to not aggregate logs in some cases, and avoid 
> connection to the NN at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to