Varun Saxena created YARN-4709:
----------------------------------
Summary: Exception while fetching all log files using yarn logs
-am command and unusable JSON produced for containerLogFiles
Key: YARN-4709
URL: https://issues.apache.org/jira/browse/YARN-4709
Project: Hadoop YARN
Issue Type: Bug
Reporter: Varun Saxena
Assignee: Varun Saxena
Following exception is thrown when we run below command.
{panel}
root@varun-Inspiron-5558:/opt1/hadoop3/bin# ./yarn logs -applicationId
application_1455999168135_0002 -am ALL -logFiles ALL
Container: container_e31_1455999168135_0002_01_000001
=======================================================
{color:red}LogType:syslogstderrstdout
Log Upload Time:Sun Feb 21 01:44:55 +0530 2016
Log Contents:
java.lang.Exception: Cannot find this log on the local disk.
End of LogType:syslogstderrstdout{color}
LogType:syslog
Log Upload Time:Sun Feb 21 01:44:55 +0530 2016
Log Contents:
2016-02-21 01:44:49,565 INFO \[main\]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
application appattempt_1455999168135_0002_000001
2016-02-21 01:44:49,914 INFO \[main\]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
/************************************************************
{panel}
This is because we annotate containerLogFiles list with XmlElementWrapper which
generates XML output as under. And when we read this XML at client side,
reading the value associated with containerLogFiles also leads to one value
being syslogstderrstdout because both parent and child tags are same. This
leads to the exception.
{noformat}
<containerLogFiles>
<containerLogFiles>syslog</containerLogFiles>
<containerLogFiles>stderr</containerLogFiles>
<containerLogFiles>stdout</containerLogFiles>
</containerLogFiles>
{noformat}
Moreover, as we use XMLElementWrapper, the JSON generated is as under. This
JSON cannot be properly parsed by JSON parser(as a list). This is because child
containerLogsFiles entries are treated as a key-value pair(map) and hence only
last entry i.e. stdout is picked up. This was found while working on YARN-4517.
This makes output unusable.
This will be an issue for 2 REST endpoints i.e. {{/ws/v1/node/containers}} and
{{/ws/v1/node/containers/\{\{containerId\}\}}}
{noformat}
"containerLogFiles":[
{
"containerLogFiles":"syslog",
"containerLogFiles":"stderr",
"containerLogFiles":"stdout"
}
]
{noformat}
Ideally the JSON output should be as under.
{noformat}
"containerLogFiles":["syslog","stderr","stdout"]
{noformat}
We can indicate in the JAXB context to ignore the outer wrapper while
marshalling to JSON. But this can only be done at class level. If we do so for
ContainerInfo, it would break backward compatibility.
Hence, to fix it we can remove XmlElementWrapper annotation for
containerLogFiles list.
Another solution would be to wrap the list inside another class.
But going with former as of now as we do not specify XmlElementWrapper for
lists at most of the places in our code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
