[
https://issues.apache.org/jira/browse/YARN-11764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17923241#comment-17923241
]
ASF GitHub Bot commented on YARN-11764:
---------------------------------------
slfan1989 commented on PR #7345:
URL: https://github.com/apache/hadoop/pull/7345#issuecomment-2630507008
I have fixed the YARN-related unit test errors and tested them locally.
Next, I will provide a detailed explanation of how we resolved these unit test
issues.
> hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
These unit test errors were discovered during the integration testing of the
ResourceManager module.
The error message is as follows:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7345/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
```
[ERROR] TestSystemMetricsPublisher.testPublishApplicationMetrics Time
elapsed: 1.279 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at
org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher.testPublishApplicationMetrics(TestSystemMetricsPublisher.java:230)
```
The code causing the error is as follows:
```
Assert.assertTrue(verifyAppTags(app.getApplicationTags(),
entity.getOtherInfo()));
```
We found that the contents of `app.getApplicationTags()` and
`entity.getOtherInfo()` are inconsistent, with `entity.getOtherInfo()`
containing duplicate entries.
`app.getApplicationTags()` is expected to contain `2` elements, but
`entity.getOtherInfo()` contains `4` elements.
https://github.com/apache/hadoop/blob/44a5cba78ac7d29a72d3452fbd807f31ae90c325/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java#L555-L558
The cause of this issue is that the data in `TimelineEntity.java` is being
serialized twice, as there are two properties in the class representing the
same data. As shown in the code below, this results in two identical entries in
the serialized JSON, one named `otherinfo` and the other named `otherInfoJAXB`.
https://github.com/apache/hadoop/blob/44a5cba78ac7d29a72d3452fbd807f31ae90c325/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelineEntity.java#L310-L315
The serialized JSON is as follows:
```
{
"entities": [
{
"entityType": "YARN_APPLICATION",
"entityId": "application_0_0001",
"startTime": null,
"events": [
{
"timestamp": 2147483649,
"eventType": "YARN_APPLICATION_CREATED",
"eventInfo": {
}
}
],
"relatedEntities": {
},
"primaryFilters": {
},
"otherInfo": {
"YARN_APPLICATION_CALLER_CONTEXT": "context",
"YARN_APPLICATION_NAME": "test app",
"YARN_APPLICATION_USER": "test user",
"YARN_APPLICATION_UNMANAGED_APPLICATION": false,
"YARN_APP_NODE_LABEL_EXPRESSION": "high-cpu",
"YARN_APPLICATION_SUBMITTED_TIME": 2147483648,
"YARN_AM_CONTAINER_LAUNCH_COMMAND": [
"java -Xmx1024m"
],
"YARN_AM_NODE_LABEL_EXPRESSION": "high-mem",
"YARN_APPLICATION_QUEUE": "test queue",
"YARN_APPLICATION_TYPE": "test app type",
"YARN_APPLICATION_PRIORITY": 10,
"YARN_APPLICATION_TAGS": [
"test",
"tags"
],
"YARN_APPLICATION_STATE": "SUBMITTED"
},
"domainId": null,
"relatedEntitiesJAXB": {
},
"primaryFiltersJAXB": {
},
"otherInfoJAXB": {
"YARN_APPLICATION_CALLER_CONTEXT": "context",
"YARN_APPLICATION_NAME": "test app",
"YARN_APPLICATION_USER": "test user",
"YARN_APPLICATION_UNMANAGED_APPLICATION": false,
"YARN_APP_NODE_LABEL_EXPRESSION": "high-cpu",
"YARN_APPLICATION_SUBMITTED_TIME": 2147483648,
"YARN_AM_CONTAINER_LAUNCH_COMMAND": [
"java -Xmx1024m"
],
"YARN_AM_NODE_LABEL_EXPRESSION": "high-mem",
"YARN_APPLICATION_QUEUE": "test queue",
"YARN_APPLICATION_TYPE": "test app type",
"YARN_APPLICATION_PRIORITY": 10,
"YARN_APPLICATION_TAGS": [
"test",
"tags"
],
"YARN_APPLICATION_STATE": "SUBMITTED"
}
}
]
}
```
The `otherInfo` / `otherInfoJAXB`, `relatedEntities` /
`relatedEntitiesJAXB`, and `primaryFilters` / `primaryFiltersJAXB` are three
pairs of duplicate data.
Taking `YARN_APPLICATION_TAGS` as an example, the same data exists in two
places, so it was parsed twice, leading to duplicate data. The original data
was `["test", "tags"]`, but it was actually converted to `["test", "tags",
"test", "tags"]`.
JAXB is an annotation for XML, which has no effect on JSON. Therefore, I
used annotations to exclude this property and reviewed all the Timeline classes
to ensure that JAXB properties are excluded during JSON parsing.
> yarn tests have stopped running.
> --------------------------------
>
> Key: YARN-11764
> URL: https://issues.apache.org/jira/browse/YARN-11764
> Project: Hadoop YARN
> Issue Type: Bug
> Components: gpg, nodemanager, resourcemanager
> Affects Versions: 3.5.0
> Reporter: Shilun Fan
> Assignee: Shilun Fan
> Priority: Major
> Labels: pull-request-available
>
> After completing HADOOP-15984, we added JUnit5 test dependencies to some
> modules. These dependencies caused maven-surefire to fail to recognize JUnit4
> tests, leading to the cessation of unit tests in some YARN modules. As a
> result, some YARN unit tests failed and were not detected in time. This JIRA
> will track and resolve these issues, including the stopped unit tests and
> test errors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
