slfan1989 commented on PR #7345: URL: https://github.com/apache/hadoop/pull/7345#issuecomment-2630507008
I have fixed the YARN-related unit test errors and tested them locally. Next, I will provide a detailed explanation of how we resolved these unit test issues. > hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher These unit test errors were discovered during the integration testing of the ResourceManager module. The error message is as follows: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7345/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt ``` [ERROR] TestSystemMetricsPublisher.testPublishApplicationMetrics Time elapsed: 1.279 s <<< FAILURE! java.lang.AssertionError at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.assertTrue(Assert.java:42) at org.junit.Assert.assertTrue(Assert.java:53) at org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher.testPublishApplicationMetrics(TestSystemMetricsPublisher.java:230) ``` The code causing the error is as follows: ``` Assert.assertTrue(verifyAppTags(app.getApplicationTags(), entity.getOtherInfo())); ``` We found that the contents of `app.getApplicationTags()` and `entity.getOtherInfo()` are inconsistent, with `entity.getOtherInfo()` containing duplicate entries. `app.getApplicationTags()` is expected to contain `2` elements, but `entity.getOtherInfo()` contains `4` elements. https://github.com/apache/hadoop/blob/44a5cba78ac7d29a72d3452fbd807f31ae90c325/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java#L555-L558 The cause of this issue is that the data in `TimelineEntity.java` is being serialized twice, as there are two properties in the class representing the same data. As shown in the code below, this results in two identical entries in the serialized JSON, one named `otherinfo` and the other named `otherInfoJAXB`. https://github.com/apache/hadoop/blob/44a5cba78ac7d29a72d3452fbd807f31ae90c325/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelineEntity.java#L310-L315 The serialized JSON is as follows: ``` { "entities": [ { "entityType": "YARN_APPLICATION", "entityId": "application_0_0001", "startTime": null, "events": [ { "timestamp": 2147483649, "eventType": "YARN_APPLICATION_CREATED", "eventInfo": { } } ], "relatedEntities": { }, "primaryFilters": { }, "otherInfo": { "YARN_APPLICATION_CALLER_CONTEXT": "context", "YARN_APPLICATION_NAME": "test app", "YARN_APPLICATION_USER": "test user", "YARN_APPLICATION_UNMANAGED_APPLICATION": false, "YARN_APP_NODE_LABEL_EXPRESSION": "high-cpu", "YARN_APPLICATION_SUBMITTED_TIME": 2147483648, "YARN_AM_CONTAINER_LAUNCH_COMMAND": [ "java -Xmx1024m" ], "YARN_AM_NODE_LABEL_EXPRESSION": "high-mem", "YARN_APPLICATION_QUEUE": "test queue", "YARN_APPLICATION_TYPE": "test app type", "YARN_APPLICATION_PRIORITY": 10, "YARN_APPLICATION_TAGS": [ "test", "tags" ], "YARN_APPLICATION_STATE": "SUBMITTED" }, "domainId": null, "relatedEntitiesJAXB": { }, "primaryFiltersJAXB": { }, "otherInfoJAXB": { "YARN_APPLICATION_CALLER_CONTEXT": "context", "YARN_APPLICATION_NAME": "test app", "YARN_APPLICATION_USER": "test user", "YARN_APPLICATION_UNMANAGED_APPLICATION": false, "YARN_APP_NODE_LABEL_EXPRESSION": "high-cpu", "YARN_APPLICATION_SUBMITTED_TIME": 2147483648, "YARN_AM_CONTAINER_LAUNCH_COMMAND": [ "java -Xmx1024m" ], "YARN_AM_NODE_LABEL_EXPRESSION": "high-mem", "YARN_APPLICATION_QUEUE": "test queue", "YARN_APPLICATION_TYPE": "test app type", "YARN_APPLICATION_PRIORITY": 10, "YARN_APPLICATION_TAGS": [ "test", "tags" ], "YARN_APPLICATION_STATE": "SUBMITTED" } } ] } ``` The `otherInfo` / `otherInfoJAXB`, `relatedEntities` / `relatedEntitiesJAXB`, and `primaryFilters` / `primaryFiltersJAXB` are three pairs of duplicate data. Taking `YARN_APPLICATION_TAGS` as an example, the same data exists in two places, so it was parsed twice, leading to duplicate data. The original data was `["test", "tags"]`, but it was actually converted to `["test", "tags", "test", "tags"]`. JAXB is an annotation for XML, which has no effect on JSON. Therefore, I used annotations to exclude this property and reviewed all the Timeline classes to ensure that JAXB properties are excluded during JSON parsing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
