[
https://issues.apache.org/jira/browse/MAPREDUCE-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782800#comment-17782800
]
ASF GitHub Bot commented on MAPREDUCE-7461:
-------------------------------------------
kavvya97 opened a new pull request, #6252:
URL: https://github.com/apache/hadoop/pull/6252
**Setup:**
Java version: openjdk 11.0.20.1
Maven version: Apache Maven 3.6.3
### **Issue**: https://issues.apache.org/jira/browse/MAPREDUCE-7461
### Description of PR
The following tests can fail due to flakiness while comparing the contents
of the generated XML response.
**Module**:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
`org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs#testJobIdXML`
`org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs#testJobsXML`
**Module**:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
`org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs#testJobIdXML`
### Steps to reproduce
1. `git clone https://github.com/apache/hadoop`
2. `mvn install -pl
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
-am -DskipTests`
3. Run the tests
`mvn -pl
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
test
-Dtests=org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs#testJobIdXML`
4. Run the test with the Nondex tool and observe the test results
`mvn -pl
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
edu.illinois:nondex-maven-plugin:2.1.1:nondex
-Dtest=org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs#testJobsIdXML`
- Test Fails when Running Nondex in ONE mode (Assumes deterministic
implementation of code but shuffled once different from underlying
implementation) `-DnondexMode=ONE` & FULL Mode ` -DnondexMode=FULL` (shuffles
differently for each call)
### Root Cause
The test attempts to send a HTTP GET request to a specific URL and expects a
response in XML format. However, XML response order is not necessarily
guaranteed. The contents of the XML and the tags are compared with Job contents
from `appContext` based on [job
Id](https://github.com/kavvya97/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java#L493-L494)
in
[verifyAMJobXML](https://github.com/kavvya97/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java#L486)
/
[verifyHsJobXML](https://github.com/kavvya97/hadoop/blob/9c621fcea72a988c930ef614a7c22de00d0c7d21/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java#L232).
When comparing the the XML contents, The `<name>` tag occurs in multiple
places inside <job> field. However, the root element within <job> is not always
compared due to non-deterministic order.
When the name tag is being compared, the test utilizes
[WebServicesTestUtils.java
getXmlString](https://github.com/kavvya97/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/WebServicesTestUtils.java#L78)
method for retrieving the name from the XML content. However, It always takes
the first <name> tag irrespective of whether it is nested or in root which
causes the test to fail and become flaky.
Since the XML contents are not ordered, The following errors occur
```
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
8.377 s <<< FAILURE! - in
org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs
[ERROR]
testJobIdXML(org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs)
Time elapsed: 8.361 s <<< FAILURE!
java.lang.AssertionError:
[name]
Expecting:
"mapreduce.job.acl-view-job"
to match pattern:
"RandomWriter"
```
### Fix
Since [WebServicesTestUtils.java
getXmlString](https://github.com/kavvya97/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/WebServicesTestUtils.java#L78)
compares the first <name> tag only which might not necessary be the root
<name> tag, The Fix uses Xpath to resolve the conflicts by identify the root
<name> tag. Thus the test passes since the root tag is always retrieved
irrespective of xml order.
### How was this patch tested?
The fix was tested by adding a suitable fix and running the Nondex plugin
again and ensuring that all the tests pass in FULL Mode and ONE Mode of the
Nondex runs.
> Fixed assertionComparision failure by resolving xml paths for child elements
> correctly
> --------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7461
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7461
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 3.3.6
> Reporter: Rajiv Ramachandran
> Priority: Minor
> Fix For: 3.3.6
>
>
> The following tests depend on underlying implementation orders which are not
> guarenteed, while comparing the contents of the generated XML response.
> _org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs#testJobIdXML_
> _org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs#testJobsXML_
> _org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs#testJobIdXML_
> The test attempts to send a HTTP GET request to a specific URL and expects a
> response in XML format. However, XML response order is not necessarily
> guaranteed. When comparing the the XML contents, The `<name>` tag occurs in
> multiple places inside <job> field. However, the root element within <job> is
> not always compared due to non-deterministic order. The [getXmlString
> utility|https://github.com/kavvya97/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/WebServicesTestUtils.java#L78]
> always takes the first <name> tag irrespective of whether it is nested or in
> root which causes the test to become flaky.
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.377
> s <<< FAILURE! - in
> org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs
> [ERROR]
> testJobIdXML(org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs)
> Time elapsed: 8.361 s <<< FAILURE!
> java.lang.AssertionError:
> [name]
> Expecting:
> "mapreduce.job.acl-view-job"
> to match pattern:
> "RandomWriter"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
