rRajivramachandran opened a new pull request, #6215: URL: https://github.com/apache/hadoop/pull/6215
**Setup:** Java version: openjdk 11.0.20.1 Maven version: Apache Maven 3.6.3 ### Description of PR The test `org.apache.hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter#testHumanPrinterAll` can fail due to flakiness. These flakiness occurs because the test utilizes Hashmaps values and converts the values to string to perform the comparision and the order of the objects returned may not be necessarily maintained. This can be detected by utilizing the [Nondex](https://github.com/TestingResearchIllinois/NonDex) plugin. **Steps to reproduce:** 1. `git clone https://github.com/apache/hadoop` 2. `mvn install -pl hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core -am -DskipTests` 3. Run the tests `mvn -pl hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core test -Dtests=org.apache.hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter#testHumanPrinterAll` 4. Run the test with the Nondex tool and observe the test results `mvn -pl hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.apache.hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter#testHumanPrinterAll` - Test Fails when Running Nondex in ONE mode (Assumes deterministic implementation of code but shuffled once different from underlying implementation) `-DnondexMode=ONE` & FULL Mode ` -DnondexMode=FULL` (shuffles differently for each call) ### For code changes: The test utilizes [Hashmap](https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java#L491C18-L491C50) for storing job information and builts the string using [HashMap.values()](https://github.com/apache/hadoop/blob/ea3cb12ec80210fcad985e9cfc37d129afec888b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HumanReadableHistoryViewerPrinter.java#L192). However, the order of the objects returned by a Hashmap may not be maintained. Hence the test fails due to the string comparision. The following error occurs ``` testHumanPrinterAll(org.apache.hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter) Time elapsed: 0.297 s <<< FAILURE! org.junit.ComparisonFailure: expected:<...8501754_0001_m_00000[7 6-Oct-2011 19:15:09 6-Oct-2011 19:15:16 (7sec) SUCCEEDED MAP task list for job_1317928501754_0001 TaskId StartTime FinishTime Error InputSplits ==================================================== task_1317928501754_0001_m_000006 6-Oct-2011 19:15:08 6-Oct-2011 19:15:14 (6sec) SUCCEEDED MAP task list for job_1317928501754_0001 TaskId StartTime FinishTime Error InputSplits ==================================================== task_1317928501754_0001_m_000005 6-Oct-2011 19:15:07 6-Oct-2011 19:15:12 (5sec) SUCCEEDED MAP task list for job_1317928501754_0001 TaskId StartTime FinishTime Error InputSplits ==================================================== task_1317928501754_0001_m_000004 6-Oct-2011 19:15:06 6-Oct-2011 19:15:10 (4sec) SUCCEEDED MAP task list for job_1317928501754_0001 TaskId StartTime FinishTime Error InputSplits ==================================================== task_1317928501754_0001_m_000003 6-Oct-2011 19:15:05 6-Oct-2011 19:15:08 (3sec) SUCCEEDED REDUCE task list for job_1317928501754_0001 TaskId StartTime FinishTime Error ==================================================== task_1317928501754_0001_r_000008 6-Oct-2011 19:15:10 6-Oct-2011 19:15:18 (8sec) SUCCEEDED JOB_CLEANUP task list for job_1317928501754_0001 TaskId StartTime FinishTime Error ==================================================== task_1317928501754_0001_c_000009 6-Oct-2011 19:15:11 6-Oct-2011 19:15:20 (9sec) JOB_SETUP task list for job_1317928501754_0001 TaskId StartTime FinishTime HostName Error TaskLogs ==================================================== attempt_1317928501754_0001_s_000001_1 6-Oct-2011 19:15:03 6-Oct-2011 19:15:04 (1sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_s_000001_1 MAP task list for job_1317928501754_0001 TaskId StartTime FinishTime HostName Error TaskLogs ==================================================== attempt_1317928501754_0001_m_000007_1 6-Oct-2011 19:15:09 6-Oct-2011 19:15:16 (7sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000007_1 attempt_1317928501754_0001_m_000002_1 6-Oct-2011 19:15:04 6-Oct-2011 19:15:06 (2sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000002_1 attempt_1317928501754_0001_m_000006_1 6-Oct-2011 19:15:08 6-Oct-2011 19:15:14 (6sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000006_1 attempt_1317928501754_0001_m_000005_1 6-Oct-2011 19:15:07 6-Oct-2011 19:15:12 (5sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000005_1 attempt_1317928501754_0001_m_000004_1 6-Oct-2011 19:15:06 6-Oct-2011 19:15:10 (4sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000004_1 attempt_1317928501754_0001_m_000003_1 6-Oct-2011 19:15:05 6-Oct-2011 19:15:08 (3sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000003]_1 REDUCE task list...> but was:<...8501754_0001_m_00000[5 6-Oct-2011 19:15:07 6-Oct-2011 19:15:12 (5sec) SUCCEEDED MAP task list for job_1317928501754_0001 TaskId StartTime FinishTime Error InputSplits ==================================================== task_1317928501754_0001_m_000006 6-Oct-2011 19:15:08 6-Oct-2011 19:15:14 (6sec) SUCCEEDED MAP task list for job_1317928501754_0001 TaskId StartTime FinishTime Error InputSplits ==================================================== task_1317928501754_0001_m_000004 6-Oct-2011 19:15:06 6-Oct-2011 19:15:10 (4sec) SUCCEEDED MAP task list for job_1317928501754_0001 TaskId StartTime FinishTime Error InputSplits ==================================================== task_1317928501754_0001_m_000007 6-Oct-2011 19:15:09 6-Oct-2011 19:15:16 (7sec) SUCCEEDED MAP task list for job_1317928501754_0001 TaskId StartTime FinishTime Error InputSplits ==================================================== task_1317928501754_0001_m_000003 6-Oct-2011 19:15:05 6-Oct-2011 19:15:08 (3sec) SUCCEEDED REDUCE task list for job_1317928501754_0001 TaskId StartTime FinishTime Error ==================================================== task_1317928501754_0001_r_000008 6-Oct-2011 19:15:10 6-Oct-2011 19:15:18 (8sec) SUCCEEDED JOB_CLEANUP task list for job_1317928501754_0001 TaskId StartTime FinishTime Error ==================================================== task_1317928501754_0001_c_000009 6-Oct-2011 19:15:11 6-Oct-2011 19:15:20 (9sec) JOB_SETUP task list for job_1317928501754_0001 TaskId StartTime FinishTime HostName Error TaskLogs ==================================================== attempt_1317928501754_0001_s_000001_1 6-Oct-2011 19:15:03 6-Oct-2011 19:15:04 (1sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_s_000001_1 MAP task list for job_1317928501754_0001 TaskId StartTime FinishTime HostName Error TaskLogs ==================================================== attempt_1317928501754_0001_m_000004_1 6-Oct-2011 19:15:06 6-Oct-2011 19:15:10 (4sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000004_1 attempt_1317928501754_0001_m_000005_1 6-Oct-2011 19:15:07 6-Oct-2011 19:15:12 (5sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000005_1 attempt_1317928501754_0001_m_000003_1 6-Oct-2011 19:15:05 6-Oct-2011 19:15:08 (3sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000003_1 attempt_1317928501754_0001_m_000007_1 6-Oct-2011 19:15:09 6-Oct-2011 19:15:16 (7sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000007_1 attempt_1317928501754_0001_m_000002_1 6-Oct-2011 19:15:04 6-Oct-2011 19:15:06 (2sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000002_1 attempt_1317928501754_0001_m_000006_1 6-Oct-2011 19:15:08 6-Oct-2011 19:15:14 (6sec) localhost http://t:1234/tasklog?attemptid=attempt_1317928501754_0001_m_000006]_1 ``` ### Fix The Fix involves extracting a single line of the string and sorting them to perform the string comparision. Since the strings are sorted, The comparision passes successfully. The Alternate way of sorting was converting to json but string being built is not json. Moreover, There was no library available to convert string to json. ### How was this patch tested? The fix was tested by adding a suitable fix and running the Nondex plugin again and ensuring that all the tests pass in FULL Mode and ONE Mode of the Nondex runs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
