LuciferYang opened a new pull request, #36917:
URL: https://github.com/apache/spark/pull/36917
### What changes were proposed in this pull request?
Build `yarn` module with `-Phadoop2` profile failed now as follows:
```
[ERROR] [Error]
/basedir/spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:454:
value DECOMMISSIONING is not a member of object
org.apache.hadoop.yarn.api.records.NodeState
```
The above compilation error due to `hadoop-2.7` not support
`NodeState.DECOMMISSIONING`, so this pr change to use string comparison instead
for compilation, and the test suite `Test YARN container decommissioning` in
`YarnAllocatorSuite` should only run when `VersionUtils.isHadoop3` is true
### Why are the changes needed?
Fix yarn module compilation error with `-Phadoop2` profile
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GA
- Manual test
run `mvn clean install -DskipTests -pl resource-managers/yarn -am -Pyarn
-Phadoop-2`
**Before**
```
[ERROR] [Error]
/basedirspark-source/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:454:
value DECOMMISSIONING is not a member of object
org.apache.hadoop.yarn.api.records.NodeState
[ERROR] one error found
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 3.252
s]
[INFO] Spark Project Tags ................................. SUCCESS [ 5.735
s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 5.492
s]
[INFO] Spark Project Networking ........................... SUCCESS [ 8.251
s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 6.334
s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 15.326
s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 4.905
s]
[INFO] Spark Project Core ................................. SUCCESS [02:07
min]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 17.382
s]
[INFO] Spark Project YARN ................................. FAILURE [ 7.718
s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 03:22 min
[INFO] Finished at: 2022-06-20T11:57:54+08:00
[INFO]
------------------------------------------------------------------------
```
**After**
```
[INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 5.451
s]
[INFO] Spark Project Tags ................................. SUCCESS [ 5.739
s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 5.908
s]
[INFO] Spark Project Networking ........................... SUCCESS [ 8.310
s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 5.857
s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 8.439
s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 4.795
s]
[INFO] Spark Project Core ................................. SUCCESS [02:36
min]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 15.044
s]
[INFO] Spark Project YARN ................................. SUCCESS [ 32.517
s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 04:09 min
[INFO] Finished at: 2022-06-20T13:10:04+08:00
[INFO]
------------------------------------------------------------------------
```
run `mvn clean install -pl resource-managers/yarn -Pyarn -Phadoop-2
-Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnAllocatorSuite`
```
- Test YARN container decommissioning !!! CANCELED !!!
org.apache.spark.util.VersionUtils.isHadoop3 was false
(YarnAllocatorSuite.scala:749)
Run completed in 2 seconds, 140 milliseconds.
Total number of tests run: 16
Suites: completed 2, aborted 0
Tests: succeeded 16, failed 0, canceled 6, ignored 0, pending 0
All tests passed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
