otterc commented on pull request #30349:
URL: https://github.com/apache/spark/pull/30349#issuecomment-726249419
@tgravescs @HyukjinKwon I am not entirely sure why these tests would fail
with the errors below only for hadoop-2.7 profile.
I see that the `network-yarn` module is build differently. The target jar of
this module is a `spark-{version}-yarn-shuffle.jar`. However, that doesn't
explain why this will fail only with hadoop-2.7 profile and not for hadoop 3.2.
It could also be what @mridulm pointed out that some jar is missing from
hadoop-2.7 profile. However, I didn't add any additional dependencies. I didn't
create a new `logger` instance. Just using the existing `logger` instance in
`YarnShuffleService`.
The reason I am deleting this
`common/network-yarn/src/test/java/org/apache/spark/network/yarn/YarnShuffleServiceSuite.java`
was that I added this with https://github.com/apache/spark/pull/30062 because
I missed that `
resource-managers/yarn/src/test/scala/org/apache/spark/network/yarn/YarnShuffleServiceSuite.scala`
existed. I reckon there is no need to have 2 test classes for same class.
However, it would have been better to have the test class in the same module.
I can dig deeper into why this happened but that would require more time. In
the meantime just wanted to fix the failures with hadoop-2.7 build.
```
[ERROR]
testCreateDefaultMergedShuffleFileManagerInstance(org.apache.spark.network.yarn.YarnShuffleServiceSuite)
Time elapsed: 0.421 s <<< ERROR!
java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
at
org.apache.spark.network.yarn.YarnShuffleServiceSuite.testCreateDefaultMergedShuffleFileManagerInstance(YarnShuffleServiceSuite.java:37)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.logging.LogFactory
at
org.apache.spark.network.yarn.YarnShuffleServiceSuite.testCreateDefaultMergedShuffleFileManagerInstance(YarnShuffleServiceSuite.java:37)
[ERROR]
testCreateRemoteBlockPushResolverInstance(org.apache.spark.network.yarn.YarnShuffleServiceSuite)
Time elapsed: 0 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.spark.network.yarn.YarnShuffleService
at
org.apache.spark.network.yarn.YarnShuffleServiceSuite.testCreateRemoteBlockPushResolverInstance(YarnShuffleServiceSuite.java:47)
[ERROR]
testInvalidClassNameOfMergeManagerWillUseNoOpInstance(org.apache.spark.network.yarn.YarnShuffleServiceSuite)
Time elapsed: 0.001 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.spark.network.yarn.YarnShuffleService
at
org.apache.spark.network.yarn.YarnShuffleServiceSuite.testInvalidClassNameOfMergeManagerWillUseNoOpInstance(YarnShuffleServiceSuite.java:57)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
