GitHub user chesterxgchen opened a pull request:
https://github.com/apache/spark/pull/2111
SPARK-3177 : Yarn-alpha ClientBaseSuite Unit test failed
This second try to fix the Yarn-Alpha unit test failure due to Yarn API
changes.
I have to include SPARK-3175 (pom.xml) changes in order to run the test.
Otherwise, the SBT won't be able to start the project.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/AlpineNow/spark SPARK-3177
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2111.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2111
----
commit c594232b98bbe9214e27b0a9d15e64aad7641c83
Author: chesterxgchen <[email protected]>
Date: 2014-08-21T20:45:45Z
SPARK-3175 : Branch-1.1 SBT build failed for Yarn-Alpha
The issue is that the yarn/alpha/pom.xml using 1.1.0 instead of
1.1.1-SNAPSHOT version.
update the pom.xml to 1.1.1-SNAPSHOT (same as yarn/stable/pom.xml)
commit 30a0ac910808556f1e3477fd077b144eb6f8d9f1
Author: chesterxgchen <[email protected]>
Date: 2014-08-21T21:31:14Z
SPARK-3177 : Yarn-alpha ClientBaseSuite Unit test failed
The Issue is yarn-alpha and yarn has different APIs for certain class
field. In this particular case, we can directly use ClientBase method (which
use reflection )
Add a new method
def getFieldValue2[A: ClassTag, A1: ClassTag, B](clazz: Class[_], field:
String,
defaults: => B)
(mapTo: A => B)(mapTo1: A1 => B) : B =
Try(clazz.getField(field)).map(_.get(null)).map {
case v: A => mapTo(v)
case v1: A1 => mapTo1(v1)
case _ => defaults
}.toOption.getOrElse(defaults)
to handle the cases where the field type can be either type A or A1. In
this new method the type A or A1 is pattern matched and corresponding mapTo
function (mapTo or mapTo1) is used.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
