Github user dennishuo commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36709395
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala ---
@@ -170,6 +173,39 @@ class ClientSuite extends SparkFunSuite with Matchers
with BeforeAndAfterAll {
cp should contain ("/remotePath/my1.jar")
}
+ test("configuration and args propagate through
createApplicationSubmissionContext") {
--- End diff --
Indeed I did before sending out the first PR:
$ sbt/sbt -Pyarn -Phadoop-2.3 -Phive -Phive-thriftserver yarn/test
test-only
...
[info] Compiling 1 Scala source to
/home/dhuo/github/dhuo_spark2/spark/yarn/target/scala-2.10/test-classes...
[info] YarnAllocatorSuite:
[info] - single container allocated (667 milliseconds)
[info] - some containers allocated (221 milliseconds)
[info] - receive more containers than requested (200 milliseconds)
[info] - decrease total requested executors (17 milliseconds)
[info] - decrease total requested executors to less than currently running
(13 milliseconds)
[info] - kill executors (25 milliseconds)
[info] - lost executor removed from backend (26 milliseconds)
[info] - memory exceeded diagnostic regexes (5 milliseconds)
[info] YarnClusterSuite:
[info] - run Spark in yarn-client mode (14 seconds, 642 milliseconds)
[info] - run Spark in yarn-cluster mode (13 seconds, 292 milliseconds)
[info] - run Spark in yarn-cluster mode unsuccessfully (10 seconds, 150
milliseconds)
[info] - run Python application in yarn-client mode (16 seconds, 129
milliseconds)
[info] - run Python application in yarn-cluster mode (17 seconds, 323
milliseconds)
[info] - user class path first in client mode (13 seconds, 628 milliseconds)
[info] - user class path first in cluster mode (13 seconds, 120
milliseconds)
[info] ContainerPlacementStrategySuite:
[info] - allocate locality preferred containers with enough resource and no
matched existed containers (838 milliseconds)
[info] - allocate locality preferred containers with enough resource and
partially matched containers (21 milliseconds)
[info] - allocate locality preferred containers with limited resource and
partially matched containers (16 milliseconds)
[info] - allocate locality preferred containers with fully matched
containers (14 milliseconds)
[info] - allocate containers with no locality preference (16 milliseconds)
[info] YarnSparkHadoopUtilSuite:
[info] - shell script escaping (17 milliseconds)
[info] - Yarn configuration override (53 milliseconds)
[info] - test getApplicationAclsForYarn acls on (10 milliseconds)
[info] - test getApplicationAclsForYarn acls on and specify users (12
milliseconds)
[info] - test expandEnvironment result (4 milliseconds)
[info] - test getClassPathSeparator result (6 milliseconds)
[info] - check access nns empty (20 milliseconds)
[info] - check access nns unset (18 milliseconds)
[info] - check access nns (16 milliseconds)
[info] - check access nns space (17 milliseconds)
[info] - check access two nns (16 milliseconds)
[info] - check token renewer (206 milliseconds)
[info] - check token renewer default (32 milliseconds)
[info] ClientSuite:
[info] - default Yarn application classpath (29 milliseconds)
[info] - default MR application classpath (3 milliseconds)
[info] - resultant classpath for an application that defines a classpath
for YARN (16 milliseconds)
[info] - resultant classpath for an application that defines a classpath
for MR (11 milliseconds)
[info] - resultant classpath for an application that defines both
classpaths, YARN and MR (9 milliseconds)
[info] - Local jar URIs (40 milliseconds)
[info] - Jar path propagation through SparkConf (115 milliseconds)
[info] - Cluster path translation (14 milliseconds)
[info] - configuration and args propagate through
createApplicationSubmissionContext (8 milliseconds)
[info] ClientDistributedCacheManagerSuite:
[info] - test getFileStatus empty (24 milliseconds)
[info] - test getFileStatus cached (1 millisecond)
[info] - test addResource (10 milliseconds)
[info] - test addResource link null (3 milliseconds)
[info] - test addResource appmaster only (1 millisecond)
[info] - test addResource archive (7 milliseconds)
[info] ScalaTest
[info] Run completed in 2 minutes.
[info] Total number of tests run: 48
[info] Suites: completed 6, aborted 0
[info] Tests: succeeded 48, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 48, Failed 0, Errors 0, Passed 48
[success] Total time: 136 s, completed Aug 7, 2015 6:47:43 PM
$ sbt/sbt -Pyarn -Phadoop-2.4 -Phive -Phive-thriftserver yarn/test
...
[info] YarnAllocatorSuite:
[info] - single container allocated (1 second, 508 milliseconds)
[info] - some containers allocated (163 milliseconds)
[info] - receive more containers than requested (160 milliseconds)
[info] - decrease total requested executors (26 milliseconds)
[info] - decrease total requested executors to less than currently running
(13 milliseconds)
[info] - kill executors (96 milliseconds)
[info] - lost executor removed from backend (48 milliseconds)
[info] - memory exceeded diagnostic regexes (13 milliseconds)
[info] YarnClusterSuite:
[info] - run Spark in yarn-client mode (22 seconds, 554 milliseconds)
[info] - run Spark in yarn-cluster mode (15 seconds, 772 milliseconds)
[info] - run Spark in yarn-cluster mode unsuccessfully (10 seconds, 285
milliseconds)
[info] - run Python application in yarn-client mode (18 seconds, 747
milliseconds)
[info] - run Python application in yarn-cluster mode (16 seconds, 356
milliseconds)
[info] - user class path first in client mode (13 seconds, 841 milliseconds)
[info] - user class path first in cluster mode (14 seconds, 344
milliseconds)
[info] ContainerPlacementStrategySuite:
[info] - allocate locality preferred containers with enough resource and no
matched existed containers (719 milliseconds)
[info] - allocate locality preferred containers with enough resource and
partially matched containers (14 milliseconds)
[info] - allocate locality preferred containers with limited resource and
partially matched containers (15 milliseconds)
[info] - allocate locality preferred containers with fully matched
containers (14 milliseconds)
[info] - allocate containers with no locality preference (16 milliseconds)
[info] YarnSparkHadoopUtilSuite:
[info] - shell script escaping (18 milliseconds)
[info] - Yarn configuration override (58 milliseconds)
[info] - test getApplicationAclsForYarn acls on (11 milliseconds)
[info] - test getApplicationAclsForYarn acls on and specify users (17
milliseconds)
[info] - test expandEnvironment result (9 milliseconds)
[info] - test getClassPathSeparator result (5 milliseconds)
[info] - check access nns empty (22 milliseconds)
[info] - check access nns unset (21 milliseconds)
[info] - check access nns (22 milliseconds)
[info] - check access nns space (17 milliseconds)
[info] - check access two nns (18 milliseconds)
[info] - check token renewer (176 milliseconds)
[info] - check token renewer default (82 milliseconds)
[info] ClientSuite:
[info] - default Yarn application classpath (293 milliseconds)
[info] - default MR application classpath (5 milliseconds)
[info] - resultant classpath for an application that defines a classpath
for YARN (50 milliseconds)
[info] - resultant classpath for an application that defines a classpath
for MR (19 milliseconds)
[info] - resultant classpath for an application that defines both
classpaths, YARN and MR (16 milliseconds)
[info] - Local jar URIs (87 milliseconds)
[info] - Jar path propagation through SparkConf (172 milliseconds)
[info] - Cluster path translation (12 milliseconds)
[info] - configuration and args propagate through
createApplicationSubmissionContext (62 milliseconds)
[info] ClientDistributedCacheManagerSuite:
[info] - test getFileStatus empty (30 milliseconds)
[info] - test getFileStatus cached (0 milliseconds)
[info] - test addResource (12 milliseconds)
[info] - test addResource link null (3 milliseconds)
[info] - test addResource appmaster only (1 millisecond)
[info] - test addResource archive (14 milliseconds)
[info] ScalaTest
[info] Run completed in 2 minutes, 18 seconds.
[info] Total number of tests run: 48
[info] Suites: completed 6, aborted 0
[info] Tests: succeeded 48, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 48, Failed 0, Errors 0, Passed 48
[success] Total time: 156 s, completed Aug 10, 2015 10:58:20 AM
Kicking off a new round of build/tests locally now after the changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
