Manoj Samel created SPARK-6653:
----------------------------------
Summary: New configuration property to specify port for
sparkYarnAM actor system
Key: SPARK-6653
URL: https://issues.apache.org/jira/browse/SPARK-6653
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 1.3.0
Environment: Spark On Yarn
Reporter: Manoj Samel
In 1.3.0 code line sparkYarnAM actor system is started on random port. See
org.apache.spark.deploy.yarn ApplicationMaster.scala:282
actorSystem = AkkaUtils.createActorSystem("sparkYarnAM", Utils.localHostName,
0, conf = sparkConf, securityManager = securityMgr)._1
This may be issue when ports between Spark client and the Yarn cluster are
limited by firewall and not all ports are open between client and Yarn cluster.
Proposal is to introduce new property spark.am.actor.port and change code to
val port = sparkConf.getInt("spark.am.actor.port", 0)
actorSystem = AkkaUtils.createActorSystem("sparkYarnAM",
Utils.localHostName, port,
conf = sparkConf, securityManager = securityMgr)._1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
