[jira] [Commented] (SPARK-6653) New configuration property to specify port for sparkYarnAM actor system
[ https://issues.apache.org/jira/browse/SPARK-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14589699#comment-14589699 ] Jeff Zhang commented on SPARK-6653: --- Although this is already committed, would it be better to specify a port range ? New configuration property to specify port for sparkYarnAM actor system --- Key: SPARK-6653 URL: https://issues.apache.org/jira/browse/SPARK-6653 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 1.3.1 Environment: Spark On Yarn Reporter: Manoj Samel Assignee: Shekhar Bansal Priority: Minor Fix For: 1.4.0 In 1.3.0 code line sparkYarnAM actor system is started on random port. See org.apache.spark.deploy.yarn ApplicationMaster.scala:282 actorSystem = AkkaUtils.createActorSystem(sparkYarnAM, Utils.localHostName, 0, conf = sparkConf, securityManager = securityMgr)._1 This may be issue when ports between Spark client and the Yarn cluster are limited by firewall and not all ports are open between client and Yarn cluster. Proposal is to introduce new property spark.am.actor.port and change code to val port = sparkConf.getInt(spark.am.actor.port, 0) actorSystem = AkkaUtils.createActorSystem(sparkYarnAM, Utils.localHostName, port, conf = sparkConf, securityManager = securityMgr)._1 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6653) New configuration property to specify port for sparkYarnAM actor system
[ https://issues.apache.org/jira/browse/SPARK-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14513936#comment-14513936 ] Apache Spark commented on SPARK-6653: - User 'zuxqoj' has created a pull request for this issue: https://github.com/apache/spark/pull/5719 New configuration property to specify port for sparkYarnAM actor system --- Key: SPARK-6653 URL: https://issues.apache.org/jira/browse/SPARK-6653 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 1.3.0 Environment: Spark On Yarn Reporter: Manoj Samel In 1.3.0 code line sparkYarnAM actor system is started on random port. See org.apache.spark.deploy.yarn ApplicationMaster.scala:282 actorSystem = AkkaUtils.createActorSystem(sparkYarnAM, Utils.localHostName, 0, conf = sparkConf, securityManager = securityMgr)._1 This may be issue when ports between Spark client and the Yarn cluster are limited by firewall and not all ports are open between client and Yarn cluster. Proposal is to introduce new property spark.am.actor.port and change code to val port = sparkConf.getInt(spark.am.actor.port, 0) actorSystem = AkkaUtils.createActorSystem(sparkYarnAM, Utils.localHostName, port, conf = sparkConf, securityManager = securityMgr)._1 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6653) New configuration property to specify port for sparkYarnAM actor system
[ https://issues.apache.org/jira/browse/SPARK-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391993#comment-14391993 ] Shixiong Zhu commented on SPARK-6653: - Could you send a pull request to https://github.com/apache/spark ? And because this is a yarn configuration, I recommend spark.yarn.am.port. New configuration property to specify port for sparkYarnAM actor system --- Key: SPARK-6653 URL: https://issues.apache.org/jira/browse/SPARK-6653 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 1.3.0 Environment: Spark On Yarn Reporter: Manoj Samel In 1.3.0 code line sparkYarnAM actor system is started on random port. See org.apache.spark.deploy.yarn ApplicationMaster.scala:282 actorSystem = AkkaUtils.createActorSystem(sparkYarnAM, Utils.localHostName, 0, conf = sparkConf, securityManager = securityMgr)._1 This may be issue when ports between Spark client and the Yarn cluster are limited by firewall and not all ports are open between client and Yarn cluster. Proposal is to introduce new property spark.am.actor.port and change code to val port = sparkConf.getInt(spark.am.actor.port, 0) actorSystem = AkkaUtils.createActorSystem(sparkYarnAM, Utils.localHostName, port, conf = sparkConf, securityManager = securityMgr)._1 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
