[ https://issues.apache.org/jira/browse/SPARK-6343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482496#comment-14482496 ]
Apache Spark commented on SPARK-6343: ------------------------------------- User 'parente' has created a pull request for this issue: https://github.com/apache/spark/pull/5382 > Make doc more explicit regarding network connectivity requirements > ------------------------------------------------------------------ > > Key: SPARK-6343 > URL: https://issues.apache.org/jira/browse/SPARK-6343 > Project: Spark > Issue Type: Improvement > Components: Documentation > Reporter: Peter Parente > Priority: Minor > > As a new user of Spark, I read through the official documentation before > attempting to stand-up my own cluster and write my own driver application. > But only after attempting to run my app remotely against my cluster did I > realize that full network connectivity (layer 3) is necessary between my > driver program and worker nodes (i.e., my driver was *listening* for > connections from my workers). > I returned to the documentation to see how I had missed this requirement. On > a second read-through, I saw that the doc hints at it in a few places (e.g., > [driver > config|http://spark.apache.org/docs/1.2.0/configuration.html#networking], > [submitting applications > suggestion|http://spark.apache.org/docs/1.2.0/submitting-applications.html], > [cluster overview|http://spark.apache.org/docs/1.2.0/cluster-overview.html]) > but never outright says it. > I think it would help would-be users better understand how Spark works to > state the network connectivity requirements right up-front in the overview > section of the doc. I suggest revising the diagram and accompanying text > found on the [overview > page|http://spark.apache.org/docs/1.2.0/cluster-overview.html]: > !http://spark.apache.org/docs/1.2.0/img/cluster-overview.png! > so that it depicts at least the directionality of the network connections > initiated (perhaps like so): > !http://i.imgur.com/2dqGbCr.png! > and states that the driver must listen for and accept connections from other > Spark components on a variety of ports. > Please treat my diagram and text as strawmen: I expect more experienced Spark > users and developers will have better ideas on how to convey these > requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org