[
https://issues.apache.org/jira/browse/SPARK-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-2110:
------------------------------
Component/s: PySpark
Documentation
Priority: Blocker (was: Major)
Target Version/s: 1.1.0
Affects Version/s: 1.1.0
I was just affected by this issue, too, and found it really confusing.
I launched a new Spark EC2 cluster and tried to launch a PySpark shell in
{{local}} mode using the command {{./bin/pyspark \-\-master local}}, which
makes sense after reading the current {{\-\-help}}. The {{\-\-master}} option
seems to be ignored here; it seems that the only way to use a different master
is to add {{spark.master local}} to my SparkConf file.
This is a regression from 1.02 since it breaks the pyspark shell examples in
the documentation (Spark Programming Guide). To reproduce the problem locally
with 1.1.0-snapshot1, try running {{./bin/pyspark --master
local-cluster\[2,2,512\]}}; you'll see that no extra executors are started
because the option is ignored, so we end up running in local mode.
> Misleading help displayed for interactive mode pyspark --help
> -------------------------------------------------------------
>
> Key: SPARK-2110
> URL: https://issues.apache.org/jira/browse/SPARK-2110
> Project: Spark
> Issue Type: Bug
> Components: Documentation, PySpark
> Affects Versions: 1.1.0
> Reporter: Prashant Sharma
> Assignee: Patrick Wendell
> Priority: Blocker
>
> The help displayed with command pyspark --help is not relevant for
> interactive mode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
