[
https://issues.apache.org/jira/browse/SPARK-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-18340:
---------------------------------
Description:
It seems there are some problems with handling wrong options as below:
*{{spark-submit}} script - this one looks fine
{code}
spark-submit --aabbcc
Error: Unrecognized option: --aabbcc
Usage: spark-submit [options] <app jar | python file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]
Options:
--master MASTER_URL spark://host:port, mesos://host:port, yarn, or
local.
--deploy-mode DEPLOY_MODE Whether to launch the driver program locally
("client") or
on one of the worker machines inside the cluster
("cluster")
(Default: client).
--class CLASS_NAME Your application's main class (for Java / Scala
apps).
--name NAME A name of your application.
...
{code}
*{{spark-sql}} script - this one looks fine
{code}
spark-sql --aabbcc
Unrecognized option: --aabbcc
usage: hive
-d,--define <key=value> Variable subsitution to apply to hive
commands. e.g. -d A=B or --define A=B
--database <databasename> Specify the database to use
...
{code}
*{{sparkr}} script - this one might be a bit major because users possibly
mistakenly put some wrong options with typos and the error message does not
indicate the options are wrong.
{code}
sparkr --aabbcc
...
Error in sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap, :
JVM is not ready after 10 seconds
>
{code}
*{{pyspark}} script - we could make the error message consistently with the
others
{code}
pyspark --aabbcc
Exception in thread "main" java.lang.IllegalArgumentException: pyspark does not
support any application options.
at
org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
at
org.apache.spark.launcher.SparkSubmitCommandBuilder.buildPySparkShellCommand(SparkSubmitCommandBuilder.java:290)
at
org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:147)
at org.apache.spark.launcher.Main.main(Main.java:86)
{code}
*{{spark-shell}} - it seems the error message is not pretty kind like
{{spark-submit}} or {{spark-sql}}.
{code}
spark-shell --aabbcc
bad option: '--aabbcc'
{code}
was:
It seems there are some problems with handling wrong options as below:
*{{spark-submit}} script - this one looks fine
{code}
spark-submit --aabbcc
Error: Unrecognized option: --aabbcc
Usage: spark-submit [options] <app jar | python file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]
Options:
--master MASTER_URL spark://host:port, mesos://host:port, yarn, or
local.
--deploy-mode DEPLOY_MODE Whether to launch the driver program locally
("client") or
on one of the worker machines inside the cluster
("cluster")
(Default: client).
--class CLASS_NAME Your application's main class (for Java / Scala
apps).
--name NAME A name of your application.
...
{code}
*{{spark-sql}} script - this one looks fine
{code}
spark-sql --aabbcc
Unrecognized option: --aabbcc
usage: hive
-d,--define <key=value> Variable subsitution to apply to hive
commands. e.g. -d A=B or --define A=B
--database <databasename> Specify the database to use
...
{code}
*{{sparkr}} script - this one might be a bit major because users possibly
mistakenly put some wrong options with typos and the error message does not
indicate the options are wrong.
{code}
sparkr --aabbcc
...
Error in sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap, :
JVM is not ready after 10 seconds
>
{code}
*{{pyspark}} script - we could make the error message consistently with the
others
{code}
pyspark --aabbcc
Exception in thread "main" java.lang.IllegalArgumentException: pyspark does not
support any application options.
at
org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
at
org.apache.spark.launcher.SparkSubmitCommandBuilder.buildPySparkShellCommand(SparkSubmitCommandBuilder.java:290)
at
org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:147)
at org.apache.spark.launcher.Main.main(Main.java:86)
{code}
> Inconsistent error messages in launching scripts and hanging in sparkr script
> for wrong options
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-18340
> URL: https://issues.apache.org/jira/browse/SPARK-18340
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell, Spark Submit
> Reporter: Hyukjin Kwon
> Priority: Minor
>
> It seems there are some problems with handling wrong options as below:
> *{{spark-submit}} script - this one looks fine
> {code}
> spark-submit --aabbcc
> Error: Unrecognized option: --aabbcc
> Usage: spark-submit [options] <app jar | python file> [app arguments]
> Usage: spark-submit --kill [submission ID] --master [spark://...]
> Usage: spark-submit --status [submission ID] --master [spark://...]
> Usage: spark-submit run-example [options] example-class [example args]
> Options:
> --master MASTER_URL spark://host:port, mesos://host:port, yarn, or
> local.
> --deploy-mode DEPLOY_MODE Whether to launch the driver program locally
> ("client") or
> on one of the worker machines inside the
> cluster ("cluster")
> (Default: client).
> --class CLASS_NAME Your application's main class (for Java / Scala
> apps).
> --name NAME A name of your application.
> ...
> {code}
> *{{spark-sql}} script - this one looks fine
> {code}
> spark-sql --aabbcc
> Unrecognized option: --aabbcc
> usage: hive
> -d,--define <key=value> Variable subsitution to apply to hive
> commands. e.g. -d A=B or --define A=B
> --database <databasename> Specify the database to use
> ...
> {code}
> *{{sparkr}} script - this one might be a bit major because users possibly
> mistakenly put some wrong options with typos and the error message does not
> indicate the options are wrong.
> {code}
> sparkr --aabbcc
> ...
> Error in sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap, :
> JVM is not ready after 10 seconds
> >
> {code}
> *{{pyspark}} script - we could make the error message consistently with the
> others
> {code}
> pyspark --aabbcc
> Exception in thread "main" java.lang.IllegalArgumentException: pyspark does
> not support any application options.
> at
> org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
> at
> org.apache.spark.launcher.SparkSubmitCommandBuilder.buildPySparkShellCommand(SparkSubmitCommandBuilder.java:290)
> at
> org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:147)
> at org.apache.spark.launcher.Main.main(Main.java:86)
> {code}
> *{{spark-shell}} - it seems the error message is not pretty kind like
> {{spark-submit}} or {{spark-sql}}.
> {code}
> spark-shell --aabbcc
> bad option: '--aabbcc'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
