Re: spark-shell is broken! (bad option: '--master')
an issue 3 - 4 PR, spark dev community is really active :) it seems currently spark-shell takes only some SUBMISSION_OPTS, but no APPLICATION_OPTS do you have plan to add some APPLICATION_OPTS or CLI_OPTS like hive -e hive -f hive -hivevar then we can use our scala code as scripts, run them direktly via spark-shell, without compiling, building, packing and so on... those APPLICATION_OPTS should be some augments for org.apache.spark.repl.Main right? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7798.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: spark-shell is broken! (bad option: '--master')
Just opened a PR based on the branch Patrick mentioned for this issue https://github.com/apache/spark/pull/1864 On Sat, Aug 9, 2014 at 6:48 AM, Patrick Wendell wrote: > Cheng Lian also has a fix for this. I've asked him to make a PR - he > is on China time so it probably won't come until tonight: > > > https://github.com/liancheng/spark/compare/apache:master...liancheng:spark-2894 > > On Fri, Aug 8, 2014 at 3:46 PM, Sandy Ryza > wrote: > > Hi Chutium, > > > > This is currently being addressed in > > https://github.com/apache/spark/pull/1825 > > > > -Sandy > > > > > > On Fri, Aug 8, 2014 at 2:26 PM, chutium wrote: > > > >> no one use spark-shell in master branch? > >> > >> i created a PR as follow up commit of SPARK-2678 and PR #1801: > >> > >> https://github.com/apache/spark/pull/1861 > >> > >> > >> > >> -- > >> View this message in context: > >> > http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7780.html > >> Sent from the Apache Spark Developers List mailing list archive at > >> Nabble.com. > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > >> For additional commands, e-mail: dev-h...@spark.apache.org > >> > >> > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
Re: spark-shell is broken! (bad option: '--master')
Cheng Lian also has a fix for this. I've asked him to make a PR - he is on China time so it probably won't come until tonight: https://github.com/liancheng/spark/compare/apache:master...liancheng:spark-2894 On Fri, Aug 8, 2014 at 3:46 PM, Sandy Ryza wrote: > Hi Chutium, > > This is currently being addressed in > https://github.com/apache/spark/pull/1825 > > -Sandy > > > On Fri, Aug 8, 2014 at 2:26 PM, chutium wrote: > >> no one use spark-shell in master branch? >> >> i created a PR as follow up commit of SPARK-2678 and PR #1801: >> >> https://github.com/apache/spark/pull/1861 >> >> >> >> -- >> View this message in context: >> http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7780.html >> Sent from the Apache Spark Developers List mailing list archive at >> Nabble.com. >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> >> - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: spark-shell is broken! (bad option: '--master')
Hi Chutium, This is currently being addressed in https://github.com/apache/spark/pull/1825 -Sandy On Fri, Aug 8, 2014 at 2:26 PM, chutium wrote: > no one use spark-shell in master branch? > > i created a PR as follow up commit of SPARK-2678 and PR #1801: > > https://github.com/apache/spark/pull/1861 > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7780.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
Re: spark-shell is broken! (bad option: '--master')
no one use spark-shell in master branch? i created a PR as follow up commit of SPARK-2678 and PR #1801: https://github.com/apache/spark/pull/1861 -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7780.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: spark-shell is broken! (bad option: '--master')
maybe this commit is the reason? https://github.com/apache/spark/commit/a6cd31108f0d73ce6823daafe8447677e03cfd13 i fand some discuss in its PR: https://github.com/apache/spark/pull/1801 important is what vanzin said: https://github.com/apache/spark/pull/1801#issuecomment-51545117 i tried to use this: bin/spark-class org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main spark-shell --master local[2] also bad option: '--master' but this works bin/spark-class org.apache.spark.deploy.SparkSubmit --master local[2] --class org.apache.spark.repl.Main spark-shell and some more try out: ~/tachyon/spark$ bin/spark-class org.apache.spark.deploy.SparkSubmit --master local[2] --class org.apache.spark.repl.Main spark-shell --foo bar Spark assembly has been built with Hive, including Datanucleus jars on classpath bad option: '--foo' ~/tachyon/spark$ bin/spark-class org.apache.spark.deploy.SparkSubmit --master local[2] --foo bar --class org.apache.spark.repl.Main spark-shell Spark assembly has been built with Hive, including Datanucleus jars on classpath Error: Unrecognized option '--foo'. Run with --help for usage help or --verbose for debug output god... no test suite for those shell scripts? such a issue is really awful~ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7779.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
spark-shell is broken! (bad option: '--master')
git clone https://github.com/apache/spark.git mvn -Pyarn -Phive -Phive-thriftserver -Dhadoop.version=2.3.0-cdh5.1.0 -DskipTests clean package ./bin/spark-shell --master local[2] then i get Spark assembly has been built with Hive, including Datanucleus jars on classpath bad option: '--master' unbelievable... -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org