Mich's invocation is for starting a Spark application against an already running Spark standalone cluster. It will not start the cluster for you.
We used to not use "spark-submit", but we started using it when it solved some problem for us. Perhaps that day has also come for you? :) On Fri, Jul 15, 2016 at 5:14 PM, Jean Georges Perrin <j...@jgp.net> wrote: > I don't use submit: I start my standalone cluster and connect to it > remotely. Is that a bad practice? > > I'd like to be able to it dynamically as the system knows whether it needs > more or less resources based on its own context > > On Jul 15, 2016, at 10:55 AM, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Hi, > > You can also do all this at env or submit time with spark-submit which I > believe makes it more flexible than coding in. > > Example > > ${SPARK_HOME}/bin/spark-submit \ > --packages com.databricks:spark-csv_2.11:1.3.0 \ > --driver-memory 2G \ > --num-executors 2 \ > --executor-cores 3 \ > --executor-memory 2G \ > --master spark://50.140.197.217:7077 \ > --conf "spark.scheduler.mode=FAIR" \ > --conf > "spark.executor.extraJavaOptions=-XX:+PrintGCDetails > -XX:+PrintGCTimeStamps" \ > --jars > /home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar \ > --class "${FILE_NAME}" \ > --conf "spark.ui.port=${SP}" \ > > HTH > > Dr Mich Talebzadeh > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > http://talebzadehmich.wordpress.com > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 15 July 2016 at 13:48, Jean Georges Perrin <j...@jgp.net> wrote: > >> Merci Nihed, this is one of the tests I did :( still not working >> >> >> >> On Jul 15, 2016, at 8:41 AM, nihed mbarek <nihe...@gmail.com> wrote: >> >> can you try with : >> SparkConf conf = new SparkConf().setAppName("NC Eatery app").set( >> "spark.executor.memory", "4g") >> .setMaster("spark://10.0.100.120:7077"); >> if (restId == 0) { >> conf = conf.set("spark.executor.cores", "22"); >> } else { >> conf = conf.set("spark.executor.cores", "2"); >> } >> JavaSparkContext javaSparkContext = new JavaSparkContext(conf); >> >> On Fri, Jul 15, 2016 at 2:31 PM, Jean Georges Perrin <j...@jgp.net> wrote: >> >>> Hi, >>> >>> Configuration: standalone cluster, Java, Spark 1.6.2, 24 cores >>> >>> My process uses all the cores of my server (good), but I am trying to >>> limit it so I can actually submit a second job. >>> >>> I tried >>> >>> SparkConf conf = new SparkConf().setAppName("NC Eatery app").set( >>> "spark.executor.memory", "4g") >>> .setMaster("spark://10.0.100.120:7077"); >>> if (restId == 0) { >>> conf = conf.set("spark.executor.cores", "22"); >>> } else { >>> conf = conf.set("spark.executor.cores", "2"); >>> } >>> JavaSparkContext javaSparkContext = new JavaSparkContext(conf); >>> >>> and >>> >>> SparkConf conf = new SparkConf().setAppName("NC Eatery app").set( >>> "spark.executor.memory", "4g") >>> .setMaster("spark://10.0.100.120:7077"); >>> if (restId == 0) { >>> conf.set("spark.executor.cores", "22"); >>> } else { >>> conf.set("spark.executor.cores", "2"); >>> } >>> JavaSparkContext javaSparkContext = new JavaSparkContext(conf); >>> >>> but it does not seem to take it. Any hint? >>> >>> jg >>> >>> >>> >> >> >> -- >> >> M'BAREK Med Nihed, >> Fedora Ambassador, TUNISIA, Northern Africa >> http://www.nihed.com >> >> <http://tn.linkedin.com/in/nihed> >> >> >> > >