Hi, I think in coarse grain mode, Spark would schedule and allocate resource by itself, just like what it do in YARN. So you only could see two executors. I think you could get more details from http://spark.apache.org/docs/latest/running-on-mesos.html#mesos-run-modes
On Sat, Feb 13, 2016 at 4:16 AM, Rodrick Brown <[email protected]> wrote: > I'm having an issue with Spark when running on Mesos in coarse grain mode > no matter what options I use or how many free resources I have I only ever > see two executors start. > > If I run Spark in fine grain mode on Mesos I see tasks eventually ramp up > to the desired number I specified. > Anyone know what could be causing this? > > I'm starting my job using the following commands -- > > /usr/bin/timeout 3600 /opt/spark-1.5.1/bin/spark-submit --master " > mesos://zk://prod-zookeeper-1.aws.xxx.com:2181,prod-zookeeper-2.aws.xxx.com:2181,prod-zookeeper-3.aws.xxx.com:2181/mesos" > --conf spark.ui.port=31278 --conf spark.mesos.coarse=true --conf > spark.mesos.extra.cores=3 --conf spark.cores.max=5 --conf > spark.mesos.constraints="rack:spark" --conf spark.sql.tungsten.enabled=true > --class > com.orchard.dataloader.library.originators.prosper.LoadTrade_Prosper > --total-executor-cores 3 --driver-memory 1536M --executor-memory 3011M > --jars /data/orchard/etc/load-tradedata-accumulo-prod > /data/orchard/jars/dataloader-library-59e0441cd06544442d2a84f5842b701e81097988-assembled.jar > > > I'm using spark-1.5.1 and Mesos 0.26 Chronos-2.4.0 is starting the spark > driver. > > > If anyone have any ideas on what/where I should be focusing on to debug > this? > > -- > > *Rodrick Brown* / Systems Engineer > > +1 917 445 6839 / [email protected] > <[email protected]> > > *Orchard Platform* > > 101 5th Avenue, 4th Floor, New York, NY 10003 > > http://www.orchardplatform.com > > Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace Lending > Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/> > > *NOTICE TO RECIPIENTS*: This communication is confidential and intended > for the use of the addressee only. If you are not an intended recipient of > this communication, please delete it immediately and notify the sender by > return email. Unauthorized reading, dissemination, distribution or copying > of this communication is prohibited. This communication does not constitute > an offer to sell or a solicitation of an indication of interest to purchase > any loan, security or any other financial product or instrument, nor is it > an offer to sell or a solicitation of an indication of interest to purchase > any products or services to any persons who are prohibited from receiving > such information under applicable law. The contents of this communication > may not be accurate or complete and are subject to change without notice. > As such, Orchard App, Inc. (including its subsidiaries and affiliates, > "Orchard") makes no representation regarding the accuracy or completeness > of the information contained herein. The intended recipient is advised to > consult its own professional advisors, including those specializing in > legal, tax and accounting matters. Orchard does not provide legal, tax or > accounting advice. > -- Best Regards, Haosdent Huang
