You have specified total executor cores of 500.
Thanks, June Taylor System Administrator, Minnesota Population Center University of Minnesota On Fri, Apr 29, 2016 at 11:35 AM, Rodrick Brown <[email protected]> wrote: > I'm running a spark app on Mesos in coarse grain mode > with dynmaic allocation after my task launches I see it still takes more > CPU's than what I specify in my config. Does the following parameter > spark.cores.max=20 work in Mesos or is this a YARN only parameter? is their > something equivalent for Mesos? > > When I go to the framework tab in Mesos I see this task consuming 30 CPU's > instead of 20 I limited it to. > > /usr/bin/timeout 0 /opt/spark-1.6.1/bin/spark-submit \ > --master "mesos://zk://prod-zookeeper-1.aws.xx.com:2181, > prod-zookeeper-2.aws.xx.com:2181,prod-zookeeper-3.aws.xx.com:2181/mesos" \ > --conf spark.ui.port=31354 \ > --conf spark.mesos.coarse=true \ > --conf spark.cores.max=20 \ > --conf spark.mesos.constraints="rack:spark" \ > --conf spark.sql.tungsten.enabled=true \ > --conf spark.shuffle.service.enabled=true \ > --conf spark.dynamicAllocation.enabled=true \ > --conf spark.mesos.executor.memoryOverhead=1606 \ > --class com.orchard.sparkjob.library.kafka.KafkaDirectConsumer \ > --total-executor-cores 500 \ > --driver-memory 2867M \ > --executor-memory 4014M \ > --jars /data/orchard/etc/config/es-services.jar \ > /data/orchard/jars/spark-job-library-3e30539922ff540f4632d1d0745501c48300b89b-assembled.jar > > > -- > > *Rodrick Brown* / Systems Engineer > > +1 917 445 6839 / [email protected] > <[email protected]> > > *Orchard Platform* > > 101 5th Avenue, 4th Floor, New York, NY 10003 > > http://www.orchardplatform.com > > Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace Lending > Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/> > > *NOTICE TO RECIPIENTS*: This communication is confidential and intended > for the use of the addressee only. If you are not an intended recipient of > this communication, please delete it immediately and notify the sender by > return email. Unauthorized reading, dissemination, distribution or copying > of this communication is prohibited. This communication does not constitute > an offer to sell or a solicitation of an indication of interest to purchase > any loan, security or any other financial product or instrument, nor is it > an offer to sell or a solicitation of an indication of interest to purchase > any products or services to any persons who are prohibited from receiving > such information under applicable law. The contents of this communication > may not be accurate or complete and are subject to change without notice. > As such, Orchard App, Inc. (including its subsidiaries and affiliates, > "Orchard") makes no representation regarding the accuracy or completeness > of the information contained herein. The intended recipient is advised to > consult its own professional advisors, including those specializing in > legal, tax and accounting matters. Orchard does not provide legal, tax or > accounting advice. >
