Hi Rodrick, You should check out Cook (github.com/twosigma/cook). It acts as a server for Spark drivers, except that it can manage preemption and fair-sharing between those drivers, so that you don't encounter starvation. If you'd like to discuss it, I'd be happy to hop on a call.
Best, David On Thu, Apr 14, 2016 at 11:09 PM haosdent <[email protected]> wrote: > Hi, @Rodrick Which spark mode are you running? coarse-grained > or fine-grained? > > On Fri, Apr 15, 2016 at 10:05 AM, Rodrick Brown <[email protected]> > wrote: > >> I have hundreds of small spark jobs running on my Mesos cluster >> causing starvation to other frameworks like Marathon on my cluster. >> >> Is their a way to prevent these frameworks from getting offers so often? >> >> Apr 15 02:00:12 prod-mesos-m-3.$SERVER.com mesos-master[10259]: I0415 >> 02:00:12.503734 10266 master.cpp:3641] Processing DECLINE call for offers: >> [ 50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-O7112667 ] for >> framework 50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-15936 >> (KafkaDirectConsumer[trades-topic]) >> at [email protected]:34858. >> >> >> >> >> >> -- >> >> *Rodrick Brown* / Systems Engineer >> >> +1 917 445 6839 / [email protected] >> <[email protected]> >> >> *Orchard Platform* >> >> 101 5th Avenue, 4th Floor, New York, NY 10003 >> >> http://www.orchardplatform.com >> >> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace >> Lending Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/> >> >> *NOTICE TO RECIPIENTS*: This communication is confidential and intended >> for the use of the addressee only. If you are not an intended recipient of >> this communication, please delete it immediately and notify the sender >> by return email. Unauthorized reading, dissemination, distribution or >> copying of this communication is prohibited. This communication does not >> constitute >> an offer to sell or a solicitation of an indication of interest to purchase >> any loan, security or any other financial product or instrument, nor is it >> an offer to sell or a solicitation of an indication of interest to purchase >> any products or services to any persons who are prohibited from receiving >> such information under applicable law. The contents of this communication >> may not be accurate or complete and are subject to change without notice. >> As such, Orchard App, Inc. (including its subsidiaries and affiliates, >> "Orchard") makes no representation regarding the accuracy or >> completeness of the information contained herein. The intended recipient is >> advised to consult its own professional advisors, including those >> specializing in legal, tax and accounting matters. Orchard does not >> provide legal, tax or accounting advice. >> > > > > -- > Best Regards, > Haosdent Huang >
