Hi Peter, What parameters are you putting in your spark streaming configuration? What are you putting as number of executor instances and how many cores per executor are you setting in your Spark job?
Best, Khaled On Mon, Oct 15, 2018 at 9:18 PM Peter Liu <peter.p...@gmail.com> wrote: > Hi there, > > I have a system with 80 vcores and a relatively light spark streaming > workload. Overcomming the vcore resource (i.e. > 80) in the config (see (a) > below) seems to help to improve the average spark batch time (see (b) > below). > > Is there any best practice guideline on resource overcommit with cpu / > vcores, such as yarn config options, candidate cases ideal for > overcommiting vcores etc.? > > the slide below (from 2016 though) seems to address the memory overcommit > topic and hint a "future" topic on cpu overcommit: > > https://www.slideshare.net/HadoopSummit/investing-the-effects-of-overcommitting-yarn-resources > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.slideshare.net_HadoopSummit_investing-2Dthe-2Deffects-2Dof-2Dovercommitting-2Dyarn-2Dresources&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=ZCbfeVtFh_TC0b2e0fobq62qrBKhQPtyBNfMsVcVzmo&s=UXeomeHkGRlHg9Bxgb81T98oH7zj7T6OmF4dsfhK0Sg&e=> > > Would like to know if this is a reasonable config practice and why this is > not achievable without overcommit. Any help/hint would be very much > appreciated! > > Thanks! > > Peter > > (a) yarn-site.xml > <property> > <name>yarn.nodemanager.resource.cpu-vcores</name> > <value>110</value> > </property> > > <property> > <name>yarn.scheduler.maximum-allocation-vcores</name> > <value>110</value> > </property> > > > (b) > FYI: > I have a system with 80 vcores and a relatively light spark streaming > workload. overcomming the vocore resource (here 100) seems to help the > average spark batch time. need more understanding on this practice. > Skylake (1 x 900K msg/sec) total batch# (avg) avg batch time in ms (avg) avg > user cpu (%) nw read (mb/sec) > 70vocres 178.20 8154.69 n/a n/a > 80vocres 177.40 7865.44 27.85 222.31 > 100vcores 177.00 7,209.37 30.02 220.86 > >
