Guangya Liu, Depend on sprint festival, the detail log will share later. also i will summary the case asap.
2016-02-04 18:12 GMT+08:00 Guangya Liu <[email protected]>: > Which version of mesos are you using? I suspect that your resources might > have been filtered out by the allocator, can you please also attach some > logs for mesos master? > > On Tue, Feb 2, 2016 at 11:29 PM, tommy xiao <[email protected]> wrote: > >> interesting on this case, please follow Shuai Lin's suggestion, provide >> some logs to check >> >> 2016-02-02 23:05 GMT+08:00 Shuai Lin <[email protected]>: >> >>> Is there any warning/error message in marathon logs when it takes a long >>> time to deploy/redeploy your micro service? Also worth take a look of the >>> mesos slave logs. >>> >>> On Tue, Feb 2, 2016 at 6:55 AM, Rodrick Brown <[email protected]> >>> wrote: >>> >>>> My cluster consist of 9 slaves server split in 1/2 for two primary >>>> applications (Spark | Scala Microservices) >>>> >>>> - Spark - (server 1,2,3,4,8) attributes: "rack:spark" >>>> - Long running Microservices (server 5,6,7,9) attributes "rack:ms" >>>> >>>> >>>> The spark jobs run in coarse mode and the majority of them are short >>>> lived they run for about ~10-15 minutes via Chronos and shutdown. They >>>> start every 15 minutes about ~45 jobs. >>>> >>>> We do lots of deploys daily mostly to the "rack:ms" nodes where these >>>> jobs are started via Marathon and run until we need to deploy a new release >>>> of code. >>>> >>>> Recently I started noticing jobs are taking forever to restart or >>>> startup like they're not receiving valid offers. >>>> The cluster resources consists of the following resources I always have >>>> more than enough idle resources available to bring up/down new services yet >>>> I've seen one scenario where a service took almost 10 minutes to restart. >>>> >>>> >>>> CPUs Mem >>>> Total 120 456.8 GB >>>> Used 53.6 140.5 GB >>>> Offered 0 0 B >>>> Idle 66.4 316.3 GB >>>> How can I combat this delay? I'm not using roles could this be the >>>> problem? >>>> Chronos jobs always seem to run fine but they require much less >>>> resource than my long running Scala services. >>>> Here is a sample job definition for in Marathon. >>>> >>>> { >>>> "id": "production/index-service", >>>> "cmd": "env && /opt/orchard/production/index-server/bin/run_jar.sh", >>>> "cpus": 1.0, >>>> "mem": 4096, >>>> "disk": 1000, >>>> "user": "orchard", >>>> "instances": 2, >>>> "constraints": [ >>>> [ >>>> "hostname","UNIQUE" >>>> ], >>>> [ >>>> "rack", "LIKE", "ms" >>>> ] >>>> ], >>>> "requirePorts": true, >>>> "labels": { >>>> "ENV": "production", >>>> "HAPROXY_GROUP": "microservice" >>>> }, >>>> "ports": [ >>>> 31703, >>>> 31803, >>>> 31903 >>>> ], >>>> "maxLaunchDelaySeconds": 3, >>>> "backoffFactor": 1.20, >>>> "healthChecks": [ >>>> { >>>> "gracePeriodSeconds": 3, >>>> "intervalSeconds": 5, >>>> "maxConsecutiveFailures": 3, >>>> "protocol": "TCP", >>>> "portIndex": 1, >>>> "timeoutSeconds": 5 >>>> } >>>> ], >>>> "upgradeStrategy": { >>>> "minimumHealthCapacity": 0.5, >>>> "maximumOverCapacity": 0.2 >>>> } >>>> } >>>> >>>> Any advice appreciated thanks. >>>> >>>> *NOTICE TO RECIPIENTS*: This communication is confidential and >>>> intended for the use of the addressee only. If you are not an intended >>>> recipient of this communication, please delete it immediately and >>>> notify the sender by return email. Unauthorized reading, dissemination, >>>> distribution or copying of this communication is prohibited. This >>>> communication does not constitute an offer to sell or a solicitation >>>> of an indication of interest to purchase any loan, security or any other >>>> financial product or instrument, nor is it an offer to sell or a >>>> solicitation of an indication of interest to purchase any products or >>>> services to any persons who are prohibited from receiving such information >>>> under applicable law. The contents of this communication may not >>>> be accurate or complete and are subject to change without notice. As such, >>>> Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") >>>> makes no representation regarding the accuracy or completeness of the >>>> information contained herein. The intended recipient is advised to consult >>>> its own professional advisors, including those specializing in legal, tax >>>> and accounting matters. Orchard does not provide legal, tax or >>>> accounting advice. >>>> >>> >>> >> >> >> -- >> Deshi Xiao >> Twitter: xds2000 >> E-mail: xiaods(AT)gmail.com >> > > -- Deshi Xiao Twitter: xds2000 E-mail: xiaods(AT)gmail.com
