Which version of mesos are you using? I suspect that your resources might have been filtered out by the allocator, can you please also attach some logs for mesos master?
On Tue, Feb 2, 2016 at 11:29 PM, tommy xiao <[email protected]> wrote: > interesting on this case, please follow Shuai Lin's suggestion, provide > some logs to check > > 2016-02-02 23:05 GMT+08:00 Shuai Lin <[email protected]>: > >> Is there any warning/error message in marathon logs when it takes a long >> time to deploy/redeploy your micro service? Also worth take a look of the >> mesos slave logs. >> >> On Tue, Feb 2, 2016 at 6:55 AM, Rodrick Brown <[email protected]> >> wrote: >> >>> My cluster consist of 9 slaves server split in 1/2 for two primary >>> applications (Spark | Scala Microservices) >>> >>> - Spark - (server 1,2,3,4,8) attributes: "rack:spark" >>> - Long running Microservices (server 5,6,7,9) attributes "rack:ms" >>> >>> >>> The spark jobs run in coarse mode and the majority of them are short >>> lived they run for about ~10-15 minutes via Chronos and shutdown. They >>> start every 15 minutes about ~45 jobs. >>> >>> We do lots of deploys daily mostly to the "rack:ms" nodes where these >>> jobs are started via Marathon and run until we need to deploy a new release >>> of code. >>> >>> Recently I started noticing jobs are taking forever to restart or >>> startup like they're not receiving valid offers. >>> The cluster resources consists of the following resources I always have >>> more than enough idle resources available to bring up/down new services yet >>> I've seen one scenario where a service took almost 10 minutes to restart. >>> >>> >>> CPUs Mem >>> Total 120 456.8 GB >>> Used 53.6 140.5 GB >>> Offered 0 0 B >>> Idle 66.4 316.3 GB >>> How can I combat this delay? I'm not using roles could this be the >>> problem? >>> Chronos jobs always seem to run fine but they require much less resource >>> than my long running Scala services. >>> Here is a sample job definition for in Marathon. >>> >>> { >>> "id": "production/index-service", >>> "cmd": "env && /opt/orchard/production/index-server/bin/run_jar.sh", >>> "cpus": 1.0, >>> "mem": 4096, >>> "disk": 1000, >>> "user": "orchard", >>> "instances": 2, >>> "constraints": [ >>> [ >>> "hostname","UNIQUE" >>> ], >>> [ >>> "rack", "LIKE", "ms" >>> ] >>> ], >>> "requirePorts": true, >>> "labels": { >>> "ENV": "production", >>> "HAPROXY_GROUP": "microservice" >>> }, >>> "ports": [ >>> 31703, >>> 31803, >>> 31903 >>> ], >>> "maxLaunchDelaySeconds": 3, >>> "backoffFactor": 1.20, >>> "healthChecks": [ >>> { >>> "gracePeriodSeconds": 3, >>> "intervalSeconds": 5, >>> "maxConsecutiveFailures": 3, >>> "protocol": "TCP", >>> "portIndex": 1, >>> "timeoutSeconds": 5 >>> } >>> ], >>> "upgradeStrategy": { >>> "minimumHealthCapacity": 0.5, >>> "maximumOverCapacity": 0.2 >>> } >>> } >>> >>> Any advice appreciated thanks. >>> >>> *NOTICE TO RECIPIENTS*: This communication is confidential and intended >>> for the use of the addressee only. If you are not an intended recipient of >>> this communication, please delete it immediately and notify the sender >>> by return email. Unauthorized reading, dissemination, distribution or >>> copying of this communication is prohibited. This communication does not >>> constitute >>> an offer to sell or a solicitation of an indication of interest to purchase >>> any loan, security or any other financial product or instrument, nor is it >>> an offer to sell or a solicitation of an indication of interest to purchase >>> any products or services to any persons who are prohibited from receiving >>> such information under applicable law. The contents of this communication >>> may not be accurate or complete and are subject to change without notice. >>> As such, Orchard App, Inc. (including its subsidiaries and affiliates, >>> "Orchard") makes no representation regarding the accuracy or >>> completeness of the information contained herein. The intended recipient is >>> advised to consult its own professional advisors, including those >>> specializing in legal, tax and accounting matters. Orchard does not >>> provide legal, tax or accounting advice. >>> >> >> > > > -- > Deshi Xiao > Twitter: xds2000 > E-mail: xiaods(AT)gmail.com >
