Massive instance rescheduling outage upon traffic spike

2018-06-18 Thread Tengfei Mu 
Hi,

We have had a few incidents when service under unexpected traffic/load
spike then container starts to respond slow/fail health check, which caused
massive instance rescheduling in Aurora. This could be a bad cycle that
instances rescheduled (being started) causing more load on other instances,
then more and more instances hammered down. Any one can share some best
practice/lessons for preventing such outage caused by dynamic rescheduling
in production cluster?


Best,
Tengfei

Re: [REPORT] Apache Aurora - June 2018

2018-06-18 Thread Jake Farrell 
+1, submitted

Thanks Renan for drafting

-Jake

On Sat, Jun 16, 2018 at 4:03 PM, Renan DelValle  wrote:

> Please find the draft report for June below, if anyone has any
> modifications or addition please let me know.
>
> Jake, feel free to submit this on the community's behalf once all
> modifications and additions are done.
>
> -Renan
>
> Apache Aurora is a stateless and fault tolerant service scheduler used to
> schedule jobs onto Apache Mesos such as long-running services, cron jobs,
> and one off tasks.
>
> Project Status
> -
> The Apache Aurora community has seen growth from new
> contributors and user activity over the last quarter. We have successfully
> released one new versions of Apache Aurora during this time: a regular
> planned release of 0.20.0.
>
> Community
> ---
> Latest Additions:
>
> * Jordan Ly was added to the PMC on Wed May 02 2018
> * Santhosh Kumar Shanmugham was added to the PMC on Wed May 02 2018
>
> Issue backlog status since last report:
>
> * Created:   10 in the last 3 months
> * Resolved: 8 in the last 3 months
>
> Mailing list activity since last report:
>
> * @dev 108 messages
> * @user20 messages
> * @reviews   397 messages
>
> Releases
> ---
> Last release:
> * Apache Aurora 0.20.0 released 4.2.2018
>

Build failed in Jenkins: aurora-packaging-nightly #1035

2018-06-18 Thread Apache Jenkins Server 
See 


--
[...truncated 13.70 MB...]
| | `-- warning@3.0.0 
| +-- invariant@2.2.4 
| +-- react-router@4.3.1 
| | +-- hoist-non-react-statics@2.5.4 
| | `-- path-to-regexp@1.7.0 
| |   `-- isarray@0.0.1 
| `-- warning@4.0.1 
+-- react-test-renderer@16.4.1 
| `-- react-is@16.4.1 
+-- sass.js@0.10.9 
+-- sassjs-loader@1.0.0 
| +-- q@1.5.1 
| `-- q-io@1.13.5 
|   +-- collections@0.2.2 
|   | `-- weak-map@1.0.0 
|   +-- mimeparse@0.1.4 
|   +-- qs@6.5.2 
|   `-- url2@0.0.0 
+-- style-loader@0.18.2 
| `-- schema-utils@0.3.0 
+-- url-loader@0.6.2 
| +-- mime@1.6.0 
| `-- schema-utils@0.3.0 
`-- webpack@2.7.0 
  +-- acorn@5.7.1 
  +-- acorn-dynamic-import@2.0.2 
  | `-- acorn@4.0.13 
  +-- ajv@4.11.8 
  +-- ajv-keywords@1.5.1 
  +-- async@2.6.1 
  +-- enhanced-resolve@3.4.1 
  +-- interpret@1.1.0 
  +-- json-loader@0.5.7 
  +-- loader-runner@2.3.0 
  +-- loader-utils@0.2.17 
  +-- memory-fs@0.4.1 
  +-- node-libs-browser@2.1.0 
  | +-- assert@1.4.1 
  | | `-- util@0.10.3 
  | |   `-- inherits@2.0.1 
  | +-- browserify-zlib@0.2.0 
  | | `-- pako@1.0.6 
  | +-- buffer@4.9.1 
  | | +-- base64-js@1.3.0 
  | | +-- ieee754@1.1.12 
  | | `-- isarray@1.0.0 
  | +-- console-browserify@1.1.0 
  | | `-- date-now@0.1.4 
  | +-- constants-browserify@1.0.0 
  | +-- crypto-browserify@3.12.0 
  | | +-- browserify-cipher@1.0.1 
  | | | +-- browserify-aes@1.2.0 
  | | | | `-- buffer-xor@1.0.3 
  | | | +-- browserify-des@1.0.1 
  | | | | `-- des.js@1.0.0 
  | | | `-- evp_bytestokey@1.0.3 
  | | +-- browserify-sign@4.0.4 
  | | | +-- bn.js@4.11.8 
  | | | +-- browserify-rsa@4.0.1 
  | | | +-- elliptic@6.4.0 
  | | | | +-- brorand@1.1.0 
  | | | | +-- hash.js@1.1.4 
  | | | | +-- hmac-drbg@1.0.1 
  | | | | +-- minimalistic-assert@1.0.1 
  | | | | `-- minimalistic-crypto-utils@1.0.1 
  | | | `-- parse-asn1@5.1.1 
  | | |   `-- asn1.js@4.10.1 
  | | +-- create-ecdh@4.0.3 
  | | +-- create-hash@1.2.0 
  | | | +-- cipher-base@1.0.4 
  | | | +-- md5.js@1.3.4 
  | | | | `-- hash-base@3.0.4 
  | | | +-- ripemd160@2.0.2 
  | | | `-- sha.js@2.4.11 
  | | +-- create-hmac@1.1.7 
  | | +-- diffie-hellman@5.0.3 
  | | | `-- miller-rabin@4.0.1 
  | | +-- pbkdf2@3.0.16 
  | | +-- public-encrypt@4.0.2 
  | | +-- randombytes@2.0.6 
  | | `-- randomfill@1.0.4 
  | +-- domain-browser@1.2.0 
  | +-- events@1.1.1 
  | +-- https-browserify@1.0.0 
  | +-- os-browserify@0.3.0 
  | +-- path-browserify@0.0.0 
  | +-- process@0.11.10 
  | +-- punycode@1.4.1 
  | +-- querystring-es3@0.2.1 
  | +-- stream-browserify@2.0.1 
  | +-- stream-http@2.8.3 
  | | +-- builtin-status-codes@3.0.0 
  | | +-- to-arraybuffer@1.0.1 
  | | `-- xtend@4.0.1 
  | +-- string_decoder@1.1.1 
  | +-- timers-browserify@2.0.10 
  | +-- tty-browserify@0.0.0 
  | +-- url@0.11.0 
  | | +-- punycode@1.3.2 
  | | `-- querystring@0.2.0 
  | +-- util@0.10.4 
  | `-- vm-browserify@0.0.4 
  |   `-- indexof@0.0.1 
  +-- supports-color@3.2.3 
  | `-- has-flag@1.0.0 
  +-- tapable@0.2.8 
  +-- uglify-js@2.8.29 
  | +-- uglify-to-browserify@1.0.2 
  | `-- yargs@3.10.0 
  |   +-- camelcase@1.2.1 
  |   +-- cliui@2.1.0 
  |   | +-- center-align@0.1.3 
  |   | | +-- align-text@0.1.4 
  |   | | | +-- kind-of@3.2.2 
  |   | | | `-- longest@1.0.1 
  |   | | `-- lazy-cache@1.0.4 
  |   | +-- right-align@0.1.3 
  |   | `-- wordwrap@0.0.2 
  |   `-- window-size@0.1.0 
  +-- watchpack@1.6.0 
  | +-- chokidar@2.0.3 
  | | +-- async-each@1.0.1 
  | | +-- braces@2.3.2 
  | | | +-- extend-shallow@2.0.1 
  | | | +-- fill-range@4.0.0 
  | | | | +-- extend-shallow@2.0.1 
  | | | | +-- is-number@3.0.0 
  | | | | | `-- kind-of@3.2.2 
  | | | | `-- to-regex-range@2.1.1 
  | | | +-- isobject@3.0.1 
  | | | +-- snapdragon-node@2.1.1 
  | | | | +-- define-property@1.0.0 
  | | | | | `-- is-descriptor@1.0.2 
  | | | | |   +-- is-accessor-descriptor@1.0.0 
  | | | | |   `-- is-data-descriptor@1.0.0 
  | | | | `-- snapdragon-util@3.0.1 
  | | | |   `-- kind-of@3.2.2 
  | | | `-- split-string@3.1.0 
  | | +-- glob-parent@3.1.0 
  | | | +-- is-glob@3.1.0 
  | | | `-- path-dirname@1.0.2 
  | | +-- is-binary-path@1.0.1 
  | | | `-- binary-extensions@1.11.0 
  | | +-- is-glob@4.0.0 
  | | | `-- is-extglob@2.1.1 
  | | +-- readdirp@2.1.0 
  | | | `-- set-immediate-shim@1.0.1 
  | | `-- upath@1.1.0 
  | `-- neo-async@2.5.1 
  +-- webpack-sources@1.1.0 
  | `-- source-map@0.6.1 
  `-- yargs@6.6.0 
+-- camelcase@3.0.0 
+-- cliui@3.2.0 
+-- os-locale@1.4.0 
+-- read-pkg-up@1.0.1 
| +-- find-up@1.1.2 
| | `-- path-exists@2.1.0 
| `-- read-pkg@1.1.0 
|   +-- load-json-file@1.1.0 
|   | `-- strip-bom@2.0.0 
|   |   `-- is-utf8@0.2.1 
|   `-- path-type@1.1.0 
+-- string-width@1.0.2 
| +-- code-point-at@1.1.0 
| `-- is-fullwidth-code-point@1.0.0 
|   `-- number-is-nan@1.0.1 
+-- which-module@1.0.0 
`-- yargs-parser@4.2.1 






<=> 71% EXECUTING