sven-lange-last opened a new pull request #4520: Log more diagnostic info when invoker / controller cluster status changes URL: https://github.com/apache/incubator-openwhisk/pull/4520 * Whenever invoker or controller cluster status changes, the ShardingContainerPoolBalancer updates its state. If a new invoker registers, it is added to the load balancer status with its user memory capacity and a book-keeping semaphor is created to be used for scheduling load. If the controller cluster size changes, all invoker book-keeping semaphors are updated to reflect the user memory shard size managed by each controller. * This change logs more diagnostic information when the ShardingContainerPoolBalancer updates its state due to invoker or controller cluster size changes. * The additional information helps with understanding how much user memory is available on invokers and is available on the controller's shard. ## Description * This change only introduces log message extensions and should NOT change any behaviour. * Collecting the information for log messages is in O(number of invokers). Complexity of computation is similar to what is done anyway when updating the ShardingContainerPoolBalancer state. * In a healthy OpenWhisk system, state updates should happen rarely after startup has completed so that the additional computations should not affect scheduling performance. Sample messages after three new invokers have registered: ``` [2019-06-25T10:40:06.705Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer invoker status updated. managedInvokers = 3 blackboxInvokers = 3. number of known invokers increased: new = 3, old = 0. details: invoker0: Unhealthy / 4096 MB of 4096 MB, invoker1: Offline / 4096 MB of 4096 MB, invoker2: Offline / 4096 MB of 4096 MB. [2019-06-25T10:40:06.705Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer invoker status updated. managedInvokers = 3 blackboxInvokers = 3. no update required - number of known invokers unchanged: 3. [2019-06-25T10:40:06.705Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer invoker status updated. managedInvokers = 3 blackboxInvokers = 3. no update required - number of known invokers unchanged: 3. ``` Sample message after the controller cluster with 2 controllers is up and running. The controller has adjusted its shard size per invoker from 4096 MB to 2048 MB. ``` [2019-06-25T10:40:58.348Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer cluster size changed to 2 active nodes. 3 invokers with 2048 MB average memory size - total invoker memory 6144 MB. ``` Sample message after a controller cluster split has happened. The controller adjusts its shard size per invoker from 2048 MB to 4096 MB. ``` [2019-06-25T10:43:03.268Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer cluster size changed to 1 active nodes. 3 invokers with 4096 MB average memory size - total invoker memory 12288 MB. ``` ## My changes affect the following components <!--- Select below all system components are affected by your change. --> <!--- Enter an `x` in all applicable boxes. --> - [ ] API - [ ] Controller - [ ] Message Bus (e.g., Kafka) - [x] Loadbalancer - [ ] Invoker - [ ] Intrinsic actions (e.g., sequences, conductors) - [ ] Data stores (e.g., CouchDB) - [ ] Tests - [ ] Deployment - [ ] CLI - [ ] General tooling - [ ] Documentation ## Types of changes <!--- What types of changes does your code introduce? Use `x` in all the boxes that apply: --> - [ ] Bug fix (generally a non-breaking change which closes an issue). - [x] Enhancement or new feature (adds new functionality). - [ ] Breaking change (a bug fix or enhancement which changes existing behavior). ## Checklist: <!--- Please review the points below which help you make sure you've covered all aspects of the change you're making. --> - [x] I signed an [Apache CLA](https://github.com/apache/incubator-openwhisk/blob/master/CONTRIBUTING.md). - [x] I reviewed the [style guides](https://github.com/apache/incubator-openwhisk/wiki/Contributing:-Git-guidelines#code-readiness) and followed the recommendations (Travis CI will check :). - [ ] I added tests to cover my changes. - [ ] My changes require further changes to the documentation. - [ ] I updated the documentation where necessary.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
