style95 commented on PR #5338: URL: https://github.com/apache/openwhisk/pull/5338#issuecomment-1288042098
I could successfully test this feature.  When a new version of a scheduler and an invoker are newly deployed, there is no queue and container. So when a new version of a controller is deployed, it should create a queue and a container first accordingly wait time increases. But every single activation was successfully invoked without any failure during the deployment. We can see no error happened in the above screenshot. Regarding the wait-time increase, I think it's orthogonal to this PR and we can improve it in a subsequent PR. **Controllers** ``` [2022-10-23T06:05:01.123Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] POST /api/v1/namespaces/guest/actions/noop30 blocking=true&result=true&volatile=true [2022-10-23T06:05:01.182Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [FPCPoolBalancer] posted to scheduler0[0][51090548] [2022-10-23T06:05:01.182Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [FPCPoolBalancer] Created queue successfully for guest/guest/[email protected] on scheduler0 [2022-10-23T06:05:08.430Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [FPCPoolBalancer] received completion ack for '69a85a8cd2294ea5a85a8cd229aea531', system error=false [2022-10-23T06:05:08.433Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [FPCPoolBalancer] received result ack for '69a85a8cd2294ea5a85a8cd229aea531' ``` **schedulers** ``` [2022-10-23T06:05:01.189Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [QueueManager] Got activation message 69a85a8cd2294ea5a85a8cd229aea531 for Namespace(guest,23bc46b1-71f6-4ed5-8c54-816aa4f8c502)/guest/[email protected] from kafka. [2022-10-23T06:05:01.190Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [MemoryQueue] [guest:guest/[email protected]:Running] got a new activation message 69a85a8cd2294ea5a85a8cd229aea531 [2022-10-23T06:05:01.302Z] [INFO] [#tid_sid_unknown] [SchedulingDecisionMaker] add one container if totalContainers(0) == 0 && availableMsg(1) > 0 [guest:guest/[email protected]] [2022-10-23T06:05:01.302Z] [INFO] [#tid_sid_unknown] [MemoryQueue] [guest:guest/[email protected]:Running] Try to create a new container with creationId e3182d5e23a34ed9982d5e23a3aed9f3 [2022-10-23T06:05:01.303Z] [INFO] [#tid_sid_unknown] [ContainerManager] received 1 creation message [guest:guest/[email protected]] [2022-10-23T06:05:01.306Z] [INFO] [#tid_sid_unknown] [ContainerManager] usable total invoker size: 4 [2022-10-23T06:05:01.306Z] [INFO] [#tid_sid_unknown] [ContainerManager] 1 creation messages for guest/guest/[email protected], managedFraction:1.0, blackboxFraction:1.0, managed invoker size:4, blackboxes invoker size:4 [2022-10-23T06:05:01.306Z] [INFO] [#tid_sid_unknown] [ContainerManager] [guest/guest/[email protected]] looking for managed invokers to schedule. [2022-10-23T06:05:01.309Z] [INFO] [#tid_sid_containerCreation] [ContainerManager] posted creationId: e3182d5e23a34ed9982d5e23a3aed9f3 for guest/guest/[email protected] to invoker0[0][13720] [2022-10-23T06:05:04.233Z] [INFO] [#tid_IH8hZVMxCVzB5xW7KcMsgGiSr6X81Hz9] [MemoryQueue] [guest:guest/[email protected]:Running] Get activation request 431fdb1966598b9299d4905ac6cc76a223032352eabe06b667e9166c6671126f, send one message: 69a85a8cd2294ea5a85a8cd229aea531 ``` **Invokers** ``` [2022-10-23T06:05:02.780Z] [INFO] [#tid_sid_unknown] [FunctionPullingContainerPool] received a container creation message: e3182d5e23a34ed9982d5e23a3aed9f3 [2022-10-23T06:05:02.828Z] [INFO] [#tid_sid_invokerColdstart] [DockerClientWithFileAccess] running /usr/bin/docker run -d --cpu-shares 25 --memory 256m --memory-swap 256m --network bridge -e __OW_API_HOST=https://10.106.219.110 -e __OW_ALLOW_CONCURRENT=false --name wsk00_19_guest_noop30 --cap-drop NET_RAW --cap-drop NET_ADMIN --ulimit nofile=1024:1024 --pids-limit 1024 --log-driver json-file openwhisk/action-nodejs-v14:nightly (timeout: 1 minute) [2022-10-23T06:05:04.224Z] [INFO] [#tid_sid_unknown] [WatcherService] watch endpoint: WatchEndpoint(bmt-new-110/namespace/guest/guest/noop30/15-e0e9a79b4d38b0b010f35023a7464ef6/invoker0/container/431fdb1966598b9299d4905ac6cc76a223032352eabe06b667e9166c6671126f,,false,data-management-service,Set(DeleteEvent)) [2022-10-23T06:05:04.227Z] [INFO] [#tid_sid_unknown] [FPCInvokerReactive] Posted success ack of container creation e3182d5e23a34ed9982d5e23a3aed9f3 for guest/guest/[email protected] [2022-10-23T06:05:04.240Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [FunctionPullingContainerProxy] received a message 69a85a8cd2294ea5a85a8cd229aea531 for guest/[email protected] in ClientCreated [2022-10-23T06:05:04.240Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [DockerContainer] sending initialization to ContainerId(431fdb1966598b9299d4905ac6cc76a223032352eabe06b667e9166c6671126f) ContainerAddress(172.17.0.21,8080) [2022-10-23T06:05:05.087Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [DockerContainer] initialization result: ok [2022-10-23T06:05:05.087Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [DockerContainer] sending arguments to /guest/noop30 at ContainerId(431fdb1966598b9299d4905ac6cc76a223032352eabe06b667e9166c6671126f) ContainerAddress(172.17.0.21,8080) [2022-10-23T06:05:05.096Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [DockerContainer] running result: ok [2022-10-23T06:05:05.099Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [MessagingActiveAck] posted result of activation 69a85a8cd2294ea5a85a8cd229aea531 [2022-10-23T06:05:05.100Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [MessagingActiveAck] posted completion of activation 69a85a8cd2294ea5a85a8cd229aea531 ``` As per the above logs, we can see that it took around 200ms for the controller to send a queue creation and the scheduler to send a container creation request. But it took around 1.4s for invokers to receive the creation request, it took another 1.4s for the invoker to create a container. Another thing is it took around 3.3s for the controller to actually receive the activation ack after the invoker sent the acks. I think this is still an improvement point especially regarding Kakfa messing. Anyway, I have achieved what I wanted, I would include https://github.com/apache/openwhisk/pull/5334 into this PR and update all CI tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
