Hooking into pause/unpause/destroy of containers seems plausible, instead of hooking into the Maps in ContainerPool.
So in the existing PR, the ContainerPool uses an alternate impl for Map to store freePool and prewarmPool, and that alternate impl initiates the attach to existing containers, when it becomes active. The ContainerPool could instead potentially delegate to the ContainerFactory, e.g. a ContainerFactory.reviveContainers(childFactory) => (freePool, prewarmPool) - we will still need a way to trigger this on demand (e.g. when the standby pool becomes active, in our case, but I think that is a minor detail). I can try it out; I will be out next week, but if you test any of this in the meantime, let me know. Thanks Tyson > On Mar 30, 2018, at 9:58 AM, David P Grove <[email protected]> wrote: > > > Tyson Norris <[email protected]> wrote on 03/27/2018 06:25:59 PM: >> >> Do you have an example of the labels working? I guess the labels are >> changed over time through the lifecycle of the container? >> > > Apologies for brutally chopping the email chain; my mail client made a > horrible hash of it. > > Right now, all we are doing with Kube labels is to label each action > container with its owning invoker on startup. This lets us delete orphaned > containers if the invoker crashes and needs to be restarted. The labeling > happens at [1] and the removal of orphans using the labels at [2]. > > I think the Kube-native version of part of what you are doing with the > DistributedData for Mesos would be to add and remove additional labels to > give us the option of attaching a new invoker instance to orphaned > containers instead of just destroying them. Interacting with the > Kubernetes API server to do a labeling operation takes around 10ms, so we > couldn't do this on a truly hot path. But we could probably afford to > update container labels in parallel with pause/unpause operations, which > could enable re-attachment to any paused containers. > > --dave > > [1] > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk%2Fblob%2F0b20df0f725a671f8e51c9e8793116476fd22f76%2Fcore%2Finvoker%2Fsrc%2Fmain%2Fscala%2Fwhisk%2Fcore%2Fcontainerpool%2Fkubernetes%2FKubernetesContainerFactory.scala%23L81&data=02%7C01%7Ctnorris%40adobe.com%7C3ea96a8a416141db52b208d59660052f%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636580261502275400&sdata=6XagwDT7CnCoj1nOIHK%2B02bincKYogLkKy0vUXh8jY8%3D&reserved=0 > [2] > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk%2Fblob%2F0b20df0f725a671f8e51c9e8793116476fd22f76%2Fcore%2Finvoker%2Fsrc%2Fmain%2Fscala%2Fwhisk%2Fcore%2Fcontainerpool%2Fkubernetes%2FKubernetesContainerFactory.scala%23L57&data=02%7C01%7Ctnorris%40adobe.com%7C3ea96a8a416141db52b208d59660052f%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636580261502275400&sdata=f6VQl9UMW7gtoFheibT9opXz973hGUVmivlDJg%2FF5Co%3D&reserved=0
