I believe a lot of Tyson's observations around Mesos integration also apply to Kubernetes integration. For example, I've been experimenting with an approach that replaces the DockerContainer with a KubernetesContainer. The KubernetesContainer also ignores the pause and resume of functions and also deploys containers to arbitrary nodes in the Kubernetes cluster.
Here's a rough but working (except for log shipping) prototype of that - https://github.com/projectodd/incubator-openwhisk/blob/kube-container/core/invoker/src/main/scala/whisk/core/containerpool/kubernetes/KubernetesContainer.scala So, I'd like to add a +1 Tyson's notes around isolating docker client usage and log collection. I believe they both equally apply to a Kubernetes environment as well. And, more broadly, we'll need to figure out how to make the various Container implementations pluggable (or at least configurable) for Docker, Kubernetes, & Mesos. Ben On Wed, May 31, 2017 at 5:24 PM, Carlos Santana <[email protected]> wrote: > Hi Tyson > > Thanks for reaching out, having Mesos integration in OpenWhisk is something > I have heard some users asking about and any help to figure out how to best > integrate are very welcome. > > As you already aware OpenWhisk Invoker is a critical path component > optimized for performance, to get those container pause and unpause in > single digit milliseconds that's the reason for the recent change from > using docker engine down to runc > > Here is my initial notes, take into account I don't have first hand > experience with Mesos, which is why is good to have folks like you join :-) > > I think extending in a way that a different mode can be use would be useful > to plug a different orchestration methodology. > > For log collection we are looking into sending logs into ELK (Elastic > search) there is some early investigation going on, so I see after this > integration, mesos can be told to which ELK to send the logs, and the > Invoker to get the results. > > In terms of scala vs. nodejs, I would recommend to use scala with the java > client, since there could be some components being integrated into Invoker > and code sharing. > > How you want to follow up, maybe you have something to show us or maybe > setup a hangout call to meet > > I think Dragos from Adobe, might have a lot more feedback than me since I > think he has first hand experience > > --Carlos > > > > On Sat, May 27, 2017 at 2:44 PM Tyson Norris <[email protected]> > wrote: > >> Hi All - >> I’ve been considering how we can leverage mesos resource scheduling within >> OpenWhisk, and wanted to share some ideas and solicit some feedback. >> >> In general, I’ve been experimenting with an approach that replaces the >> DockerContainer implementation with a MesosTask. The MesosTask will have >> effectively the same lifecycle (ignoring pause/resume functions) and the >> same interface (HTTP api to the container), the main difference being that >> the container will be deployed to some arbitrary host in the cluster. >> >> There are a few broad topics around this, including: >> - docker client usage needs to be better isolated (e.g. cleaning up >> existing containers, reconciling containers already running at time of >> invoker startup, etc). I think this will be mostly straightforward. >> - log collection - I’m not sure the best approach here, but one is to >> completely decouple log collection from activation execution, and then >> provide a mesos-specific impl >> - container state tracking + load balancing - obviously there is potential >> for conflict if 2 invokers schedule activations on the same container in >> the cluster (since the state tracking of the container would still be in >> the invoker). This would imply some extension to the ContainerPool as well. >> - mesos framework - we’ve discussed internally a bit, and some preferences >> I have are: leverage the mesos http api (avoid the older jni libs if >> possible) and provide an independent framework application which provides a >> (simpler) http api that is consumed by the invoker (if mesos integration >> is enabled in the deployment). This way the framework deployment can be >> isolated from controller/invoker deployment, and interaction with docker >> containers is mostly the same (except for logging). >> - There are a few options of implementing mesos http api clients like: >> rxjava client, and nodejs client, but I've not seen any good Scala client >> to date, so we may either provide a scala app using the rxjava client, or >> create.a new client in Scala. >> >> Let me know if you have any thoughts around any of these, and I will share >> more details as they come. >> >> Thanks >> Tyson >> >> >>
