Yeah I agree between Mesos and Kubernetes work there should be some common integration extension points that would serve both use cases to some extent
On Wed, May 31, 2017 at 5:42 PM Ben Browning <[email protected]> wrote: > I believe a lot of Tyson's observations around Mesos integration also > apply to Kubernetes integration. For example, I've been experimenting > with an approach that replaces the DockerContainer with a > KubernetesContainer. The KubernetesContainer also ignores the pause > and resume of functions and also deploys containers to arbitrary nodes > in the Kubernetes cluster. > > Here's a rough but working (except for log shipping) prototype of that > - > https://github.com/projectodd/incubator-openwhisk/blob/kube-container/core/invoker/src/main/scala/whisk/core/containerpool/kubernetes/KubernetesContainer.scala > > > So, I'd like to add a +1 Tyson's notes around isolating docker client > usage and log collection. I believe they both equally apply to a > Kubernetes environment as well. And, more broadly, we'll need to > figure out how to make the various Container implementations pluggable > (or at least configurable) for Docker, Kubernetes, & Mesos. > > Ben > > > On Wed, May 31, 2017 at 5:24 PM, Carlos Santana <[email protected]> > wrote: > > Hi Tyson > > > > Thanks for reaching out, having Mesos integration in OpenWhisk is > something > > I have heard some users asking about and any help to figure out how to > best > > integrate are very welcome. > > > > As you already aware OpenWhisk Invoker is a critical path component > > optimized for performance, to get those container pause and unpause in > > single digit milliseconds that's the reason for the recent change from > > using docker engine down to runc > > > > Here is my initial notes, take into account I don't have first hand > > experience with Mesos, which is why is good to have folks like you join > :-) > > > > I think extending in a way that a different mode can be use would be > useful > > to plug a different orchestration methodology. > > > > For log collection we are looking into sending logs into ELK (Elastic > > search) there is some early investigation going on, so I see after this > > integration, mesos can be told to which ELK to send the logs, and the > > Invoker to get the results. > > > > In terms of scala vs. nodejs, I would recommend to use scala with the > java > > client, since there could be some components being integrated into > Invoker > > and code sharing. > > > > How you want to follow up, maybe you have something to show us or maybe > > setup a hangout call to meet > > > > I think Dragos from Adobe, might have a lot more feedback than me since I > > think he has first hand experience > > > > --Carlos > > > > > > > > On Sat, May 27, 2017 at 2:44 PM Tyson Norris <[email protected]> > > wrote: > > > >> Hi All - > >> I’ve been considering how we can leverage mesos resource scheduling > within > >> OpenWhisk, and wanted to share some ideas and solicit some feedback. > >> > >> In general, I’ve been experimenting with an approach that replaces the > >> DockerContainer implementation with a MesosTask. The MesosTask will have > >> effectively the same lifecycle (ignoring pause/resume functions) and the > >> same interface (HTTP api to the container), the main difference being > that > >> the container will be deployed to some arbitrary host in the cluster. > >> > >> There are a few broad topics around this, including: > >> - docker client usage needs to be better isolated (e.g. cleaning up > >> existing containers, reconciling containers already running at time of > >> invoker startup, etc). I think this will be mostly straightforward. > >> - log collection - I’m not sure the best approach here, but one is to > >> completely decouple log collection from activation execution, and then > >> provide a mesos-specific impl > >> - container state tracking + load balancing - obviously there is > potential > >> for conflict if 2 invokers schedule activations on the same container in > >> the cluster (since the state tracking of the container would still be in > >> the invoker). This would imply some extension to the ContainerPool as > well. > >> - mesos framework - we’ve discussed internally a bit, and some > preferences > >> I have are: leverage the mesos http api (avoid the older jni libs if > >> possible) and provide an independent framework application which > provides a > >> (simpler) http api that is consumed by the invoker (if mesos > integration > >> is enabled in the deployment). This way the framework deployment can be > >> isolated from controller/invoker deployment, and interaction with docker > >> containers is mostly the same (except for logging). > >> - There are a few options of implementing mesos http api clients like: > >> rxjava client, and nodejs client, but I've not seen any good Scala > client > >> to date, so we may either provide a scala app using the rxjava client, > or > >> create.a new client in Scala. > >> > >> Let me know if you have any thoughts around any of these, and I will > share > >> more details as they come. > >> > >> Thanks > >> Tyson > >> > >> > >> >
