Thanks. One thing I think will help is isolating the http contract with containers from the management (start/stop etc) of the container. (Currently the http api client is built into the DockerContainer class)
After that, providing an interface for a configured container management would be able to dictate the approach to use in a particular env. Note that this would not preclude using the “docker” container management within a mesos or kubernetes env, if people wanted to still do that (there will be tradeoffs, like start/stop time vs cluster-level pool visibility). RE: mesos details:I have some concerns with using existing java libs from Scala - the old libs all use libmesos.so, which can be a nuisance to deal with, and the newer libs rely on some java method overloading which brings up some compiler issues in Scala. I haven’t exhaustively looked at it, but currently I’m testing out the mesos-rxjava lib. If anyone has opinions or experience with this lib, let me know. (I work with Dragos BTW, so will be soliciting feedback from him regularly) Thanks Tyson > On May 31, 2017, at 3:19 PM, Carlos Santana <[email protected]> wrote: > > Yeah I agree between Mesos and Kubernetes work there should be some common > integration extension points that would serve both use cases to some extent > > > > On Wed, May 31, 2017 at 5:42 PM Ben Browning <[email protected]> wrote: > >> I believe a lot of Tyson's observations around Mesos integration also >> apply to Kubernetes integration. For example, I've been experimenting >> with an approach that replaces the DockerContainer with a >> KubernetesContainer. The KubernetesContainer also ignores the pause >> and resume of functions and also deploys containers to arbitrary nodes >> in the Kubernetes cluster. >> >> Here's a rough but working (except for log shipping) prototype of that >> - >> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprojectodd%2Fincubator-openwhisk%2Fblob%2Fkube-container%2Fcore%2Finvoker%2Fsrc%2Fmain%2Fscala%2Fwhisk%2Fcore%2Fcontainerpool%2Fkubernetes%2FKubernetesContainer.scala&data=02%7C01%7C%7C869495e5dcfb4facf10d08d4a8732801%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636318659922108109&sdata=K%2BAKqn87gawixF2GdofV5ik3ETosndqPSbhVO5mPfBY%3D&reserved=0 >> >> >> So, I'd like to add a +1 Tyson's notes around isolating docker client >> usage and log collection. I believe they both equally apply to a >> Kubernetes environment as well. And, more broadly, we'll need to >> figure out how to make the various Container implementations pluggable >> (or at least configurable) for Docker, Kubernetes, & Mesos. >> >> Ben >> >> >> On Wed, May 31, 2017 at 5:24 PM, Carlos Santana <[email protected]> >> wrote: >>> Hi Tyson >>> >>> Thanks for reaching out, having Mesos integration in OpenWhisk is >> something >>> I have heard some users asking about and any help to figure out how to >> best >>> integrate are very welcome. >>> >>> As you already aware OpenWhisk Invoker is a critical path component >>> optimized for performance, to get those container pause and unpause in >>> single digit milliseconds that's the reason for the recent change from >>> using docker engine down to runc >>> >>> Here is my initial notes, take into account I don't have first hand >>> experience with Mesos, which is why is good to have folks like you join >> :-) >>> >>> I think extending in a way that a different mode can be use would be >> useful >>> to plug a different orchestration methodology. >>> >>> For log collection we are looking into sending logs into ELK (Elastic >>> search) there is some early investigation going on, so I see after this >>> integration, mesos can be told to which ELK to send the logs, and the >>> Invoker to get the results. >>> >>> In terms of scala vs. nodejs, I would recommend to use scala with the >> java >>> client, since there could be some components being integrated into >> Invoker >>> and code sharing. >>> >>> How you want to follow up, maybe you have something to show us or maybe >>> setup a hangout call to meet >>> >>> I think Dragos from Adobe, might have a lot more feedback than me since I >>> think he has first hand experience >>> >>> --Carlos >>> >>> >>> >>> On Sat, May 27, 2017 at 2:44 PM Tyson Norris <[email protected]> >>> wrote: >>> >>>> Hi All - >>>> I’ve been considering how we can leverage mesos resource scheduling >> within >>>> OpenWhisk, and wanted to share some ideas and solicit some feedback. >>>> >>>> In general, I’ve been experimenting with an approach that replaces the >>>> DockerContainer implementation with a MesosTask. The MesosTask will have >>>> effectively the same lifecycle (ignoring pause/resume functions) and the >>>> same interface (HTTP api to the container), the main difference being >> that >>>> the container will be deployed to some arbitrary host in the cluster. >>>> >>>> There are a few broad topics around this, including: >>>> - docker client usage needs to be better isolated (e.g. cleaning up >>>> existing containers, reconciling containers already running at time of >>>> invoker startup, etc). I think this will be mostly straightforward. >>>> - log collection - I’m not sure the best approach here, but one is to >>>> completely decouple log collection from activation execution, and then >>>> provide a mesos-specific impl >>>> - container state tracking + load balancing - obviously there is >> potential >>>> for conflict if 2 invokers schedule activations on the same container in >>>> the cluster (since the state tracking of the container would still be in >>>> the invoker). This would imply some extension to the ContainerPool as >> well. >>>> - mesos framework - we’ve discussed internally a bit, and some >> preferences >>>> I have are: leverage the mesos http api (avoid the older jni libs if >>>> possible) and provide an independent framework application which >> provides a >>>> (simpler) http api that is consumed by the invoker (if mesos >> integration >>>> is enabled in the deployment). This way the framework deployment can be >>>> isolated from controller/invoker deployment, and interaction with docker >>>> containers is mostly the same (except for logging). >>>> - There are a few options of implementing mesos http api clients like: >>>> rxjava client, and nodejs client, but I've not seen any good Scala >> client >>>> to date, so we may either provide a scala app using the rxjava client, >> or >>>> create.a new client in Scala. >>>> >>>> Let me know if you have any thoughts around any of these, and I will >> share >>>> more details as they come. >>>> >>>> Thanks >>>> Tyson >>>> >>>> >>>> >>
