Hi Ben, this looks great! Any feedback on the container abstractions?
I agree the ContainerPool takes a couple of assumptions as of today, I'd be greatly interested in reducing those and support using orchestrator APIs just as you're trying to do. Great work :) Cheers, Markus Von meinem iPhone gesendet > Am 15.07.2017 um 15:50 schrieb Ben Browning <[email protected]>: > > I'm one of those that desires an alternative implementation to launching > containers on Kubernetes, as Tyson mentioned. Instead of talking to Docker > directly we're exploring talking to the Kubernetes API so that resource > scheduling is still handled by the Kubernetes scheduler and we only have a > couple of Invoker instances for high availability. > > The route we have right now is an alternative Container implementation ( > https://github.com/projectodd/incubator-openwhisk/blob/3772c344dbfa421bf361631f96f00f7876eb2adf/core/invoker/src/main/scala/whisk/core/containerpool/kubernetes/KubernetesContainer.scala) > that we're prototyping. That implementation is not ready for production use > by any means and has some serious performance issues we'll need to address. > But, it works enough that we can explore the tradeoffs when talking > directly to Docker versus talking to the Kubernetes API and letting > Kubernetes schedule function runtimes. > > This may or may not be the right approach to take on Kubernetes and Mesos. > The load-balancing done in the controller and the container pooling done in > the Invokers today really assume each Invoker manages an independent set of > Docker containers on separate hardware versus Invokers managing independent > sets of Docker containers on a shared resource, such as the Kubernetes > cluster. > > Ben > > > On Fri, Jul 14, 2017 at 1:07 PM, Tyson Norris <[email protected]> > wrote: > >> Hi Tim - >> As you mention, running OW in a mixed-workload Mesos cluster can create >> some issues since the resource scheduling will be conflicting between Mesos >> and OW. I think this will be true of other cluster managers as well, but >> I’m not sure. >> >> We are experimenting with how to allow OW to operate as a Mesos framework, >> and delegate container management to Mesos. There are a number of obstacles >> currently, such as having no way to provide an alternate mechanism for >> launching containers. The simplest version of this would be a system where >> a single invoker exists and acts as the Mesos framework, but really needs >> to support multiple instances with a shared cache of data to provide high >> availability. We are working through some ideas on how to do this, and I >> believe there is also some similar interest from others that are using >> Kubernetes to do something similar. We will be sharing our work on this >> with the community asap, but there are a few steps to getting to that point. >> >> Regards, >> Tyson >> >> >>> On Jul 14, 2017, at 9:08 AM, Boldt, Tim (GE Digital) < >> [email protected]> wrote: >>> >>> Duy, >>> >>> Thanks for the updated PR! It contains the fixes I was looking for. >>> >>> One of the concerns I have about the current situation is that the >> config assumes that OW has carte-blanche permissions in Mesos. For example, >> both the registrator and invoker talk directly to the Docker socket, >> several of the services use HOST networking, and most of the services >> reserve dedicated ports instead of using randomly assigned ones. The owners >> of our shared Mesos cluster don’t like me doing stuff like this. :-) >>> >>> Are most OW-on-Mesos users assuming they will have a dedicated Mesos >> cluster for OW, or is the intention to make OW work with a locked-down, >> shared cluster eventually? >>> >>> Tim >>> >>> >>> On 7/13/17, 3:49 AM, "Duy Nguyen" <[email protected]> wrote: >>> >>> Hi Tim, >>> >>> Great to see more interest in running OpenWhisk on Mesos. >>> >>> With the deployment structure in the PR (https://na01.safelinks. >> protection.outlook.com/?url=https%3A%2F%2Fgithub.com% >> 2Fapache%2Finc&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02% >> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata= >> 6SCLSLNPTVUyyZMmAZc9NoYtEpB69wSqQ3I2fU9v7UY%3D&reserved=0 >>> ubator-openwhisk-devtools/pull/20), we have our OW instance running >> on >>> DC/OS. Basically, we wrap each service in a DC/OS package and publish >> a >>> dedicated universe for it. A deployment means launching those >> services from >>> the pre-defined packages and apply your apigateway config to set up >> the >>> server endpoint. >>> >>> Currently one of the main limitations is with the Invoker service: it >>> independently manages resources for running action containers on a >> single >>> node and does not communicate back to Marathon, hence Marathon is >> unable to >>> make proper decision on allocating its resources. Some ideas to solve >> this >>> were discussed in a previous thread [1]. >>> >>> I have updated the PR which works well for us with the latest >> OpenWhisk >>> images. It is still blocked by a Jenkins job for building the >>> "whisk-couchdb" image (currently it's using a manually built couchdb >>> version). When it is ready we should be good to merge. >>> >>> Would love to hear if you have more ideas / suggestions to this >> approach. >>> >>> Thanks, >>> Duy >>> >>> [1] >>> https://na01.safelinks.protection.outlook.com/?url= >> https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox% >> 2Fincubator-openwhisk-dev%2F201705.mbox%2F%253C70F7A242- >> 3BBB-46A5-A49A-61C35C77F68A%40adobe.com%253E&data=02%7C01%7C% >> 7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178de >> cee1%7C0%7C0%7C636356453251742975&sdata=p7%2BoWRcbkYftLI6oWlLTl1WVAH4xyhb >> qywALjgPiLT0%3D&reserved=0 >>> >>> >>> On Wed, Jul 12, 2017 at 11:01 PM, Boldt, Tim (GE Digital) < >>> [email protected]> wrote: >>> >>>> Is anyone using (or experimenting with) OpenWhisk on Mesos? >>>> >>>> I see a 3-month old discussion (https://na01.safelinks. >> protection.outlook.com/?url=https%3A%2F%2Fgithub.com% >> 2Fapache%2F&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02% >> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata= >> yrvpqE73Buyn42B5hupK2by07S7GzdOEySmM90c22UM%3D&reserved=0 >>>> incubator-openwhisk-devtools/issues/17) and a month-old PR ( >>>> https://na01.safelinks.protection.outlook.com/?url= >> https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk-devtools%2Fpull% >> 2F20&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02% >> 7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata= >> Mf0SSiYACYxW2A7nZEHABNYcjYYCuXyjAN0V1u1Yw0U%3D&reserved=0). However, >>>> when I tried it, I had to hack it quite a bit to get it to work for me. >>>> >>>> If anyone else has gotten OW working on Mesos, I’d love to hear about >> your >>>> experiences. >>>> >>>> >>>> >>> >>> >> >>
