Hi Tim - As you mention, running OW in a mixed-workload Mesos cluster can create some issues since the resource scheduling will be conflicting between Mesos and OW. I think this will be true of other cluster managers as well, but I’m not sure.
We are experimenting with how to allow OW to operate as a Mesos framework, and delegate container management to Mesos. There are a number of obstacles currently, such as having no way to provide an alternate mechanism for launching containers. The simplest version of this would be a system where a single invoker exists and acts as the Mesos framework, but really needs to support multiple instances with a shared cache of data to provide high availability. We are working through some ideas on how to do this, and I believe there is also some similar interest from others that are using Kubernetes to do something similar. We will be sharing our work on this with the community asap, but there are a few steps to getting to that point. Regards, Tyson > On Jul 14, 2017, at 9:08 AM, Boldt, Tim (GE Digital) <[email protected]> > wrote: > > Duy, > > Thanks for the updated PR! It contains the fixes I was looking for. > > One of the concerns I have about the current situation is that the config > assumes that OW has carte-blanche permissions in Mesos. For example, both the > registrator and invoker talk directly to the Docker socket, several of the > services use HOST networking, and most of the services reserve dedicated > ports instead of using randomly assigned ones. The owners of our shared Mesos > cluster don’t like me doing stuff like this. :-) > > Are most OW-on-Mesos users assuming they will have a dedicated Mesos cluster > for OW, or is the intention to make OW work with a locked-down, shared > cluster eventually? > > Tim > > > On 7/13/17, 3:49 AM, "Duy Nguyen" <[email protected]> wrote: > > Hi Tim, > > Great to see more interest in running OpenWhisk on Mesos. > > With the deployment structure in the PR > (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Finc&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=6SCLSLNPTVUyyZMmAZc9NoYtEpB69wSqQ3I2fU9v7UY%3D&reserved=0 > ubator-openwhisk-devtools/pull/20), we have our OW instance running on > DC/OS. Basically, we wrap each service in a DC/OS package and publish a > dedicated universe for it. A deployment means launching those services from > the pre-defined packages and apply your apigateway config to set up the > server endpoint. > > Currently one of the main limitations is with the Invoker service: it > independently manages resources for running action containers on a single > node and does not communicate back to Marathon, hence Marathon is unable to > make proper decision on allocating its resources. Some ideas to solve this > were discussed in a previous thread [1]. > > I have updated the PR which works well for us with the latest OpenWhisk > images. It is still blocked by a Jenkins job for building the > "whisk-couchdb" image (currently it's using a manually built couchdb > version). When it is ready we should be good to merge. > > Would love to hear if you have more ideas / suggestions to this approach. > > Thanks, > Duy > > [1] > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fincubator-openwhisk-dev%2F201705.mbox%2F%253C70F7A242-3BBB-46A5-A49A-61C35C77F68A%40adobe.com%253E&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=p7%2BoWRcbkYftLI6oWlLTl1WVAH4xyhbqywALjgPiLT0%3D&reserved=0 > > > On Wed, Jul 12, 2017 at 11:01 PM, Boldt, Tim (GE Digital) < > [email protected]> wrote: > >> Is anyone using (or experimenting with) OpenWhisk on Mesos? >> >> I see a 3-month old discussion >> (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2F&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=yrvpqE73Buyn42B5hupK2by07S7GzdOEySmM90c22UM%3D&reserved=0 >> incubator-openwhisk-devtools/issues/17) and a month-old PR ( >> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk-devtools%2Fpull%2F20&data=02%7C01%7C%7Cc61c763eff874cc137a408d4cad29a02%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636356453251742975&sdata=Mf0SSiYACYxW2A7nZEHABNYcjYYCuXyjAN0V1u1Yw0U%3D&reserved=0). >> However, >> when I tried it, I had to hack it quite a bit to get it to work for me. >> >> If anyone else has gotten OW working on Mesos, I’d love to hear about your >> experiences. >> >> >> > >
