Re: Propose to create a Kubernetes framework for Mesos

[Jie Yu]

+ user list as well to hear more feedback from Mesos users.

I am +1 on this proposal to create a Mesos framework that exposes k8s API,
and provide nodeless

experience to users.

Creating Mesos framework that provides k8s API is not a new idea. For
instance, the following are the two prior attempts:
1. https://github.com/kubernetes-retired/kube-mesos-framework
2. https://mesosphere.com/product/kubernetes-engine/

Both of the above solutions will run unmodified kubelets for workloads
(i.e., pods). Some users might prefer that way, and we should not preclude
that on Mesos. However, the reason this nodeless (aka, virtual kubelet)
idea got me very excited is because it provides us an opportunity to create
a truly integrated solution to bridge k8s and Mesos.

K8s gets popular for reasons. IMO, the followings are the key:
(1) API machinery. This includes API extension mechanism (CRD
),
simple-to-program client, versioning, authn/authz, etc.
(2) It expose basic scheduling primitives, and let users/vendors focus on
orchestration (i.e., Operators). In contrast, Mesos framework is
significantly harder to program due to the need for doing scheduling also.
Although we have scheduling libraries like Fenzo
, the whole community suffers from
fragmentation because there's no "default" solution.

** Why this proposal is more integrated than prior solutions?*

This is because prior solutions are more like installer for k8s. You either
need to pre-reserve resources
 for kubelet, or fork
k8s scheduler to bring up kubelet on demand
. Complexity is
definitely a concern since both systems are involved. In contrast, the
proposal propose to run k8s workloads (pods) directly on Mesos by
translating pod spec to tasks/executors in Mesos. It's just another Mesos
framework, but you can extend that framework behavior using k8s API
extension mechanism (CRD and Operator)!

** Compare to just using k8s?*

First of all, IMO, k8s is just an API spec. Any implementation that passes
conformance tests is vanilla k8s experience. I understand that by going
nodeless, some of the concepts in k8s no longer applies (e.g.,
NodeAffinity, NodeSelector). I am actually less worried about this for two
reasons: 1) Big stakeholders are behind nodeless, including Microsoft, AWS,
Alicloud, etc; 2) K8s API is evolving, and nodeless has real use cases
(e.g., in public clouds).

In fact, we can also choose to implement those k8s APIs that make the most
sense first, and maybe define our own APIs, leveraging the extensibility of
the k8s API machinery!

If we do want to compare to upstream k8s implementation, i think the main
benefit is that:
1) You can still run your existing custom Mesos frameworks as it is today,
but start to provide your users some k8s API experiences
2) Scalability. Mesos is inherently more scalable than k8s because it takes
different trade-offs. You can run multiple copies of the same frameworks
(similar to marathon on marathon) to reach large scale if the k8s framework
itself cannot scale beyond certain limit.

** Why putting this framework in Mesos repo?*

Historically, the problem with Mesos community is fragmentation. People
create different solutions for the same set of problems. Having a "blessed"
solution in the Mesos repo has the following benefits:
1) License and ownership. It's under Apache already.
2) Attract contributions. Less fragmentation.
3) Established high quality in the repository.

 What's my suggestion for next steps? 

I suggest we create a working group for this. Any other PMC that likes this
idea, please chime in here.

- Jie

On Fri, Nov 23, 2018 at 5:24 AM 张冬冬  wrote:

>
>
> 发自我的 iPhone
>
> > 在 2018年11月23日，20:37，Alex Rukletsov  写道：
> >
> > I'm in favour of the proposal, Cameron. Building a bridge between Mesos
> and
> > Kubernetes will be beneficial for both communities. Virtual kubelet
> effort
> > looks promising indeed and is definitely a worthwhile approach to build
> the
> > bridge.
> >
> > While we will need some sort of a scheduler when implementing a provider
> > for mesos, we don't need to implement and use a "default" one: a simple
> > mesos-go based scheduler will be fine for the start. We can of course
> > consider building a default scheduler, but this will significantly
> increase
> > the size of the project.
> >
> > An exercise we will have to do here is determine which parts of a
> > kubernetes task specification can be "converted" and hence launched on a
> > Mesos cluster. Once we have a working prototype we can start testing and
> > collecting data.
> >
> > Do you want to come up with a plan and maybe a more detailed proposal?
> >
> > Best,
> > Alex
>

[API WG] Meeting today, 11am PST

[Greg Mann]

Hi all,
The API working group is meeting this morning at 11am PST. We'll be
discussing some potential changes to the agent's executor API to address
agent/executor communication issues, as well as an update to the master's
operator API to mitigate problems we've seen with Amazon's ELB. Feel free
to add other items of discussion to the agenda

!

Cheers,
Greg