Hey, My team is at the stage where we're seriously considering k8s - now running mostly Marathon / Chronos (to be replaced with Rhythm <https://github.com/mlowicki/rhythm>) on Mesos. Partially because its much bigger community + the current state of Marathon which drifts more into DC/OS. Option to have solid support for k8s experience would be really beneficial since running multiple frameworks is a big plus.
I would be happy to help with it with things like prototyping on top of mesos-go or whatever will be needed. On Wed, Nov 28, 2018 at 5:38 AM Jie Yu <[email protected]> wrote: > + user list as well to hear more feedback from Mesos users. > > I am +1 on this proposal to create a Mesos framework that exposes k8s API, > and provide nodeless > <https://docs.google.com/document/d/1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms/edit> > experience to users. > > Creating Mesos framework that provides k8s API is not a new idea. For > instance, the following are the two prior attempts: > 1. https://github.com/kubernetes-retired/kube-mesos-framework > 2. https://mesosphere.com/product/kubernetes-engine/ > > Both of the above solutions will run unmodified kubelets for workloads > (i.e., pods). Some users might prefer that way, and we should not preclude > that on Mesos. However, the reason this nodeless (aka, virtual kubelet) > idea got me very excited is because it provides us an opportunity to create > a truly integrated solution to bridge k8s and Mesos. > > K8s gets popular for reasons. IMO, the followings are the key: > (1) API machinery. This includes API extension mechanism (CRD > <https://docs.google.com/document/d/1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms/edit>), > simple-to-program client, versioning, authn/authz, etc. > (2) It expose basic scheduling primitives, and let users/vendors focus on > orchestration (i.e., Operators). In contrast, Mesos framework is > significantly harder to program due to the need for doing scheduling also. > Although we have scheduling libraries like Fenzo > <https://github.com/Netflix/Fenzo>, the whole community suffers from > fragmentation because there's no "default" solution. > > ** Why this proposal is more integrated than prior solutions?* > > This is because prior solutions are more like installer for k8s. You > either need to pre-reserve resources > <https://mesosphere.com/product/kubernetes-engine/> for kubelet, or fork > k8s scheduler to bring up kubelet on demand > <https://github.com/kubernetes-retired/kube-mesos-framework>. Complexity > is definitely a concern since both systems are involved. In contrast, the > proposal propose to run k8s workloads (pods) directly on Mesos by > translating pod spec to tasks/executors in Mesos. It's just another Mesos > framework, but you can extend that framework behavior using k8s API > extension mechanism (CRD and Operator)! > > ** Compare to just using k8s?* > > First of all, IMO, k8s is just an API spec. Any implementation that passes > conformance tests is vanilla k8s experience. I understand that by going > nodeless, some of the concepts in k8s no longer applies (e.g., > NodeAffinity, NodeSelector). I am actually less worried about this for two > reasons: 1) Big stakeholders are behind nodeless, including Microsoft, AWS, > Alicloud, etc; 2) K8s API is evolving, and nodeless has real use cases > (e.g., in public clouds). > > In fact, we can also choose to implement those k8s APIs that make the most > sense first, and maybe define our own APIs, leveraging the extensibility of > the k8s API machinery! > > If we do want to compare to upstream k8s implementation, i think the main > benefit is that: > 1) You can still run your existing custom Mesos frameworks as it is today, > but start to provide your users some k8s API experiences > 2) Scalability. Mesos is inherently more scalable than k8s because it > takes different trade-offs. You can run multiple copies of the same > frameworks (similar to marathon on marathon) to reach large scale if the > k8s framework itself cannot scale beyond certain limit. > > ** Why putting this framework in Mesos repo?* > > Historically, the problem with Mesos community is fragmentation. People > create different solutions for the same set of problems. Having a "blessed" > solution in the Mesos repo has the following benefits: > 1) License and ownership. It's under Apache already. > 2) Attract contributions. Less fragmentation. > 3) Established high quality in the repository. > > **** What's my suggestion for next steps? **** > > I suggest we create a working group for this. Any other PMC that likes > this idea, please chime in here. > > - Jie > > On Fri, Nov 23, 2018 at 5:24 AM 张冬冬 <[email protected]> > wrote: > >> >> >> 发自我的 iPhone >> >> > 在 2018年11月23日，20:37，Alex Rukletsov <[email protected]> 写道： >> > >> > I'm in favour of the proposal, Cameron. Building a bridge between Mesos >> and >> > Kubernetes will be beneficial for both communities. Virtual kubelet >> effort >> > looks promising indeed and is definitely a worthwhile approach to build >> the >> > bridge. >> > >> > While we will need some sort of a scheduler when implementing a provider >> > for mesos, we don't need to implement and use a "default" one: a simple >> > mesos-go based scheduler will be fine for the start. We can of course >> > consider building a default scheduler, but this will significantly >> increase >> > the size of the project. >> > >> > An exercise we will have to do here is determine which parts of a >> > kubernetes task specification can be "converted" and hence launched on a >> > Mesos cluster. Once we have a working prototype we can start testing and >> > collecting data. >> > >> > Do you want to come up with a plan and maybe a more detailed proposal? >> > >> > Best, >> > Alex >> > -- BR, Michał Łowicki
