I generally shy away from technical goals that are based on a choice of language. What is it about a go client that can't be done by extending the existing client?
> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rdelv...@binghamton.edu> wrote: > > Hi David, > > On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <da...@dmclaughlin.com> > wrote: > >> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya < >> meghdoo...@yahoo.com.invalid> wrote: >> >>> Comments inline >>> >>> From: David McLaughlin <dmclaugh...@apache.org> >>> To: dev@aurora.apache.org >>> Sent: Thursday, June 9, 2016 5:13 PM >>> Subject: Re: Golang Aurora lib, multiple executor support, integrate >>> mesos task related fields >>> >>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rdelv...@binghamton.edu> >>> wrote: >>> >>>> Hello all, >>>> >>>> I'd like to (re-)introduce myself. My name's Renan DelValle and I've >> had >>>> the pleasure of being part of the Aurora community for the last year or >>> so. >>>> >>>> Last year I worked to allow Aurora to utilize custom executors. With >> the >>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature >>> became >>>> a reality and made into Aurora late last year. (Also got to show an >> early >>>> beta at MesosCon!) >>>> >>>> For this year's summer, I have some new goals which I invite anyone to >>>> provide input on. >>>> >>>> 1. Create a Golang library that works as an alternative to Aurora's >>> Python >>>> client and Pystachio. The initial goal of this library is to support >> the >>>> most critical job related Thrift API's. (Admin operations can continue >> to >>>> be done from the Aurora CLI). Since we support custom executors, we >> need >>> a >>>> way that's not so tied to thermos (as Pystachio is) to send jobs >>>> configurations to Aurora (and by extension the custom executor). >>> >>> >>> You can easily add support for a different configuration format without >>> having to rewrite the CLI in Go. You'd do that by adding your own custom >>> noun/verbs to the client or replacing existing commands. For example, at >>> Twitter we have our own version of 'aurora update start' that plugs into >> an >>> internal Deploy Orchestration Service and we have a whole other command >> for >>> federated deploys. I can show you how we do that. >>> >>> The first thing I thought of though was - this seems like a perfect >>> starting point to get serious about a HTTP+JSON API for Aurora. Having >> that >>> would make it trivial to do a CLI in any language you want, and the >> Python >>> CLI would really only be there to do Pystachio evaluation. >>>>> CLI integrations is not useful for a lot of different reasons. We need >>> API's from other orchestration points from different components and hence >>> we would package it in a go library. Uber, I believe has taken the same >>> route (info from this mesoscon) >>> We are not writing a replacement cli tool. As noted admin operations >> would >>> heavily leverage current aurora cli. >>> I think REST work has been postponed and attempted few times in past. We >>> cannot wait for that. >> >> >> This is the dev list for contributions to the project, so I assumed we were >> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift >> has Go bindings so you're all set for whatever custom tooling you're >> building. Please use the users list for feedback on that. > > > While I'm very thankful for your input on the matter, my original text very > clearly said: > "1. Create a Golang library that works as an *alternative* to > Aurora's Python client and Pystachio." > > I believe every single item I have listed has the potential to become a > useful contribution to this project. > Therefore, I feel that it is prudent continue the discussion here (unless > others feel similarly). > > -Renan > >> >>> >>> >>>> >>>> 2. Create support for using multiple executors within a single Aurora >>>> scheduler instance. As of right now, only a single executor can exist >> for >>>> each Aurora scheduler instance. The idea is to allow the Aurora >> scheduler >>>> to support multiple executors such that they can be used alongside one >>>> another, providing flexibility for running jobs. >>> >>> Just for my own understanding - what problems are you solving with your >>> custom executor? Sorry if you've explained this to the lists before. >>> >>> I ask because I'd really like to see Aurora stop treating the executor >>> config as an opaque blob and being more opinionated. The >> Thermos/Scheduler >>> abstraction really hinders what type of user experience (UI) we can >> build, >>> and other frameworks do a much better job by being more opinionated and >>> pulling executor data into the scheduler UI. >>>>> We probably don't need to debate on this point. Executor is a first >>> class mesos citizen and time is right for aurora to have good support for >>> it.In our case, we have kubernetes like pods modeled through >> docker-compose >>> and the executor manages that. Scheduler UI main features should not get >>> bogged down or be held back for a particular executor. That feels >>> incredibly restrictive. If a special UI mode for thermos executor is >>> created, that should be fine. We do have to differentiate the scheduler >> and >>> thermos and aurora team has done a great job of not hard coupling the >> two. >> >> >>>> >>>> 3. I'd like to add support for some first class Mesos task related >>> fields. >>>> These would be optional and/or have defaults so that the regular Aurora >>> CLI >>>> does not break. One of the examples I'm interested in integrating is >> the >>>> Mesos Fetcher so that resources can be downloaded into the sandbox that >>> the >>>> custom executor may need. (The executor path will never be exposed as >>> this >>>> will be defined serverside and be static). >>> >>> >>> Wouldn't the mesos-fetcher call just be another process to be passed to >>> your executor? I have to admit I don't know enough about how the Mesos >>> fetcher works. >>>>> Apache Mesos - Fetcher >> >> >> >>> We may add other first class fields. >>> >>> >>>> >>>> If anyone has any feedback on these, I'd be very glad to hear it. >>>> >>>> That's it for me for now, thanks! >>>> >>>> -Renan >>> >>> >>> >>> >>> | >>> | >>> | >>> | | | >>> >>> | >>> >>> | >>> | >>> | | >>> Apache Mesos - Fetcher >>> | | >>> >>> | >>> >>> | >>
