Hi David, On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <da...@dmclaughlin.com> wrote:
> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya < > meghdoo...@yahoo.com.invalid> wrote: > > > Comments inline > > > > From: David McLaughlin <dmclaugh...@apache.org> > > To: dev@aurora.apache.org > > Sent: Thursday, June 9, 2016 5:13 PM > > Subject: Re: Golang Aurora lib, multiple executor support, integrate > > mesos task related fields > > > > On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rdelv...@binghamton.edu> > > wrote: > > > > > Hello all, > > > > > > I'd like to (re-)introduce myself. My name's Renan DelValle and I've > had > > > the pleasure of being part of the Aurora community for the last year or > > so. > > > > > > Last year I worked to allow Aurora to utilize custom executors. With > the > > > help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature > > became > > > a reality and made into Aurora late last year. (Also got to show an > early > > > beta at MesosCon!) > > > > > > For this year's summer, I have some new goals which I invite anyone to > > > provide input on. > > > > > > 1. Create a Golang library that works as an alternative to Aurora's > > Python > > > client and Pystachio. The initial goal of this library is to support > the > > > most critical job related Thrift API's. (Admin operations can continue > to > > > be done from the Aurora CLI). Since we support custom executors, we > need > > a > > > way that's not so tied to thermos (as Pystachio is) to send jobs > > > configurations to Aurora (and by extension the custom executor). > > > > > > > > > You can easily add support for a different configuration format without > > having to rewrite the CLI in Go. You'd do that by adding your own custom > > noun/verbs to the client or replacing existing commands. For example, at > > Twitter we have our own version of 'aurora update start' that plugs into > an > > internal Deploy Orchestration Service and we have a whole other command > for > > federated deploys. I can show you how we do that. > > > > The first thing I thought of though was - this seems like a perfect > > starting point to get serious about a HTTP+JSON API for Aurora. Having > that > > would make it trivial to do a CLI in any language you want, and the > Python > > CLI would really only be there to do Pystachio evaluation. > > >> CLI integrations is not useful for a lot of different reasons. We need > > API's from other orchestration points from different components and hence > > we would package it in a go library. Uber, I believe has taken the same > > route (info from this mesoscon) > > We are not writing a replacement cli tool. As noted admin operations > would > > heavily leverage current aurora cli. > > I think REST work has been postponed and attempted few times in past. We > > cannot wait for that. > > > > > This is the dev list for contributions to the project, so I assumed we were > discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift > has Go bindings so you're all set for whatever custom tooling you're > building. Please use the users list for feedback on that. While I'm very thankful for your input on the matter, my original text very clearly said: "1. Create a Golang library that works as an *alternative* to Aurora's Python client and Pystachio." I believe every single item I have listed has the potential to become a useful contribution to this project. Therefore, I feel that it is prudent continue the discussion here (unless others feel similarly). -Renan > > > > > > > > > > > 2. Create support for using multiple executors within a single Aurora > > > scheduler instance. As of right now, only a single executor can exist > for > > > each Aurora scheduler instance. The idea is to allow the Aurora > scheduler > > > to support multiple executors such that they can be used alongside one > > > another, providing flexibility for running jobs. > > > > > > > Just for my own understanding - what problems are you solving with your > > custom executor? Sorry if you've explained this to the lists before. > > > > I ask because I'd really like to see Aurora stop treating the executor > > config as an opaque blob and being more opinionated. The > Thermos/Scheduler > > abstraction really hinders what type of user experience (UI) we can > build, > > and other frameworks do a much better job by being more opinionated and > > pulling executor data into the scheduler UI. > > >> We probably don't need to debate on this point. Executor is a first > > class mesos citizen and time is right for aurora to have good support for > > it.In our case, we have kubernetes like pods modeled through > docker-compose > > and the executor manages that. Scheduler UI main features should not get > > bogged down or be held back for a particular executor. That feels > > incredibly restrictive. If a special UI mode for thermos executor is > > created, that should be fine. We do have to differentiate the scheduler > and > > thermos and aurora team has done a great job of not hard coupling the > two. > > > > > > > > 3. I'd like to add support for some first class Mesos task related > > fields. > > > These would be optional and/or have defaults so that the regular Aurora > > CLI > > > does not break. One of the examples I'm interested in integrating is > the > > > Mesos Fetcher so that resources can be downloaded into the sandbox that > > the > > > custom executor may need. (The executor path will never be exposed as > > this > > > will be defined serverside and be static). > > > > > > Wouldn't the mesos-fetcher call just be another process to be passed to > > your executor? I have to admit I don't know enough about how the Mesos > > fetcher works. > > >> Apache Mesos - Fetcher > > > > > We may add other first class fields. > > > > > > > > > > If anyone has any feedback on these, I'd be very glad to hear it. > > > > > > That's it for me for now, thanks! > > > > > > -Renan > > > > > > > > > > > > > | > > | > > | > > | | | > > > > | > > > > | > > | > > | | > > Apache Mesos - Fetcher > > | | > > > > | > > > > | > > > > > > >
