On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rdelv...@binghamton.edu> wrote:
> Hello all, > > I'd like to (re-)introduce myself. My name's Renan DelValle and I've had > the pleasure of being part of the Aurora community for the last year or so. > > Last year I worked to allow Aurora to utilize custom executors. With the > help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature became > a reality and made into Aurora late last year. (Also got to show an early > beta at MesosCon!) > > For this year's summer, I have some new goals which I invite anyone to > provide input on. > > 1. Create a Golang library that works as an alternative to Aurora's Python > client and Pystachio. The initial goal of this library is to support the > most critical job related Thrift API's. (Admin operations can continue to > be done from the Aurora CLI). Since we support custom executors, we need a > way that's not so tied to thermos (as Pystachio is) to send jobs > configurations to Aurora (and by extension the custom executor). > You can easily add support for a different configuration format without having to rewrite the CLI in Go. You'd do that by adding your own custom noun/verbs to the client or replacing existing commands. For example, at Twitter we have our own version of 'aurora update start' that plugs into an internal Deploy Orchestration Service and we have a whole other command for federated deploys. I can show you how we do that. The first thing I thought of though was - this seems like a perfect starting point to get serious about a HTTP+JSON API for Aurora. Having that would make it trivial to do a CLI in any language you want, and the Python CLI would really only be there to do Pystachio evaluation. > > 2. Create support for using multiple executors within a single Aurora > scheduler instance. As of right now, only a single executor can exist for > each Aurora scheduler instance. The idea is to allow the Aurora scheduler > to support multiple executors such that they can be used alongside one > another, providing flexibility for running jobs. > Just for my own understanding - what problems are you solving with your custom executor? Sorry if you've explained this to the lists before. I ask because I'd really like to see Aurora stop treating the executor config as an opaque blob and being more opinionated. The Thermos/Scheduler abstraction really hinders what type of user experience (UI) we can build, and other frameworks do a much better job by being more opinionated and pulling executor data into the scheduler UI. > > 3. I'd like to add support for some first class Mesos task related fields. > These would be optional and/or have defaults so that the regular Aurora CLI > does not break. One of the examples I'm interested in integrating is the > Mesos Fetcher so that resources can be downloaded into the sandbox that the > custom executor may need. (The executor path will never be exposed as this > will be defined serverside and be static). Wouldn't the mesos-fetcher call just be another process to be passed to your executor? I have to admit I don't know enough about how the Mesos fetcher works. > > If anyone has any feedback on these, I'd be very glad to hear it. > > That's it for me for now, thanks! > > -Renan >
