Thanks, also +1 to supporting points 2 and 3. On Mon, Jun 13, 2016 at 10:33 AM, <meghdoo...@yahoo.com.invalid> wrote:
> Got it. Thx Max. > Renan will starting working on 2 and 3, involving and updating you all. > > Thx > > Sent from my iPhone > > > On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org> > wrote: > > > > David pinged me offline suggesting I misread the proposal for (2) as > > an attempt to support multiple executors per task rather than "per > > cluster". If that's the case, I am happy to change my vote for (2) to > > +1. I don't see much harm (aside from possible UI inconsistencies) > > from multiple executor types sharing the same cluster. > > > >> On Mon, Jun 13, 2016 at 9:54 AM, <meghdoo...@yahoo.com.invalid> wrote: > >> I will leave Renan for a detailed response. > >> > >> If aurora can schedule Job A with thermos and Job B that requires a > different executor that is a first class concept in mesos, why is there a > objection? Renan's patch of replacing thermos executor is already in > aurora. However right now aurora can statically configure only one > executor. And so we want a job submission dynamically can call out the > executor type. > >> Please clarify your views. > >> Right now the docker integration story in mesos is weak (current and > also the new universal containerizer). We see much value in using executors > like > >> https://github.com/mesos/docker-compose-executor > >> With aurora jobs. > >> > >> Points 2 and points 3 make the custom executor story stronger. (For > example we will run both thermos and compose executor depending on job type) > >> > >> > >> Point 1, I agree. That is outside of aurora and most likely will be a > shim on top of generated go thrift bindings but a helper to easily test > custom executors. > >> Last time when we reviewed with aurora team, it came out current aurora > cli is tightly coupled with thermos objects for jobs and hence keep it as > is. > >> > >> Thx > >> > >> > >> > >> Sent from my iPhone > >> > >>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org> > wrote: > >>> > >>> I am with Rick and David on this. The (1) and (2) feel like building a > >>> parallel universe within Aurora without a clear justification on the > >>> long-term benefits for the community. > >>> > >>> I am strong -1 on bringing yet another language into the Aurora > >>> ecosystem without identifying the strategic upside. As David > >>> suggested, any efforts towards building the first-class REST API > >>> instead would go a long way and will be much appreciated by everyone. > >>> > >>> I also don't see a reason to support multiple executors per task. That > >>> feels like re-creating thermos in a thermos-less world. > >>> > >>> As for (3), I'd like to see more details on the mechanics here but > >>> generally positive towards supporting more TaskInfo features in > >>> Aurora. > >>> > >>> > >>>> On Sun, Jun 12, 2016 at 11:46 AM, <r...@chartbeat.com> wrote: > >>>> I generally shy away from technical goals that are based on a choice > of language. What is it about a go client that can't be done by extending > the existing client? > >>>> > >>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rdelv...@binghamton.edu> > wrote: > >>>>> > >>>>> Hi David, > >>>>> > >>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin < > da...@dmclaughlin.com> > >>>>> wrote: > >>>>> > >>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya < > >>>>>> meghdoo...@yahoo.com.invalid> wrote: > >>>>>> > >>>>>>> Comments inline > >>>>>>> > >>>>>>> From: David McLaughlin <dmclaugh...@apache.org> > >>>>>>> To: dev@aurora.apache.org > >>>>>>> Sent: Thursday, June 9, 2016 5:13 PM > >>>>>>> Subject: Re: Golang Aurora lib, multiple executor support, > integrate > >>>>>>> mesos task related fields > >>>>>>> > >>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle < > rdelv...@binghamton.edu> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Hello all, > >>>>>>>> > >>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and > I've > >>>>>> had > >>>>>>>> the pleasure of being part of the Aurora community for the last > year or > >>>>>>> so. > >>>>>>>> > >>>>>>>> Last year I worked to allow Aurora to utilize custom executors. > With > >>>>>> the > >>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that > feature > >>>>>>> became > >>>>>>>> a reality and made into Aurora late last year. (Also got to show > an > >>>>>> early > >>>>>>>> beta at MesosCon!) > >>>>>>>> > >>>>>>>> For this year's summer, I have some new goals which I invite > anyone to > >>>>>>>> provide input on. > >>>>>>>> > >>>>>>>> 1. Create a Golang library that works as an alternative to > Aurora's > >>>>>>> Python > >>>>>>>> client and Pystachio. The initial goal of this library is to > support > >>>>>> the > >>>>>>>> most critical job related Thrift API's. (Admin operations can > continue > >>>>>> to > >>>>>>>> be done from the Aurora CLI). Since we support custom executors, > we > >>>>>> need > >>>>>>> a > >>>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs > >>>>>>>> configurations to Aurora (and by extension the custom executor). > >>>>>>> > >>>>>>> > >>>>>>> You can easily add support for a different configuration format > without > >>>>>>> having to rewrite the CLI in Go. You'd do that by adding your own > custom > >>>>>>> noun/verbs to the client or replacing existing commands. For > example, at > >>>>>>> Twitter we have our own version of 'aurora update start' that > plugs into > >>>>>> an > >>>>>>> internal Deploy Orchestration Service and we have a whole other > command > >>>>>> for > >>>>>>> federated deploys. I can show you how we do that. > >>>>>>> > >>>>>>> The first thing I thought of though was - this seems like a perfect > >>>>>>> starting point to get serious about a HTTP+JSON API for Aurora. > Having > >>>>>> that > >>>>>>> would make it trivial to do a CLI in any language you want, and the > >>>>>> Python > >>>>>>> CLI would really only be there to do Pystachio evaluation. > >>>>>>>>> CLI integrations is not useful for a lot of different reasons. > We need > >>>>>>> API's from other orchestration points from different components > and hence > >>>>>>> we would package it in a go library. Uber, I believe has taken the > same > >>>>>>> route (info from this mesoscon) > >>>>>>> We are not writing a replacement cli tool. As noted admin > operations > >>>>>> would > >>>>>>> heavily leverage current aurora cli. > >>>>>>> I think REST work has been postponed and attempted few times in > past. We > >>>>>>> cannot wait for that. > >>>>>> > >>>>>> > >>>>>> This is the dev list for contributions to the project, so I assumed > we were > >>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and > Thrift > >>>>>> has Go bindings so you're all set for whatever custom tooling you're > >>>>>> building. Please use the users list for feedback on that. > >>>>> > >>>>> > >>>>> While I'm very thankful for your input on the matter, my original > text very > >>>>> clearly said: > >>>>> "1. Create a Golang library that works as an *alternative* to > >>>>> Aurora's Python client and Pystachio." > >>>>> > >>>>> I believe every single item I have listed has the potential to > become a > >>>>> useful contribution to this project. > >>>>> Therefore, I feel that it is prudent continue the discussion here > (unless > >>>>> others feel similarly). > >>>>> > >>>>> -Renan > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> 2. Create support for using multiple executors within a single > Aurora > >>>>>>>> scheduler instance. As of right now, only a single executor can > exist > >>>>>> for > >>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora > >>>>>> scheduler > >>>>>>>> to support multiple executors such that they can be used > alongside one > >>>>>>>> another, providing flexibility for running jobs. > >>>>>>> > >>>>>>> Just for my own understanding - what problems are you solving with > your > >>>>>>> custom executor? Sorry if you've explained this to the lists > before. > >>>>>>> > >>>>>>> I ask because I'd really like to see Aurora stop treating the > executor > >>>>>>> config as an opaque blob and being more opinionated. The > >>>>>> Thermos/Scheduler > >>>>>>> abstraction really hinders what type of user experience (UI) we can > >>>>>> build, > >>>>>>> and other frameworks do a much better job by being more > opinionated and > >>>>>>> pulling executor data into the scheduler UI. > >>>>>>>>> We probably don't need to debate on this point. Executor is a > first > >>>>>>> class mesos citizen and time is right for aurora to have good > support for > >>>>>>> it.In our case, we have kubernetes like pods modeled through > >>>>>> docker-compose > >>>>>>> and the executor manages that. Scheduler UI main features should > not get > >>>>>>> bogged down or be held back for a particular executor. That feels > >>>>>>> incredibly restrictive. If a special UI mode for thermos executor > is > >>>>>>> created, that should be fine. We do have to differentiate the > scheduler > >>>>>> and > >>>>>>> thermos and aurora team has done a great job of not hard coupling > the > >>>>>> two. > >>>>>> > >>>>>> > >>>>>>>> > >>>>>>>> 3. I'd like to add support for some first class Mesos task related > >>>>>>> fields. > >>>>>>>> These would be optional and/or have defaults so that the regular > Aurora > >>>>>>> CLI > >>>>>>>> does not break. One of the examples I'm interested in integrating > is > >>>>>> the > >>>>>>>> Mesos Fetcher so that resources can be downloaded into the > sandbox that > >>>>>>> the > >>>>>>>> custom executor may need. (The executor path will never be > exposed as > >>>>>>> this > >>>>>>>> will be defined serverside and be static). > >>>>>>> > >>>>>>> > >>>>>>> Wouldn't the mesos-fetcher call just be another process to be > passed to > >>>>>>> your executor? I have to admit I don't know enough about how the > Mesos > >>>>>>> fetcher works. > >>>>>>>>> Apache Mesos - Fetcher > >>>>>> > >>>>>> > >>>>>> > >>>>>>> We may add other first class fields. > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> If anyone has any feedback on these, I'd be very glad to hear it. > >>>>>>>> > >>>>>>>> That's it for me for now, thanks! > >>>>>>>> > >>>>>>>> -Renan > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> | > >>>>>>> | > >>>>>>> | > >>>>>>> | | | > >>>>>>> > >>>>>>> | > >>>>>>> > >>>>>>> | > >>>>>>> | > >>>>>>> | | > >>>>>>> Apache Mesos - Fetcher > >>>>>>> | | > >>>>>>> > >>>>>>> | > >>>>>>> > >>>>>>> | > >>>>>> >
