Got it. Thx Max. Renan will starting working on 2 and 3, involving and updating you all.
Thx Sent from my iPhone > On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org> wrote: > > David pinged me offline suggesting I misread the proposal for (2) as > an attempt to support multiple executors per task rather than "per > cluster". If that's the case, I am happy to change my vote for (2) to > +1. I don't see much harm (aside from possible UI inconsistencies) > from multiple executor types sharing the same cluster. > >> On Mon, Jun 13, 2016 at 9:54 AM, <meghdoo...@yahoo.com.invalid> wrote: >> I will leave Renan for a detailed response. >> >> If aurora can schedule Job A with thermos and Job B that requires a >> different executor that is a first class concept in mesos, why is there a >> objection? Renan's patch of replacing thermos executor is already in aurora. >> However right now aurora can statically configure only one executor. And so >> we want a job submission dynamically can call out the executor type. >> Please clarify your views. >> Right now the docker integration story in mesos is weak (current and also >> the new universal containerizer). We see much value in using executors like >> https://github.com/mesos/docker-compose-executor >> With aurora jobs. >> >> Points 2 and points 3 make the custom executor story stronger. (For example >> we will run both thermos and compose executor depending on job type) >> >> >> Point 1, I agree. That is outside of aurora and most likely will be a shim >> on top of generated go thrift bindings but a helper to easily test custom >> executors. >> Last time when we reviewed with aurora team, it came out current aurora cli >> is tightly coupled with thermos objects for jobs and hence keep it as is. >> >> Thx >> >> >> >> Sent from my iPhone >> >>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org> wrote: >>> >>> I am with Rick and David on this. The (1) and (2) feel like building a >>> parallel universe within Aurora without a clear justification on the >>> long-term benefits for the community. >>> >>> I am strong -1 on bringing yet another language into the Aurora >>> ecosystem without identifying the strategic upside. As David >>> suggested, any efforts towards building the first-class REST API >>> instead would go a long way and will be much appreciated by everyone. >>> >>> I also don't see a reason to support multiple executors per task. That >>> feels like re-creating thermos in a thermos-less world. >>> >>> As for (3), I'd like to see more details on the mechanics here but >>> generally positive towards supporting more TaskInfo features in >>> Aurora. >>> >>> >>>> On Sun, Jun 12, 2016 at 11:46 AM, <r...@chartbeat.com> wrote: >>>> I generally shy away from technical goals that are based on a choice of >>>> language. What is it about a go client that can't be done by extending the >>>> existing client? >>>> >>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rdelv...@binghamton.edu> >>>>> wrote: >>>>> >>>>> Hi David, >>>>> >>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <da...@dmclaughlin.com> >>>>> wrote: >>>>> >>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya < >>>>>> meghdoo...@yahoo.com.invalid> wrote: >>>>>> >>>>>>> Comments inline >>>>>>> >>>>>>> From: David McLaughlin <dmclaugh...@apache.org> >>>>>>> To: dev@aurora.apache.org >>>>>>> Sent: Thursday, June 9, 2016 5:13 PM >>>>>>> Subject: Re: Golang Aurora lib, multiple executor support, integrate >>>>>>> mesos task related fields >>>>>>> >>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rdelv...@binghamton.edu> >>>>>>> wrote: >>>>>>> >>>>>>>> Hello all, >>>>>>>> >>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and I've >>>>>> had >>>>>>>> the pleasure of being part of the Aurora community for the last year or >>>>>>> so. >>>>>>>> >>>>>>>> Last year I worked to allow Aurora to utilize custom executors. With >>>>>> the >>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature >>>>>>> became >>>>>>>> a reality and made into Aurora late last year. (Also got to show an >>>>>> early >>>>>>>> beta at MesosCon!) >>>>>>>> >>>>>>>> For this year's summer, I have some new goals which I invite anyone to >>>>>>>> provide input on. >>>>>>>> >>>>>>>> 1. Create a Golang library that works as an alternative to Aurora's >>>>>>> Python >>>>>>>> client and Pystachio. The initial goal of this library is to support >>>>>> the >>>>>>>> most critical job related Thrift API's. (Admin operations can continue >>>>>> to >>>>>>>> be done from the Aurora CLI). Since we support custom executors, we >>>>>> need >>>>>>> a >>>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs >>>>>>>> configurations to Aurora (and by extension the custom executor). >>>>>>> >>>>>>> >>>>>>> You can easily add support for a different configuration format without >>>>>>> having to rewrite the CLI in Go. You'd do that by adding your own custom >>>>>>> noun/verbs to the client or replacing existing commands. For example, at >>>>>>> Twitter we have our own version of 'aurora update start' that plugs into >>>>>> an >>>>>>> internal Deploy Orchestration Service and we have a whole other command >>>>>> for >>>>>>> federated deploys. I can show you how we do that. >>>>>>> >>>>>>> The first thing I thought of though was - this seems like a perfect >>>>>>> starting point to get serious about a HTTP+JSON API for Aurora. Having >>>>>> that >>>>>>> would make it trivial to do a CLI in any language you want, and the >>>>>> Python >>>>>>> CLI would really only be there to do Pystachio evaluation. >>>>>>>>> CLI integrations is not useful for a lot of different reasons. We need >>>>>>> API's from other orchestration points from different components and >>>>>>> hence >>>>>>> we would package it in a go library. Uber, I believe has taken the same >>>>>>> route (info from this mesoscon) >>>>>>> We are not writing a replacement cli tool. As noted admin operations >>>>>> would >>>>>>> heavily leverage current aurora cli. >>>>>>> I think REST work has been postponed and attempted few times in past. We >>>>>>> cannot wait for that. >>>>>> >>>>>> >>>>>> This is the dev list for contributions to the project, so I assumed we >>>>>> were >>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift >>>>>> has Go bindings so you're all set for whatever custom tooling you're >>>>>> building. Please use the users list for feedback on that. >>>>> >>>>> >>>>> While I'm very thankful for your input on the matter, my original text >>>>> very >>>>> clearly said: >>>>> "1. Create a Golang library that works as an *alternative* to >>>>> Aurora's Python client and Pystachio." >>>>> >>>>> I believe every single item I have listed has the potential to become a >>>>> useful contribution to this project. >>>>> Therefore, I feel that it is prudent continue the discussion here (unless >>>>> others feel similarly). >>>>> >>>>> -Renan >>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> 2. Create support for using multiple executors within a single Aurora >>>>>>>> scheduler instance. As of right now, only a single executor can exist >>>>>> for >>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora >>>>>> scheduler >>>>>>>> to support multiple executors such that they can be used alongside one >>>>>>>> another, providing flexibility for running jobs. >>>>>>> >>>>>>> Just for my own understanding - what problems are you solving with your >>>>>>> custom executor? Sorry if you've explained this to the lists before. >>>>>>> >>>>>>> I ask because I'd really like to see Aurora stop treating the executor >>>>>>> config as an opaque blob and being more opinionated. The >>>>>> Thermos/Scheduler >>>>>>> abstraction really hinders what type of user experience (UI) we can >>>>>> build, >>>>>>> and other frameworks do a much better job by being more opinionated and >>>>>>> pulling executor data into the scheduler UI. >>>>>>>>> We probably don't need to debate on this point. Executor is a first >>>>>>> class mesos citizen and time is right for aurora to have good support >>>>>>> for >>>>>>> it.In our case, we have kubernetes like pods modeled through >>>>>> docker-compose >>>>>>> and the executor manages that. Scheduler UI main features should not get >>>>>>> bogged down or be held back for a particular executor. That feels >>>>>>> incredibly restrictive. If a special UI mode for thermos executor is >>>>>>> created, that should be fine. We do have to differentiate the scheduler >>>>>> and >>>>>>> thermos and aurora team has done a great job of not hard coupling the >>>>>> two. >>>>>> >>>>>> >>>>>>>> >>>>>>>> 3. I'd like to add support for some first class Mesos task related >>>>>>> fields. >>>>>>>> These would be optional and/or have defaults so that the regular Aurora >>>>>>> CLI >>>>>>>> does not break. One of the examples I'm interested in integrating is >>>>>> the >>>>>>>> Mesos Fetcher so that resources can be downloaded into the sandbox that >>>>>>> the >>>>>>>> custom executor may need. (The executor path will never be exposed as >>>>>>> this >>>>>>>> will be defined serverside and be static). >>>>>>> >>>>>>> >>>>>>> Wouldn't the mesos-fetcher call just be another process to be passed to >>>>>>> your executor? I have to admit I don't know enough about how the Mesos >>>>>>> fetcher works. >>>>>>>>> Apache Mesos - Fetcher >>>>>> >>>>>> >>>>>> >>>>>>> We may add other first class fields. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> If anyone has any feedback on these, I'd be very glad to hear it. >>>>>>>> >>>>>>>> That's it for me for now, thanks! >>>>>>>> >>>>>>>> -Renan >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> | >>>>>>> | >>>>>>> | >>>>>>> | | | >>>>>>> >>>>>>> | >>>>>>> >>>>>>> | >>>>>>> | >>>>>>> | | >>>>>>> Apache Mesos - Fetcher >>>>>>> | | >>>>>>> >>>>>>> | >>>>>>> >>>>>>> | >>>>>>
