Expect that to be up by mid next week and put in 0.16 docs. Sent from my iPhone
> On Aug 5, 2016, at 9:23 AM, Zhitao Li <zhitaoli...@gmail.com> wrote: > > This sounds awesome! > > Is there a document for how to use the feature? > > On Fri, Aug 5, 2016 at 12:35 AM, meghdoot bhattacharya < > meghdoo...@yahoo.com.invalid> wrote: > >> Renaming the subject. >> Renan's latest patch in 0.16 has completed the goal of implementing both 2 >> and 3. So, Aurora now can not only support custom executors but run >> multiple executors in the cluster!! Mesos fetcher has been integrated as >> well to help the custom executors have access to downloaded artifacts that >> it needs. We are going to add dedicated documentation on how to enable this >> in 0.16. >> This was 2 summers in making with Renan's internship slots. Special thanks >> to Maxim, Joshua and Bill to make this happen and mentoring Renan in the >> process. >> >> >> As for point 1, we are going to put out the golang client in few weeks to >> easily test different executors (including thermos) with Aurora with real >> examples. During the same time we will release production grade >> docker-compose executor simulating running container pods with Aurora. >> Expect a detailed blog in a month. >> Thx >> From: David McLaughlin <dmclaugh...@apache.org> >> To: dev@aurora.apache.org >> Sent: Monday, June 13, 2016 10:39 AM >> Subject: Re: Golang Aurora lib, multiple executor support, integrate >> mesos task related fields >> >> Thanks, also +1 to supporting points 2 and 3. >> >>> On Mon, Jun 13, 2016 at 10:33 AM, <meghdoo...@yahoo.com.invalid> wrote: >>> >>> Got it. Thx Max. >>> Renan will starting working on 2 and 3, involving and updating you all. >>> >>> Thx >>> >>> Sent from my iPhone >>> >>>>> On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org> >>>> wrote: >>>> >>>> David pinged me offline suggesting I misread the proposal for (2) as >>>> an attempt to support multiple executors per task rather than "per >>>> cluster". If that's the case, I am happy to change my vote for (2) to >>>> +1. I don't see much harm (aside from possible UI inconsistencies) >>>> from multiple executor types sharing the same cluster. >>>> >>>>> On Mon, Jun 13, 2016 at 9:54 AM, <meghdoo...@yahoo.com.invalid> >> wrote: >>>>> I will leave Renan for a detailed response. >>>>> >>>>> If aurora can schedule Job A with thermos and Job B that requires a >>> different executor that is a first class concept in mesos, why is there a >>> objection? Renan's patch of replacing thermos executor is already in >>> aurora. However right now aurora can statically configure only one >>> executor. And so we want a job submission dynamically can call out the >>> executor type. >>>>> Please clarify your views. >>>>> Right now the docker integration story in mesos is weak (current and >>> also the new universal containerizer). We see much value in using >> executors >>> like >>>>> https://github.com/mesos/docker-compose-executor >>>>> With aurora jobs. >>>>> >>>>> Points 2 and points 3 make the custom executor story stronger. (For >>> example we will run both thermos and compose executor depending on job >> type) >>>>> >>>>> >>>>> Point 1, I agree. That is outside of aurora and most likely will be a >>> shim on top of generated go thrift bindings but a helper to easily test >>> custom executors. >>>>> Last time when we reviewed with aurora team, it came out current >> aurora >>> cli is tightly coupled with thermos objects for jobs and hence keep it as >>> is. >>>>> >>>>> Thx >>>>> >>>>> >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org> >>> wrote: >>>>>> >>>>>> I am with Rick and David on this. The (1) and (2) feel like building >> a >>>>>> parallel universe within Aurora without a clear justification on the >>>>>> long-term benefits for the community. >>>>>> >>>>>> I am strong -1 on bringing yet another language into the Aurora >>>>>> ecosystem without identifying the strategic upside. As David >>>>>> suggested, any efforts towards building the first-class REST API >>>>>> instead would go a long way and will be much appreciated by everyone. >>>>>> >>>>>> I also don't see a reason to support multiple executors per task. >> That >>>>>> feels like re-creating thermos in a thermos-less world. >>>>>> >>>>>> As for (3), I'd like to see more details on the mechanics here but >>>>>> generally positive towards supporting more TaskInfo features in >>>>>> Aurora. >>>>>> >>>>>> >>>>>>> On Sun, Jun 12, 2016 at 11:46 AM, <r...@chartbeat.com> wrote: >>>>>>> I generally shy away from technical goals that are based on a choice >>> of language. What is it about a go client that can't be done by extending >>> the existing client? >>>>>>> >>>>>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle < >> rdelv...@binghamton.edu> >>> wrote: >>>>>>>> >>>>>>>> Hi David, >>>>>>>> >>>>>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin < >>> da...@dmclaughlin.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya < >>>>>>>>> meghdoo...@yahoo.com.invalid> wrote: >>>>>>>>> >>>>>>>>>> Comments inline >>>>>>>>>> >>>>>>>>>> From: David McLaughlin <dmclaugh...@apache.org> >>>>>>>>>> To: dev@aurora.apache.org >>>>>>>>>> Sent: Thursday, June 9, 2016 5:13 PM >>>>>>>>>> Subject: Re: Golang Aurora lib, multiple executor support, >>> integrate >>>>>>>>>> mesos task related fields >>>>>>>>>> >>>>>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle < >>> rdelv...@binghamton.edu> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hello all, >>>>>>>>>>> >>>>>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and >>> I've >>>>>>>>> had >>>>>>>>>>> the pleasure of being part of the Aurora community for the last >>> year or >>>>>>>>>> so. >>>>>>>>>>> >>>>>>>>>>> Last year I worked to allow Aurora to utilize custom executors. >>> With >>>>>>>>> the >>>>>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that >>> feature >>>>>>>>>> became >>>>>>>>>>> a reality and made into Aurora late last year. (Also got to show >>> an >>>>>>>>> early >>>>>>>>>>> beta at MesosCon!) >>>>>>>>>>> >>>>>>>>>>> For this year's summer, I have some new goals which I invite >>> anyone to >>>>>>>>>>> provide input on. >>>>>>>>>>> >>>>>>>>>>> 1. Create a Golang library that works as an alternative to >>> Aurora's >>>>>>>>>> Python >>>>>>>>>>> client and Pystachio. The initial goal of this library is to >>> support >>>>>>>>> the >>>>>>>>>>> most critical job related Thrift API's. (Admin operations can >>> continue >>>>>>>>> to >>>>>>>>>>> be done from the Aurora CLI). Since we support custom executors, >>> we >>>>>>>>> need >>>>>>>>>> a >>>>>>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs >>>>>>>>>>> configurations to Aurora (and by extension the custom executor). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> You can easily add support for a different configuration format >>> without >>>>>>>>>> having to rewrite the CLI in Go. You'd do that by adding your own >>> custom >>>>>>>>>> noun/verbs to the client or replacing existing commands. For >>> example, at >>>>>>>>>> Twitter we have our own version of 'aurora update start' that >>> plugs into >>>>>>>>> an >>>>>>>>>> internal Deploy Orchestration Service and we have a whole other >>> command >>>>>>>>> for >>>>>>>>>> federated deploys. I can show you how we do that. >>>>>>>>>> >>>>>>>>>> The first thing I thought of though was - this seems like a >> perfect >>>>>>>>>> starting point to get serious about a HTTP+JSON API for Aurora. >>> Having >>>>>>>>> that >>>>>>>>>> would make it trivial to do a CLI in any language you want, and >> the >>>>>>>>> Python >>>>>>>>>> CLI would really only be there to do Pystachio evaluation. >>>>>>>>>>>> CLI integrations is not useful for a lot of different reasons. >>> We need >>>>>>>>>> API's from other orchestration points from different components >>> and hence >>>>>>>>>> we would package it in a go library. Uber, I believe has taken >> the >>> same >>>>>>>>>> route (info from this mesoscon) >>>>>>>>>> We are not writing a replacement cli tool. As noted admin >>> operations >>>>>>>>> would >>>>>>>>>> heavily leverage current aurora cli. >>>>>>>>>> I think REST work has been postponed and attempted few times in >>> past. We >>>>>>>>>> cannot wait for that. >>>>>>>>> >>>>>>>>> >>>>>>>>> This is the dev list for contributions to the project, so I >> assumed >>> we were >>>>>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and >>> Thrift >>>>>>>>> has Go bindings so you're all set for whatever custom tooling >> you're >>>>>>>>> building. Please use the users list for feedback on that. >>>>>>>> >>>>>>>> >>>>>>>> While I'm very thankful for your input on the matter, my original >>> text very >>>>>>>> clearly said: >>>>>>>> "1. Create a Golang library that works as an *alternative* to >>>>>>>> Aurora's Python client and Pystachio." >>>>>>>> >>>>>>>> I believe every single item I have listed has the potential to >>> become a >>>>>>>> useful contribution to this project. >>>>>>>> Therefore, I feel that it is prudent continue the discussion here >>> (unless >>>>>>>> others feel similarly). >>>>>>>> >>>>>>>> -Renan >>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2. Create support for using multiple executors within a single >>> Aurora >>>>>>>>>>> scheduler instance. As of right now, only a single executor can >>> exist >>>>>>>>> for >>>>>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora >>>>>>>>> scheduler >>>>>>>>>>> to support multiple executors such that they can be used >>> alongside one >>>>>>>>>>> another, providing flexibility for running jobs. >>>>>>>>>> >>>>>>>>>> Just for my own understanding - what problems are you solving >> with >>> your >>>>>>>>>> custom executor? Sorry if you've explained this to the lists >>> before. >>>>>>>>>> >>>>>>>>>> I ask because I'd really like to see Aurora stop treating the >>> executor >>>>>>>>>> config as an opaque blob and being more opinionated. The >>>>>>>>> Thermos/Scheduler >>>>>>>>>> abstraction really hinders what type of user experience (UI) we >> can >>>>>>>>> build, >>>>>>>>>> and other frameworks do a much better job by being more >>> opinionated and >>>>>>>>>> pulling executor data into the scheduler UI. >>>>>>>>>>>> We probably don't need to debate on this point. Executor is a >>> first >>>>>>>>>> class mesos citizen and time is right for aurora to have good >>> support for >>>>>>>>>> it.In our case, we have kubernetes like pods modeled through >>>>>>>>> docker-compose >>>>>>>>>> and the executor manages that. Scheduler UI main features should >>> not get >>>>>>>>>> bogged down or be held back for a particular executor. That feels >>>>>>>>>> incredibly restrictive. If a special UI mode for thermos executor >>> is >>>>>>>>>> created, that should be fine. We do have to differentiate the >>> scheduler >>>>>>>>> and >>>>>>>>>> thermos and aurora team has done a great job of not hard coupling >>> the >>>>>>>>> two. >>>>>>>>> >>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 3. I'd like to add support for some first class Mesos task >> related >>>>>>>>>> fields. >>>>>>>>>>> These would be optional and/or have defaults so that the regular >>> Aurora >>>>>>>>>> CLI >>>>>>>>>>> does not break. One of the examples I'm interested in >> integrating >>> is >>>>>>>>> the >>>>>>>>>>> Mesos Fetcher so that resources can be downloaded into the >>> sandbox that >>>>>>>>>> the >>>>>>>>>>> custom executor may need. (The executor path will never be >>> exposed as >>>>>>>>>> this >>>>>>>>>>> will be defined serverside and be static). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Wouldn't the mesos-fetcher call just be another process to be >>> passed to >>>>>>>>>> your executor? I have to admit I don't know enough about how the >>> Mesos >>>>>>>>>> fetcher works. >>>>>>>>>>>> Apache Mesos - Fetcher >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> We may add other first class fields. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> If anyone has any feedback on these, I'd be very glad to hear >> it. >>>>>>>>>>> >>>>>>>>>>> That's it for me for now, thanks! >>>>>>>>>>> >>>>>>>>>>> -Renan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | | | >>>>>>>>>> >>>>>>>>>> | >>>>>>>>>> >>>>>>>>>> | >>>>>>>>>> | >>>>>>>>>> | | >>>>>>>>>> Apache Mesos - Fetcher >>>>>>>>>> | | >>>>>>>>>> >>>>>>>>>> | >>>>>>>>>> >>>>>>>>>> | > > > > > -- > Cheers, > > Zhitao Li