Thanks! Truth be told, it wouldn't have been possible without the mentoring and support from so many in the community. Looking forward to making further contributions in the future.
-Renan On Aug 5, 2016 10:26 AM, "David McLaughlin" <dmclaugh...@apache.org> wrote: > Congratulations Renan on contributing such a major feature back to the > community. Looking forward to reading the blog post. > > > On Fri, Aug 5, 2016 at 10:15 AM, <meghdoo...@yahoo.com.invalid> wrote: > > > Expect that to be up by mid next week and put in 0.16 docs. > > > > Sent from my iPhone > > > > > On Aug 5, 2016, at 9:23 AM, Zhitao Li <zhitaoli...@gmail.com> wrote: > > > > > > This sounds awesome! > > > > > > Is there a document for how to use the feature? > > > > > > On Fri, Aug 5, 2016 at 12:35 AM, meghdoot bhattacharya < > > > meghdoo...@yahoo.com.invalid> wrote: > > > > > >> Renaming the subject. > > >> Renan's latest patch in 0.16 has completed the goal of implementing > > both 2 > > >> and 3. So, Aurora now can not only support custom executors but run > > >> multiple executors in the cluster!! Mesos fetcher has been integrated > as > > >> well to help the custom executors have access to downloaded artifacts > > that > > >> it needs. We are going to add dedicated documentation on how to enable > > this > > >> in 0.16. > > >> This was 2 summers in making with Renan's internship slots. Special > > thanks > > >> to Maxim, Joshua and Bill to make this happen and mentoring Renan in > the > > >> process. > > >> > > >> > > >> As for point 1, we are going to put out the golang client in few weeks > > to > > >> easily test different executors (including thermos) with Aurora with > > real > > >> examples. During the same time we will release production grade > > >> docker-compose executor simulating running container pods with Aurora. > > >> Expect a detailed blog in a month. > > >> Thx > > >> From: David McLaughlin <dmclaugh...@apache.org> > > >> To: dev@aurora.apache.org > > >> Sent: Monday, June 13, 2016 10:39 AM > > >> Subject: Re: Golang Aurora lib, multiple executor support, integrate > > >> mesos task related fields > > >> > > >> Thanks, also +1 to supporting points 2 and 3. > > >> > > >>> On Mon, Jun 13, 2016 at 10:33 AM, <meghdoo...@yahoo.com.invalid> > > wrote: > > >>> > > >>> Got it. Thx Max. > > >>> Renan will starting working on 2 and 3, involving and updating you > all. > > >>> > > >>> Thx > > >>> > > >>> Sent from my iPhone > > >>> > > >>>>> On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org> > > >>>> wrote: > > >>>> > > >>>> David pinged me offline suggesting I misread the proposal for (2) as > > >>>> an attempt to support multiple executors per task rather than "per > > >>>> cluster". If that's the case, I am happy to change my vote for (2) > to > > >>>> +1. I don't see much harm (aside from possible UI inconsistencies) > > >>>> from multiple executor types sharing the same cluster. > > >>>> > > >>>>> On Mon, Jun 13, 2016 at 9:54 AM, <meghdoo...@yahoo.com.invalid> > > >> wrote: > > >>>>> I will leave Renan for a detailed response. > > >>>>> > > >>>>> If aurora can schedule Job A with thermos and Job B that requires a > > >>> different executor that is a first class concept in mesos, why is > > there a > > >>> objection? Renan's patch of replacing thermos executor is already in > > >>> aurora. However right now aurora can statically configure only one > > >>> executor. And so we want a job submission dynamically can call out > the > > >>> executor type. > > >>>>> Please clarify your views. > > >>>>> Right now the docker integration story in mesos is weak (current > and > > >>> also the new universal containerizer). We see much value in using > > >> executors > > >>> like > > >>>>> https://github.com/mesos/docker-compose-executor > > >>>>> With aurora jobs. > > >>>>> > > >>>>> Points 2 and points 3 make the custom executor story stronger. (For > > >>> example we will run both thermos and compose executor depending on > job > > >> type) > > >>>>> > > >>>>> > > >>>>> Point 1, I agree. That is outside of aurora and most likely will > be a > > >>> shim on top of generated go thrift bindings but a helper to easily > test > > >>> custom executors. > > >>>>> Last time when we reviewed with aurora team, it came out current > > >> aurora > > >>> cli is tightly coupled with thermos objects for jobs and hence keep > it > > as > > >>> is. > > >>>>> > > >>>>> Thx > > >>>>> > > >>>>> > > >>>>> > > >>>>> Sent from my iPhone > > >>>>> > > >>>>>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org> > > >>> wrote: > > >>>>>> > > >>>>>> I am with Rick and David on this. The (1) and (2) feel like > building > > >> a > > >>>>>> parallel universe within Aurora without a clear justification on > the > > >>>>>> long-term benefits for the community. > > >>>>>> > > >>>>>> I am strong -1 on bringing yet another language into the Aurora > > >>>>>> ecosystem without identifying the strategic upside. As David > > >>>>>> suggested, any efforts towards building the first-class REST API > > >>>>>> instead would go a long way and will be much appreciated by > > everyone. > > >>>>>> > > >>>>>> I also don't see a reason to support multiple executors per task. > > >> That > > >>>>>> feels like re-creating thermos in a thermos-less world. > > >>>>>> > > >>>>>> As for (3), I'd like to see more details on the mechanics here but > > >>>>>> generally positive towards supporting more TaskInfo features in > > >>>>>> Aurora. > > >>>>>> > > >>>>>> > > >>>>>>> On Sun, Jun 12, 2016 at 11:46 AM, <r...@chartbeat.com> wrote: > > >>>>>>> I generally shy away from technical goals that are based on a > > choice > > >>> of language. What is it about a go client that can't be done by > > extending > > >>> the existing client? > > >>>>>>> > > >>>>>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle < > > >> rdelv...@binghamton.edu> > > >>> wrote: > > >>>>>>>> > > >>>>>>>> Hi David, > > >>>>>>>> > > >>>>>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin < > > >>> da...@dmclaughlin.com> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya < > > >>>>>>>>> meghdoo...@yahoo.com.invalid> wrote: > > >>>>>>>>> > > >>>>>>>>>> Comments inline > > >>>>>>>>>> > > >>>>>>>>>> From: David McLaughlin <dmclaugh...@apache.org> > > >>>>>>>>>> To: dev@aurora.apache.org > > >>>>>>>>>> Sent: Thursday, June 9, 2016 5:13 PM > > >>>>>>>>>> Subject: Re: Golang Aurora lib, multiple executor support, > > >>> integrate > > >>>>>>>>>> mesos task related fields > > >>>>>>>>>> > > >>>>>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle < > > >>> rdelv...@binghamton.edu> > > >>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Hello all, > > >>>>>>>>>>> > > >>>>>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle > and > > >>> I've > > >>>>>>>>> had > > >>>>>>>>>>> the pleasure of being part of the Aurora community for the > last > > >>> year or > > >>>>>>>>>> so. > > >>>>>>>>>>> > > >>>>>>>>>>> Last year I worked to allow Aurora to utilize custom > executors. > > >>> With > > >>>>>>>>> the > > >>>>>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that > > >>> feature > > >>>>>>>>>> became > > >>>>>>>>>>> a reality and made into Aurora late last year. (Also got to > > show > > >>> an > > >>>>>>>>> early > > >>>>>>>>>>> beta at MesosCon!) > > >>>>>>>>>>> > > >>>>>>>>>>> For this year's summer, I have some new goals which I invite > > >>> anyone to > > >>>>>>>>>>> provide input on. > > >>>>>>>>>>> > > >>>>>>>>>>> 1. Create a Golang library that works as an alternative to > > >>> Aurora's > > >>>>>>>>>> Python > > >>>>>>>>>>> client and Pystachio. The initial goal of this library is to > > >>> support > > >>>>>>>>> the > > >>>>>>>>>>> most critical job related Thrift API's. (Admin operations can > > >>> continue > > >>>>>>>>> to > > >>>>>>>>>>> be done from the Aurora CLI). Since we support custom > > executors, > > >>> we > > >>>>>>>>> need > > >>>>>>>>>> a > > >>>>>>>>>>> way that's not so tied to thermos (as Pystachio is) to send > > jobs > > >>>>>>>>>>> configurations to Aurora (and by extension the custom > > executor). > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> You can easily add support for a different configuration > format > > >>> without > > >>>>>>>>>> having to rewrite the CLI in Go. You'd do that by adding your > > own > > >>> custom > > >>>>>>>>>> noun/verbs to the client or replacing existing commands. For > > >>> example, at > > >>>>>>>>>> Twitter we have our own version of 'aurora update start' that > > >>> plugs into > > >>>>>>>>> an > > >>>>>>>>>> internal Deploy Orchestration Service and we have a whole > other > > >>> command > > >>>>>>>>> for > > >>>>>>>>>> federated deploys. I can show you how we do that. > > >>>>>>>>>> > > >>>>>>>>>> The first thing I thought of though was - this seems like a > > >> perfect > > >>>>>>>>>> starting point to get serious about a HTTP+JSON API for > Aurora. > > >>> Having > > >>>>>>>>> that > > >>>>>>>>>> would make it trivial to do a CLI in any language you want, > and > > >> the > > >>>>>>>>> Python > > >>>>>>>>>> CLI would really only be there to do Pystachio evaluation. > > >>>>>>>>>>>> CLI integrations is not useful for a lot of different > reasons. > > >>> We need > > >>>>>>>>>> API's from other orchestration points from different > components > > >>> and hence > > >>>>>>>>>> we would package it in a go library. Uber, I believe has taken > > >> the > > >>> same > > >>>>>>>>>> route (info from this mesoscon) > > >>>>>>>>>> We are not writing a replacement cli tool. As noted admin > > >>> operations > > >>>>>>>>> would > > >>>>>>>>>> heavily leverage current aurora cli. > > >>>>>>>>>> I think REST work has been postponed and attempted few times > in > > >>> past. We > > >>>>>>>>>> cannot wait for that. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> This is the dev list for contributions to the project, so I > > >> assumed > > >>> we were > > >>>>>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift > > and > > >>> Thrift > > >>>>>>>>> has Go bindings so you're all set for whatever custom tooling > > >> you're > > >>>>>>>>> building. Please use the users list for feedback on that. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> While I'm very thankful for your input on the matter, my > original > > >>> text very > > >>>>>>>> clearly said: > > >>>>>>>> "1. Create a Golang library that works as an *alternative* to > > >>>>>>>> Aurora's Python client and Pystachio." > > >>>>>>>> > > >>>>>>>> I believe every single item I have listed has the potential to > > >>> become a > > >>>>>>>> useful contribution to this project. > > >>>>>>>> Therefore, I feel that it is prudent continue the discussion > here > > >>> (unless > > >>>>>>>> others feel similarly). > > >>>>>>>> > > >>>>>>>> -Renan > > >>>>>>>> > > >>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> 2. Create support for using multiple executors within a > single > > >>> Aurora > > >>>>>>>>>>> scheduler instance. As of right now, only a single executor > can > > >>> exist > > >>>>>>>>> for > > >>>>>>>>>>> each Aurora scheduler instance. The idea is to allow the > Aurora > > >>>>>>>>> scheduler > > >>>>>>>>>>> to support multiple executors such that they can be used > > >>> alongside one > > >>>>>>>>>>> another, providing flexibility for running jobs. > > >>>>>>>>>> > > >>>>>>>>>> Just for my own understanding - what problems are you solving > > >> with > > >>> your > > >>>>>>>>>> custom executor? Sorry if you've explained this to the lists > > >>> before. > > >>>>>>>>>> > > >>>>>>>>>> I ask because I'd really like to see Aurora stop treating the > > >>> executor > > >>>>>>>>>> config as an opaque blob and being more opinionated. The > > >>>>>>>>> Thermos/Scheduler > > >>>>>>>>>> abstraction really hinders what type of user experience (UI) > we > > >> can > > >>>>>>>>> build, > > >>>>>>>>>> and other frameworks do a much better job by being more > > >>> opinionated and > > >>>>>>>>>> pulling executor data into the scheduler UI. > > >>>>>>>>>>>> We probably don't need to debate on this point. Executor is > a > > >>> first > > >>>>>>>>>> class mesos citizen and time is right for aurora to have good > > >>> support for > > >>>>>>>>>> it.In our case, we have kubernetes like pods modeled through > > >>>>>>>>> docker-compose > > >>>>>>>>>> and the executor manages that. Scheduler UI main features > should > > >>> not get > > >>>>>>>>>> bogged down or be held back for a particular executor. That > > feels > > >>>>>>>>>> incredibly restrictive. If a special UI mode for thermos > > executor > > >>> is > > >>>>>>>>>> created, that should be fine. We do have to differentiate the > > >>> scheduler > > >>>>>>>>> and > > >>>>>>>>>> thermos and aurora team has done a great job of not hard > > coupling > > >>> the > > >>>>>>>>> two. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> 3. I'd like to add support for some first class Mesos task > > >> related > > >>>>>>>>>> fields. > > >>>>>>>>>>> These would be optional and/or have defaults so that the > > regular > > >>> Aurora > > >>>>>>>>>> CLI > > >>>>>>>>>>> does not break. One of the examples I'm interested in > > >> integrating > > >>> is > > >>>>>>>>> the > > >>>>>>>>>>> Mesos Fetcher so that resources can be downloaded into the > > >>> sandbox that > > >>>>>>>>>> the > > >>>>>>>>>>> custom executor may need. (The executor path will never be > > >>> exposed as > > >>>>>>>>>> this > > >>>>>>>>>>> will be defined serverside and be static). > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Wouldn't the mesos-fetcher call just be another process to be > > >>> passed to > > >>>>>>>>>> your executor? I have to admit I don't know enough about how > the > > >>> Mesos > > >>>>>>>>>> fetcher works. > > >>>>>>>>>>>> Apache Mesos - Fetcher > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>> We may add other first class fields. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> If anyone has any feedback on these, I'd be very glad to hear > > >> it. > > >>>>>>>>>>> > > >>>>>>>>>>> That's it for me for now, thanks! > > >>>>>>>>>>> > > >>>>>>>>>>> -Renan > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> | > > >>>>>>>>>> | > > >>>>>>>>>> | > > >>>>>>>>>> | | | > > >>>>>>>>>> > > >>>>>>>>>> | > > >>>>>>>>>> > > >>>>>>>>>> | > > >>>>>>>>>> | > > >>>>>>>>>> | | > > >>>>>>>>>> Apache Mesos - Fetcher > > >>>>>>>>>> | | > > >>>>>>>>>> > > >>>>>>>>>> | > > >>>>>>>>>> > > >>>>>>>>>> | > > > > > > > > > > > > > > > -- > > > Cheers, > > > > > > Zhitao Li > > >