Thanks! Truth be told, it wouldn't have been possible without the mentoring
and support from so many in the community. Looking forward to making
further contributions in the future.

-Renan

On Aug 5, 2016 10:26 AM, "David McLaughlin" <dmclaugh...@apache.org> wrote:

> Congratulations Renan on contributing such a major feature back to the
> community. Looking forward to reading the blog post.
>
>
> On Fri, Aug 5, 2016 at 10:15 AM, <meghdoo...@yahoo.com.invalid> wrote:
>
> > Expect that to be up by mid next week and put in 0.16 docs.
> >
> > Sent from my iPhone
> >
> > > On Aug 5, 2016, at 9:23 AM, Zhitao Li <zhitaoli...@gmail.com> wrote:
> > >
> > > This sounds awesome!
> > >
> > > Is there a document for how to use the feature?
> > >
> > > On Fri, Aug 5, 2016 at 12:35 AM, meghdoot bhattacharya <
> > > meghdoo...@yahoo.com.invalid> wrote:
> > >
> > >> Renaming the subject.
> > >> Renan's latest patch in 0.16 has completed the goal of implementing
> > both 2
> > >> and 3. So, Aurora now can not only support custom executors but run
> > >> multiple executors in the cluster!! Mesos fetcher has been integrated
> as
> > >> well to help the custom executors have access to downloaded artifacts
> > that
> > >> it needs. We are going to add dedicated documentation on how to enable
> > this
> > >> in 0.16.
> > >> This was 2 summers in making with Renan's internship slots. Special
> > thanks
> > >> to Maxim, Joshua and Bill to make this happen and mentoring Renan in
> the
> > >> process.
> > >>
> > >>
> > >> As for point 1, we are going to put out the golang client in few weeks
> > to
> > >> easily test different executors (including thermos) with Aurora with
> > real
> > >> examples. During the same time we will release production grade
> > >> docker-compose executor simulating running container pods with Aurora.
> > >> Expect a detailed blog in a month.
> > >> Thx
> > >>      From: David McLaughlin <dmclaugh...@apache.org>
> > >> To: dev@aurora.apache.org
> > >> Sent: Monday, June 13, 2016 10:39 AM
> > >> Subject: Re: Golang Aurora lib, multiple executor support, integrate
> > >> mesos task related fields
> > >>
> > >> Thanks, also +1 to supporting points 2 and 3.
> > >>
> > >>> On Mon, Jun 13, 2016 at 10:33 AM, <meghdoo...@yahoo.com.invalid>
> > wrote:
> > >>>
> > >>> Got it. Thx Max.
> > >>> Renan will starting working on 2 and 3, involving and updating you
> all.
> > >>>
> > >>> Thx
> > >>>
> > >>> Sent from my iPhone
> > >>>
> > >>>>> On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org>
> > >>>> wrote:
> > >>>>
> > >>>> David pinged me offline suggesting I misread the proposal for (2) as
> > >>>> an attempt to support multiple executors per task rather than "per
> > >>>> cluster". If that's the case, I am happy to change my vote for (2)
> to
> > >>>> +1. I don't see much harm (aside from possible UI inconsistencies)
> > >>>> from multiple executor types sharing the same cluster.
> > >>>>
> > >>>>> On Mon, Jun 13, 2016 at 9:54 AM,  <meghdoo...@yahoo.com.invalid>
> > >> wrote:
> > >>>>> I will leave Renan for a detailed response.
> > >>>>>
> > >>>>> If aurora can schedule Job A with thermos and Job B that requires a
> > >>> different executor that is a first class concept in mesos, why is
> > there a
> > >>> objection? Renan's patch of replacing thermos executor is already in
> > >>> aurora. However right now aurora can statically configure only one
> > >>> executor. And so we want a job submission dynamically can call out
> the
> > >>> executor type.
> > >>>>> Please clarify your views.
> > >>>>> Right now the docker integration story in mesos is weak (current
> and
> > >>> also the new universal containerizer). We see much value in using
> > >> executors
> > >>> like
> > >>>>> https://github.com/mesos/docker-compose-executor
> > >>>>> With aurora jobs.
> > >>>>>
> > >>>>> Points 2 and points 3 make the custom executor story stronger. (For
> > >>> example we will run both thermos and compose executor depending on
> job
> > >> type)
> > >>>>>
> > >>>>>
> > >>>>> Point 1, I agree. That is outside of aurora and most likely will
> be a
> > >>> shim on top of generated go thrift bindings but a helper to easily
> test
> > >>> custom executors.
> > >>>>> Last time when we reviewed with aurora team, it came out current
> > >> aurora
> > >>> cli is tightly coupled with thermos objects for jobs and hence keep
> it
> > as
> > >>> is.
> > >>>>>
> > >>>>> Thx
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Sent from my iPhone
> > >>>>>
> > >>>>>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
> > >>> wrote:
> > >>>>>>
> > >>>>>> I am with Rick and David on this. The (1) and (2) feel like
> building
> > >> a
> > >>>>>> parallel universe within Aurora without a clear justification on
> the
> > >>>>>> long-term benefits for the community.
> > >>>>>>
> > >>>>>> I am strong -1 on bringing yet another language into the Aurora
> > >>>>>> ecosystem without identifying the strategic upside. As David
> > >>>>>> suggested, any efforts towards building the first-class REST API
> > >>>>>> instead would go a long way and will be much appreciated by
> > everyone.
> > >>>>>>
> > >>>>>> I also don't see a reason to support multiple executors per task.
> > >> That
> > >>>>>> feels like re-creating thermos in a thermos-less world.
> > >>>>>>
> > >>>>>> As for (3), I'd like to see more details on the mechanics here but
> > >>>>>> generally positive towards supporting more TaskInfo features in
> > >>>>>> Aurora.
> > >>>>>>
> > >>>>>>
> > >>>>>>> On Sun, Jun 12, 2016 at 11:46 AM,  <r...@chartbeat.com> wrote:
> > >>>>>>> I generally shy away from technical goals that are based on a
> > choice
> > >>> of language. What is it about a go client that can't be done by
> > extending
> > >>> the existing client?
> > >>>>>>>
> > >>>>>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <
> > >> rdelv...@binghamton.edu>
> > >>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi David,
> > >>>>>>>>
> > >>>>>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <
> > >>> da...@dmclaughlin.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
> > >>>>>>>>> meghdoo...@yahoo.com.invalid> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Comments inline
> > >>>>>>>>>>
> > >>>>>>>>>>   From: David McLaughlin <dmclaugh...@apache.org>
> > >>>>>>>>>> To: dev@aurora.apache.org
> > >>>>>>>>>> Sent: Thursday, June 9, 2016 5:13 PM
> > >>>>>>>>>> Subject: Re: Golang Aurora lib, multiple executor support,
> > >>> integrate
> > >>>>>>>>>> mesos task related fields
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <
> > >>> rdelv...@binghamton.edu>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hello all,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle
> and
> > >>> I've
> > >>>>>>>>> had
> > >>>>>>>>>>> the pleasure of being part of the Aurora community for the
> last
> > >>> year or
> > >>>>>>>>>> so.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Last year I worked to allow Aurora to utilize custom
> executors.
> > >>> With
> > >>>>>>>>> the
> > >>>>>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that
> > >>> feature
> > >>>>>>>>>> became
> > >>>>>>>>>>> a reality and made into Aurora late last year. (Also got to
> > show
> > >>> an
> > >>>>>>>>> early
> > >>>>>>>>>>> beta at MesosCon!)
> > >>>>>>>>>>>
> > >>>>>>>>>>> For this year's summer, I have some new goals which I invite
> > >>> anyone to
> > >>>>>>>>>>> provide input on.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 1. Create a Golang library that works as an alternative to
> > >>> Aurora's
> > >>>>>>>>>> Python
> > >>>>>>>>>>> client and Pystachio. The initial goal of this library is to
> > >>> support
> > >>>>>>>>> the
> > >>>>>>>>>>> most critical job related Thrift API's. (Admin operations can
> > >>> continue
> > >>>>>>>>> to
> > >>>>>>>>>>> be done from the Aurora CLI). Since we support custom
> > executors,
> > >>> we
> > >>>>>>>>> need
> > >>>>>>>>>> a
> > >>>>>>>>>>> way that's not so tied to thermos (as Pystachio is) to send
> > jobs
> > >>>>>>>>>>> configurations to Aurora (and by extension the custom
> > executor).
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> You can easily add support for a different configuration
> format
> > >>> without
> > >>>>>>>>>> having to rewrite the CLI in Go. You'd do that by adding your
> > own
> > >>> custom
> > >>>>>>>>>> noun/verbs to the client or replacing existing commands. For
> > >>> example, at
> > >>>>>>>>>> Twitter we have our own version of 'aurora update start' that
> > >>> plugs into
> > >>>>>>>>> an
> > >>>>>>>>>> internal Deploy Orchestration Service and we have a whole
> other
> > >>> command
> > >>>>>>>>> for
> > >>>>>>>>>> federated deploys. I can show you how we do that.
> > >>>>>>>>>>
> > >>>>>>>>>> The first thing I thought of though was - this seems like a
> > >> perfect
> > >>>>>>>>>> starting point to get serious about a HTTP+JSON API for
> Aurora.
> > >>> Having
> > >>>>>>>>> that
> > >>>>>>>>>> would make it trivial to do a CLI in any language you want,
> and
> > >> the
> > >>>>>>>>> Python
> > >>>>>>>>>> CLI would really only be there to do Pystachio evaluation.
> > >>>>>>>>>>>> CLI integrations is not useful for a lot of different
> reasons.
> > >>> We need
> > >>>>>>>>>> API's from other orchestration points from different
> components
> > >>> and hence
> > >>>>>>>>>> we would package it in a go library. Uber, I believe has taken
> > >> the
> > >>> same
> > >>>>>>>>>> route (info from this mesoscon)
> > >>>>>>>>>> We are not writing a replacement cli tool. As noted admin
> > >>> operations
> > >>>>>>>>> would
> > >>>>>>>>>> heavily leverage current aurora cli.
> > >>>>>>>>>> I think REST work has been postponed and attempted few times
> in
> > >>> past. We
> > >>>>>>>>>> cannot wait for that.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> This is the dev list for contributions to the project, so I
> > >> assumed
> > >>> we were
> > >>>>>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift
> > and
> > >>> Thrift
> > >>>>>>>>> has Go bindings so you're all set for whatever custom tooling
> > >> you're
> > >>>>>>>>> building. Please use the users list for feedback on that.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> While I'm very thankful for your input on the matter, my
> original
> > >>> text very
> > >>>>>>>> clearly said:
> > >>>>>>>> "1. Create a Golang library that works as an *alternative* to
> > >>>>>>>> Aurora's Python client and Pystachio."
> > >>>>>>>>
> > >>>>>>>> I believe every single item I have listed has the potential to
> > >>> become a
> > >>>>>>>> useful contribution to this project.
> > >>>>>>>> Therefore, I feel that it is prudent continue the discussion
> here
> > >>> (unless
> > >>>>>>>> others feel similarly).
> > >>>>>>>>
> > >>>>>>>> -Renan
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2. Create support for using multiple executors within a
> single
> > >>> Aurora
> > >>>>>>>>>>> scheduler instance. As of right now, only a single executor
> can
> > >>> exist
> > >>>>>>>>> for
> > >>>>>>>>>>> each Aurora scheduler instance. The idea is to allow the
> Aurora
> > >>>>>>>>> scheduler
> > >>>>>>>>>>> to support multiple executors such that they can be used
> > >>> alongside one
> > >>>>>>>>>>> another, providing flexibility for running jobs.
> > >>>>>>>>>>
> > >>>>>>>>>> Just for my own understanding - what problems are you solving
> > >> with
> > >>> your
> > >>>>>>>>>> custom executor? Sorry if you've explained this to the lists
> > >>> before.
> > >>>>>>>>>>
> > >>>>>>>>>> I ask because I'd really like to see Aurora stop treating the
> > >>> executor
> > >>>>>>>>>> config as an opaque blob and being more opinionated. The
> > >>>>>>>>> Thermos/Scheduler
> > >>>>>>>>>> abstraction really hinders what type of user experience (UI)
> we
> > >> can
> > >>>>>>>>> build,
> > >>>>>>>>>> and other frameworks do a much better job by being more
> > >>> opinionated and
> > >>>>>>>>>> pulling executor data into the scheduler UI.
> > >>>>>>>>>>>> We probably don't need to debate on this point. Executor is
> a
> > >>> first
> > >>>>>>>>>> class mesos citizen and time is right for aurora to have good
> > >>> support for
> > >>>>>>>>>> it.In our case, we have kubernetes like pods modeled through
> > >>>>>>>>> docker-compose
> > >>>>>>>>>> and the executor manages that. Scheduler UI main features
> should
> > >>> not get
> > >>>>>>>>>> bogged down or be held back for a particular executor. That
> > feels
> > >>>>>>>>>> incredibly restrictive. If a special UI mode for thermos
> > executor
> > >>> is
> > >>>>>>>>>> created, that should be fine. We do have to differentiate the
> > >>> scheduler
> > >>>>>>>>> and
> > >>>>>>>>>> thermos and aurora team has done a great job of not hard
> > coupling
> > >>> the
> > >>>>>>>>> two.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> 3. I'd like to add support for some first class Mesos task
> > >> related
> > >>>>>>>>>> fields.
> > >>>>>>>>>>> These would be optional and/or have defaults so that the
> > regular
> > >>> Aurora
> > >>>>>>>>>> CLI
> > >>>>>>>>>>> does not break. One of the examples I'm interested in
> > >> integrating
> > >>> is
> > >>>>>>>>> the
> > >>>>>>>>>>> Mesos Fetcher so that resources can be downloaded into the
> > >>> sandbox that
> > >>>>>>>>>> the
> > >>>>>>>>>>> custom executor may need. (The executor path will never be
> > >>> exposed as
> > >>>>>>>>>> this
> > >>>>>>>>>>> will be defined serverside and be static).
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Wouldn't the mesos-fetcher call just be another process to be
> > >>> passed to
> > >>>>>>>>>> your executor? I have to admit I don't know enough about how
> the
> > >>> Mesos
> > >>>>>>>>>> fetcher works.
> > >>>>>>>>>>>> Apache Mesos - Fetcher
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> We may add other first class fields.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> If anyone has any feedback on these, I'd be very glad to hear
> > >> it.
> > >>>>>>>>>>>
> > >>>>>>>>>>> That's it for me for now, thanks!
> > >>>>>>>>>>>
> > >>>>>>>>>>> -Renan
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> |
> > >>>>>>>>>> |
> > >>>>>>>>>> |
> > >>>>>>>>>> |  |    |
> > >>>>>>>>>>
> > >>>>>>>>>> |
> > >>>>>>>>>>
> > >>>>>>>>>> |
> > >>>>>>>>>> |
> > >>>>>>>>>> |  |
> > >>>>>>>>>> Apache Mesos - Fetcher
> > >>>>>>>>>> |  |
> > >>>>>>>>>>
> > >>>>>>>>>> |
> > >>>>>>>>>>
> > >>>>>>>>>> |
> > >
> > >
> > >
> > >
> > > --
> > > Cheers,
> > >
> > > Zhitao Li
> >
>

Reply via email to