Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

I'll try to find some time, but it's really at a premium right now. On Mon, Mar 4, 2019 at 3:17 PM Xiangrui Meng wrote: > > > On Mon, Mar 4, 2019 at 3:10 PM Mark Hamstra > wrote: > >> :) Sorry, that was ambiguous. I was seconding Imran's comment. >> > > Could you also help review Xingbo's

Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

:) Sorry, that was ambiguous. I was seconding Imran's comment. On Mon, Mar 4, 2019 at 3:09 PM Xiangrui Meng wrote: > > > On Mon, Mar 4, 2019 at 1:56 PM Mark Hamstra > wrote: > >> +1 >> > > Mark, just to be clear, are you +1 on the SPIP or Imran's point? > > >> >> On Mon, Mar 4, 2019 at 12:52

Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

On Mon, Mar 4, 2019 at 3:10 PM Mark Hamstra wrote: > :) Sorry, that was ambiguous. I was seconding Imran's comment. > Could you also help review Xingbo's design sketch and help evaluate the cost? > > On Mon, Mar 4, 2019 at 3:09 PM Xiangrui Meng wrote: > >> >> >> On Mon, Mar 4, 2019 at 1:56

Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

On Mon, Mar 4, 2019 at 1:56 PM Mark Hamstra wrote: > +1 > Mark, just to be clear, are you +1 on the SPIP or Imran's point? > > On Mon, Mar 4, 2019 at 12:52 PM Imran Rashid wrote: > >> On Sun, Mar 3, 2019 at 6:51 PM Xiangrui Meng wrote: >> >>> On Sun, Mar 3, 2019 at 10:20 AM Felix Cheung

Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

+1 On Mon, Mar 4, 2019 at 12:52 PM Imran Rashid wrote: > On Sun, Mar 3, 2019 at 6:51 PM Xiangrui Meng wrote: > >> On Sun, Mar 3, 2019 at 10:20 AM Felix Cheung >> wrote: >> >>> IMO upfront allocation is less useful. Specifically too expensive for >>> large jobs. >>> >> >> This is also an

Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

On Sun, Mar 3, 2019 at 6:51 PM Xiangrui Meng wrote: > On Sun, Mar 3, 2019 at 10:20 AM Felix Cheung > wrote: > >> IMO upfront allocation is less useful. Specifically too expensive for >> large jobs. >> > > This is also an API/design discussion. > I agree with Felix -- this is more than just an

Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

It sounds like there's a discussion about the details coming, which is fine and good. That should maybe also have a VOTE. The debate here is then merely about what and when to call things a SPIP, but that's not important. On Mon, Mar 4, 2019 at 10:23 AM Xiangrui Meng wrote: > I think the two

Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

On Mon, Mar 4, 2019 at 8:23 AM Xiangrui Meng wrote: > > > On Mon, Mar 4, 2019 at 7:24 AM Sean Owen wrote: > >> To be clear, those goals sound fine to me. I don't think voting on >> those two broad points is meaningful, but, does no harm per se. If you >> mean this is just a check to see if

Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

On Mon, Mar 4, 2019 at 7:24 AM Sean Owen wrote: > To be clear, those goals sound fine to me. I don't think voting on > those two broad points is meaningful, but, does no harm per se. If you > mean this is just a check to see if people believe this is broadly > worthwhile, then +1 from me. Yes it

Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

To be clear, those goals sound fine to me. I don't think voting on those two broad points is meaningful, but, does no harm per se. If you mean this is just a check to see if people believe this is broadly worthwhile, then +1 from me. Yes it is. That means we'd want to review something more

Re: [VOTE] [SPARK-24615] SPIP: Accelerator-aware Scheduling

What finer "high level" goals do you recommend? To make progress on the vote, it would be great if you can articulate more. Current SPIP proposes two high-level changes to make Spark accelerator-aware: - At cluster manager level, we update or upgrade cluster managers to include GPU support.

Re: [DISCUSS][SQL][PySpark] Column name support for SQL functions

Hey everyone, Progress has been made with PR #23882, and it is now in a state where it could be merged with master. This is what we’re doing for now: 1. PySpark will support strings consistently throughout its API. * Arguably string