On Sun, Mar 3, 2019 at 6:51 PM Xiangrui Meng <men...@gmail.com> wrote:
> On Sun, Mar 3, 2019 at 10:20 AM Felix Cheung <felixcheun...@hotmail.com> > wrote: > >> IMO upfront allocation is less useful. Specifically too expensive for >> large jobs. >> > > This is also an API/design discussion. > I agree with Felix -- this is more than just an API question. It has a huge impact on the complexity of what you're proposing. You might be proposing big changes to a core and brittle part of spark, which is already short of experts. I don't see any value in having a vote on "does feature X sound cool?" We have to evaluate the potential benefit against the risks the feature brings and the continued maintenance cost. We don't need super low-level details, but we have to a sketch of the design to be able to make that tradeoff.
