Re: [VOTE] Spark 2.3.1 (RC1)

FYI the fix for the blocker has just been committed. I'll prepare RC2 tomorrow morning assuming jenkins is reasonably happy with the current state of the branch. On Fri, May 18, 2018 at 10:39 AM, Marcelo Vanzin wrote: > Just to give folks an update. > > In case you haven't

Re: eager execution and debuggability

+1 to job and stage info in the SQL visualization. This is one of the most difficult places for both users and our data platform team to understand. We've resorted to logging the plan that is compiled in `WholeStageCodegenExec` so at least we can go from a stage to what the plan was, but there's

Re: Revisiting Online serving of Spark models?

+1 on meeting up! From: Holden Karau Sent: Monday, May 21, 2018 2:52:20 PM To: Joseph Bradley Cc: Felix Cheung; dev Subject: Re: Revisiting Online serving of Spark models? (Oh also the write API has already been extended to take formats).

Re: Revisiting Online serving of Spark models?

I like that idea. I’ll be around Spark Summit. On Mon, May 21, 2018 at 1:52 PM Joseph Bradley wrote: > Regarding model reading and writing, I'll give quick thoughts here: > * Our approach was to use the same format but write JSON instead of > Parquet. It's easier to

Re: Revisiting Online serving of Spark models?

(Oh also the write API has already been extended to take formats). On Mon, May 21, 2018 at 2:51 PM Holden Karau wrote: > I like that idea. I’ll be around Spark Summit. > > On Mon, May 21, 2018 at 1:52 PM Joseph Bradley > wrote: > >> Regarding model

Re: Revisiting Online serving of Spark models?

Regarding model reading and writing, I'll give quick thoughts here: * Our approach was to use the same format but write JSON instead of Parquet. It's easier to parse JSON without Spark, and using the same format simplifies architecture. Plus, some people want to check files into version control,

Re: Running lint-java during PR builds?

Can we look into if there is a plugin for sbt that works and then we can put everything into one single builder? On Mon, May 21, 2018 at 11:17 AM Dongjoon Hyun wrote: > Thank you for reconsidering this, Hyukjin. :) > > Bests, > Dongjoon. > > > On Mon, May 21, 2018 at

Re: Running lint-java during PR builds?

Thank you for reconsidering this, Hyukjin. :) Bests, Dongjoon. On Mon, May 21, 2018 at 9:20 AM, Marcelo Vanzin wrote: > Is there a way to trigger it conditionally? e.g. only if the diff > touches java files. > > On Mon, May 21, 2018 at 9:17 AM, Felix Cheung

Re: Running lint-java during PR builds?

Is there a way to trigger it conditionally? e.g. only if the diff touches java files. On Mon, May 21, 2018 at 9:17 AM, Felix Cheung wrote: > One concern is with the volume of test runs on Travis. > > In ASF projects Travis could get significantly > backed up since - if

Re: Running lint-java during PR builds?

One concern is with the volume of test runs on Travis. In ASF projects Travis could get significantly backed up since - if I recall - all of ASF shares one queue. At the number of PRs Spark has this could be a big issue. From: Marcelo Vanzin

Re: Running lint-java during PR builds?

I'm fine with it. I tried to use the existing checkstyle sbt plugin (trying to fix SPARK-22269), but it depends on an ancient version of checkstyle, and I don't know sbt enough to figure out how to hack classpaths and class loaders when applying rules, so gave up. On Mon, May 21, 2018 at 1:47 AM,

Re: Running lint-java during PR builds?

I am going to open an INFRA JIRA if there's no explicit objection in few days. 2018-05-21 13:09 GMT+08:00 Hyukjin Kwon : > I would like to revive this proposal. Travis CI. Shall we give this try? I > think it's worth trying it. > > 2016-11-17 3:50 GMT+08:00 Dongjoon Hyun