[ ] Use Spark 1 & Spark 2 Support Branch [X] Use Spark 2 Only Branch On Sat, Nov 18, 2017 at 1:54 AM, Ben Sidhom <sid...@google.com.invalid> wrote:
> [ ] Use Spark 1 & Spark 2 Support Branch > [X] Use Spark 2 Only Branch > > On Fri, Nov 17, 2017 at 9:46 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > > [ ] Use Spark 1 & Spark 2 Support Branch > > [X] Use Spark 2 Only Branch > > > > On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > > > > > Hi guys, > > > > > > To illustrate the current discussion about Spark versions support, you > > can > > > take a look on: > > > > > > -- > > > Spark 1 & Spark 2 Support Branch > > > > > > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES > > > > > > This branch contains a Spark runner common module compatible with both > > > Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2 > > > modules/artifacts containing just a pom.xml to define the dependencies > > set. > > > > > > -- > > > Spark 2 Only Branch > > > > > > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY > > > > > > This branch is an upgrade to Spark 2.x and "drop" support of Spark 1.x. > > > > > > As I'm ready to merge one of the other in the PR, I would like to > > complete > > > the vote/discussion pretty soon. > > > > > > Correct me if I'm wrong, but it seems that the preference is to drop > > Spark > > > 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch). > > > > > > I would like to call a final vote to act the merge I will do: > > > > > > [ ] Use Spark 1 & Spark 2 Support Branch > > > [ ] Use Spark 2 Only Branch > > > > > > This informal vote is open for 48 hours. > > > > > > Please, let me know what your preference is. > > > > > > Thanks ! > > > Regards > > > JB > > > > > > On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote: > > > > > >> Hi Beamers, > > >> > > >> I'm forwarding this discussion & vote from the dev mailing list to the > > >> user mailing list. > > >> The goal is to have your feedback as user. > > >> > > >> Basically, we have two options: > > >> 1. Right now, in the PR, we support both Spark 1.x and 2.x using three > > >> artifacts (common, spark1, spark2). You, as users, pick up spark1 or > > spark2 > > >> in your dependencies set depending the Spark target version you want. > > >> 2. The other option is to upgrade and focus on Spark 2.x in Beam > 2.3.0. > > >> If you still want to use Spark 1.x, then, you will be stuck up to Beam > > >> 2.2.0. > > >> > > >> Thoughts ? > > >> > > >> Thanks ! > > >> Regards > > >> JB > > >> > > >> > > >> -------- Forwarded Message -------- > > >> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x > > >> Date: Wed, 8 Nov 2017 08:27:58 +0100 > > >> From: Jean-Baptiste Onofré <j...@nanthrax.net> > > >> Reply-To: dev@beam.apache.org > > >> To: dev@beam.apache.org > > >> > > >> Hi all, > > >> > > >> as you might know, we are working on Spark 2.x support in the Spark > > >> runner. > > >> > > >> I'm working on a PR about that: > > >> > > >> https://github.com/apache/beam/pull/3808 > > >> > > >> Today, we have something working with both Spark 1.x and 2.x from a > code > > >> standpoint, but I have to deal with dependencies. It's the first step > of > > >> the update as I'm still using RDD, the second step would be to support > > >> dataframe (but for that, I would need PCollection elements with > schemas, > > >> that's another topic on which Eugene, Reuven and I are discussing). > > >> > > >> However, as all major distributions now ship Spark 2.x, I don't think > > >> it's required anymore to support Spark 1.x. > > >> > > >> If we agree, I will update and cleanup the PR to only support and > focus > > >> on Spark 2.x. > > >> > > >> So, that's why I'm calling for a vote: > > >> > > >> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only > > >> [ ] 0 (I don't care ;)) > > >> [ ] -1, I would like to still support Spark 1.x, and so having > > support > > >> of both Spark 1.x and 2.x (please provide specific comment) > > >> > > >> This vote is open for 48 hours (I have the commits ready, just waiting > > >> the end of the vote to push on the PR). > > >> > > >> Thanks ! > > >> Regards > > >> JB > > >> > > > > > > -- > > > Jean-Baptiste Onofré > > > jbono...@apache.org > > > http://blog.nanthrax.net > > > Talend - http://www.talend.com > > > > > > > > > -- > -Ben >
