+1 from me. However let's notify users@ first. If we do get a lot of pushback from users (which I doubt we will), we might reconsider dropping Spark 1 support.
On Thu, Nov 9, 2017 at 11:05 AM, Kenneth Knowles <k...@google.com.invalid> wrote: > +1 from me, with a friendly deprecation process > > I am convinced by the following: > > - We don't have the resources to make both great, and anyhow it isn't > worth it > - People keeping up with Beam releases are likely to be keeping up with > Spark as well > - Spark 1 users already have a Spark 1 runner for Beam and can keep using > it (and we don't actually lose the ability to update it in a pinch) > - Key features like portability (hence Python) will be some time so we > should definitely not waste effort building that feature with Spark 1 in > mind > > I think it makes sense to communicate with email to users@ and in the > release notes of 2.2.0. That communication should be specific and indicate > whether we are planning to merely not work on it anymore or actually remove > it in 2.3.0. > > Kenn > > On Thu, Nov 9, 2017 at 6:35 AM, Amit Sela <amitsel...@gmail.com> wrote: > > > +1 for dropping Spark 1 support. > > I don't think we have enough users to justify supporting both, and its > been > > a long time since this idea originally came-up (when Spark2 wasn't > stable) > > and now Spark 2 is standard in all Hadoop distros. > > As for switching to the Dataframe API, as long as Spark 2 doesn't support > > scanning through the state periodically (even if no data for a key), > > watermarks won't fire keys that didn't see updates. > > > > On Thu, Nov 9, 2017 at 9:12 AM Thomas Weise <t...@apache.org> wrote: > > > > > +1 (non-binding) for dropping 1.x support > > > > > > I don't have the impression that there is significant adoption for Beam > > on > > > Spark 1.x ? A stronger Spark runner that works well on 2.x will be > better > > > for Beam adoption than a runner that has to compromise due to 1.x > > baggage. > > > Development efforts can go into improving the runner. > > > > > > Thanks, > > > Thomas > > > > > > > > > On Thu, Nov 9, 2017 at 4:08 AM, Srinivas Reddy < > > srinivas96all...@gmail.com > > > > > > > wrote: > > > > > > > +1 > > > > > > > > > > > > > > > > -- > > > > Srinivas Reddy > > > > > > > > http://mrsrinivas.com/ > > > > > > > > > > > > (Sent via gmail web) > > > > > > > > On 8 November 2017 at 14:27, Jean-Baptiste Onofré <j...@nanthrax.net> > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > as you might know, we are working on Spark 2.x support in the Spark > > > > runner. > > > > > > > > > > I'm working on a PR about that: > > > > > > > > > > https://github.com/apache/beam/pull/3808 > > > > > > > > > > Today, we have something working with both Spark 1.x and 2.x from a > > > code > > > > > standpoint, but I have to deal with dependencies. It's the first > step > > > of > > > > > the update as I'm still using RDD, the second step would be to > > support > > > > > dataframe (but for that, I would need PCollection elements with > > > schemas, > > > > > that's another topic on which Eugene, Reuven and I are discussing). > > > > > > > > > > However, as all major distributions now ship Spark 2.x, I don't > think > > > > it's > > > > > required anymore to support Spark 1.x. > > > > > > > > > > If we agree, I will update and cleanup the PR to only support and > > focus > > > > on > > > > > Spark 2.x. > > > > > > > > > > So, that's why I'm calling for a vote: > > > > > > > > > > [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only > > > > > [ ] 0 (I don't care ;)) > > > > > [ ] -1, I would like to still support Spark 1.x, and so having > > > support > > > > > of both Spark 1.x and 2.x (please provide specific comment) > > > > > > > > > > This vote is open for 48 hours (I have the commits ready, just > > waiting > > > > the > > > > > end of the vote to push on the PR). > > > > > > > > > > Thanks ! > > > > > Regards > > > > > JB > > > > > -- > > > > > Jean-Baptiste Onofré > > > > > jbono...@apache.org > > > > > http://blog.nanthrax.net > > > > > Talend - http://www.talend.com > > > > > > > > > > > > > > >
