I think so ;)
Regards
JB
On 11/10/2017 09:29 AM, Reuven Lax wrote:
Sounds good. I doubt we will have much opposition from users, in which case
Beam 2.3.0 can deprecate Spark 1.x
On Thu, Nov 9, 2017 at 11:54 PM, Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:
Hi all,
thanks a lot for all your feedback.
The trend is about to upgrade to Spark 2.x and drop Spark 1.x support.
However, some of you (especially Reuven and Robert) commented that users
have to be pinged as well. It makes perfect sense, and it was my intention.
I propose the following action plan:
- from the technical front, currently, I have two private branches ready:
one with Spark 1.x & Spark 2.x support (with a common module and three
artifacts), another one with an upgrade to Spark 2.x (dropping 1.x). I will
merge the later on the PR.
- I will forward the vote e-mail to the user mailing list, hopefully we
will have user feedback.
Thanks again,
Regards
JB
On 11/08/2017 08:27 AM, Jean-Baptiste Onofré wrote:
Hi all,
as you might know, we are working on Spark 2.x support in the Spark
runner.
I'm working on a PR about that:
https://github.com/apache/beam/pull/3808
Today, we have something working with both Spark 1.x and 2.x from a code
standpoint, but I have to deal with dependencies. It's the first step of
the update as I'm still using RDD, the second step would be to support
dataframe (but for that, I would need PCollection elements with schemas,
that's another topic on which Eugene, Reuven and I are discussing).
However, as all major distributions now ship Spark 2.x, I don't think
it's required anymore to support Spark 1.x.
If we agree, I will update and cleanup the PR to only support and focus
on Spark 2.x.
So, that's why I'm calling for a vote:
[ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
[ ] 0 (I don't care ;))
[ ] -1, I would like to still support Spark 1.x, and so having support
of both Spark 1.x and 2.x (please provide specific comment)
This vote is open for 48 hours (I have the commits ready, just waiting
the end of the vote to push on the PR).
Thanks !
Regards
JB
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
