+1 on moving to Spark 2.x - I think we delayed this way too long now and
there will always be some awesome feature that we'd want to support on
older Spark versions too.
Regards,
Matthias
On 1/6/2017 9:41 PM, Mike Dusenberry wrote:
Well to be fair, a user can still use the Python DSL with the SystemML 0.11
release by using `pip install -e src/main/python`. We just didn't place a
separate Python binary on the release website. Keep in mind as well that
once we release the next release with Spark 2.x support, a Spark 1.6 will
not be able to use `pip install systemml` anyway, as that PyPy package will
have been updated to the latest Spark 2.0 release.
I'm concerned that we are moving too slowly as a project without a clear
set of users holding us back on 1.6. Thoughts?
--
Michael W. Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry
On Fri, Jan 6, 2017 at 12:33 PM, Acs S <ac...@yahoo.com.invalid> wrote:
With SystemML 0.11 release we don't have Python DSL support on Spark
1.6.To have Python DSL on Spark 1.6 support we have to release SystemML
with Pip Install artifact, which was planned for SystemML 0.12.
I understand Spark 2.0 has released, at the same time there are Spark 1.6
users who will not able to use SystemML with Python DSL if we simply move
to Spark 2.0 without this artifact as planned. Understand SystemML 0.12 has
been delayed due to holiday period, but lets not get panicked and jump the
hops.
-Arvind
From: Deron Eriksson <deroneriks...@gmail.com>
To: dev@systemml.incubator.apache.org
Cc: Acs S <ac...@yahoo.com>
Sent: Friday, January 6, 2017 12:21 PM
Subject: Re: Time To Merge Spark 2.0 Support PR
I would prefer to move forward with Spark 2 now.
Deron
On Fri, Jan 6, 2017 at 12:15 PM, Mike Dusenberry <dusenberr...@gmail.com>
wrote:
I vote that we scratch that release, and move forward with adding Spark
2.x
support now, and then release on February 1st as discussed on another
thread this week. Otherwise, our PR board will continue to be backed up
without real benefit to the user.
- Mike
--
Michael W. Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry
On Fri, Jan 6, 2017 at 12:03 PM, Acs S <ac...@yahoo.com.invalid> wrote:
We are trying to get SystemML 0.12 release out, which slowed down due
to
holiday period in last 2+ weeksFinal verification pending from Glen
(Niketan, Berthed) related to run wrappers related code with 80GB data.
Once this get verified we should be in position to release 0.12 and
then
can plan to get PR related to Spark 2.0 merged.
-Arvind
From: Mike Dusenberry <dusenberr...@gmail.com>
To: dev <dev@systemml.incubator.apache.org>
Sent: Friday, January 6, 2017 11:54 AM
Subject: Time To Merge Spark 2.0 Support PR
Hi to the SystemML community!
As you may know, SystemML currently only supports Spark 1.6, and does
not
yet support Spark 2.x. However, there is an open PR (
https://github.com/apache/incubator-systemml/pull/202) that replaces
Spark
1.6 support with Spark 2.x support.
Spark 2.0.0 was released on July 26, 2016 (
https://spark.apache.org/news/spark-2-0-0-released.html), and 2.1.0 is
now
available (https://spark.apache.org/news/spark-2-1-0-released.html).
I think it is time to merge the above PR into the master branch of the
project in order to move SystemML onto Spark 2.x.
Thoughts? If no objections, I'd like to merge next week.
Cheers!
- Mike
--
Michael W. Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry
--
Deron Eriksson
Spark Technology Center
http://www.spark.tc/
