Re: Fw: [DISCUSS] SystemML with Spark 2.0 support and roadmap

2016-08-17 Thread Deron Eriksson 
+1
Continuing to support Spark 1.4/1.6 for now while setting a cutover date
for 2.0 sounds like a great idea. This allows for the creation of a really
solid release for 1.x, which greatly benefits SystemML users using Spark
1.x. It also gives these users a general date that they can use to plan
migration to Spark 2.0 when that becomes the SystemML standard so that they
can benefit from the latest improvements to the project.

Deron


On Wed, Aug 17, 2016 at 2:32 PM, Acs S  wrote:

> Seems, mail is not retaining format. I am attaching same text through PDF
> file.
> If there is any other better option please let me know.
>
>
> -Arvind
>
>
>
>
>
> - Forwarded Message -
> *From:* Acs S 
> *To:* Dev 
> *Sent:* Wednesday, August 17, 2016 2:18 PM
> *Subject:* [DISCUSS] SystemML with Spark 2.0 support and roadmap
>
>
> Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be
> uptodate with latest version of Spark. This brings us a challenge to
> support our consumers until they move to Spark 2.0.Based on some
> brainstorming, I can propose following options to keep SystemML being
> supported on latest Spark version quickly.
>
> Supporting SystemML on Spark 1.x We can continue to support SystemML on
> Spark 1.x code base for short period of time by adding fixes and features
> on main branch.  We will release SystemML with support to Spark 1.x next
> version (0.11) around beginning of Oct 2016 (Lets target for Oct 1st 2016)
> Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark
> 2.0, we can make SystemML on Spark 2.0 immediately based on branch created
> on top of latest master branch code. Glen has some prototype code to
> transform SystemML code to be compatible with Spark 2.0, he can merge his
> code with new branch targeted to support SystemML on Spark 2.0 This would
> be "Preview" version code, and we can update it on frequent basis (on
> bi-monthy basis).  Supporting SystemML on Spark 2.0 We will have full
> support of SystemML on Spark 2.0 before end of year 2016. We will formalize
> release date by end of Sept 2016. At the same time we will discuss if we
> can move support of SystemML on Spark 1.x to maintenance mode (Only
> required bug fixes will be merged from main branch) or we need to support
> both SystemML on Spark 2.0 and Spark 1.x for some additional time.
> SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep
> Learning (Library of Network layers?) - Frame - New MLContext API-
> Python DSL integration (Preview) - Compressed Linear Algebra (Preview) -
> Hydra R integration - New Algorithms (?)
> 0.12 (Spark 2.0)  (Targeted to 4Q 2016) - GPU support (Local
> mode/Distributed mode?)
> - New Algorithms (?)
> Please feel free to comment on support and roadmap points.
>
>
> -Arvind
>
>

[DISCUSS] SystemML with Spark 2.0 support and roadmap

2016-08-17 Thread Acs S 

Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be uptodate 
with latest version of Spark. This brings us a challenge to support our 
consumers until they move to Spark 2.0.Based on some brainstorming, I can 
propose following options to keep SystemML being supported on latest Spark 
version quickly.

 Supporting SystemML on Spark 1.x We can continue to support SystemML on Spark 
1.x code base for short period of time by adding fixes and features on main 
branch.  We will release SystemML with support to Spark 1.x next version (0.11) 
around beginning of Oct 2016 (Lets target for Oct 1st 2016)
 Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark 2.0, 
we can make SystemML on Spark 2.0 immediately based on branch created on top of 
latest master branch code. Glen has some prototype code to transform SystemML 
code to be compatible with Spark 2.0, he can merge his code with new branch 
targeted to support SystemML on Spark 2.0 This would be "Preview" version code, 
and we can update it on frequent basis (on bi-monthy basis).  Supporting 
SystemML on Spark 2.0 We will have full support of SystemML on Spark 2.0 before 
end of year 2016. We will formalize release date by end of Sept 2016. At the 
same time we will discuss if we can move support of SystemML on Spark 1.x to 
maintenance mode (Only required bug fixes will be merged from main branch) or 
we need to support both SystemML on Spark 2.0 and Spark 1.x for some additional 
time. 
SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep Learning 
(Library of Network layers?) - Frame - New MLContext API        - Python DSL 
integration (Preview) - Compressed Linear Algebra (Preview) - Hydra R 
integration - New Algorithms (?)
 0.12 (Spark 2.0)  (Targeted to 4Q 2016) - GPU support (Local mode/Distributed 
mode?)
 - New Algorithms (?)
Please feel free to comment on support and roadmap points.


-Arvind

Re: Calling System ML from sparkR

2016-08-17 Thread Deron Eriksson 
Hi Sourav,

Great question. Work is currently being performed by Alok Singh (see
https://issues.apache.org/jira/browse/SYSTEMML-860) regarding this topic.

Deron


On Mon, Aug 15, 2016 at 9:31 AM, Sourav Mazumder <
sourav.mazumde...@gmail.com> wrote:

> Hi,
>
> Is there any work going on to call System ML dml scripts form SparkR using
> R syntax ?
>
> I understand it was possible using BigR (available in IBM Big Insights).
>
> Wondering whether something similar can be achieved from Spark R.
>
> Regards,
> Sourav
>

Re: Preview tag, was Re: [2/2] incubator-systemml git commit: Preparing SystemML development version 0.11.0-incubating-SNAPSHOT.

2016-08-17 Thread Luciano Resende 
Great, thanks.

On Wed, Aug 17, 2016 at 2:32 PM,  wrote:

> Thanks, Luciano for pointing this out. As you mentioned, the intent was
> definitely just to tag a commit that was known to be stable on the Spark
> 1.x line. I've deleted the existing tag, and created a new
> "spark-1.x-stable" tag simply pointing to a previous commit that was tested
> on Spark 1.x.
>
> Thanks!
>
> -Mike
>
> --
>
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> Sent from my iPhone.
>
>
> > On Aug 17, 2016, at 11:18 AM, Luciano Resende 
> wrote:
> >
> > -1
> >
> > Sorry Folks, this isn't a voted release and thus creating a tag without
> > SNAPSHOT is not valid. Please delete this tag.
> >
> > If what is wanted is to have a stable point in the codebase where folks
> can
> > go back if a release is needed for 1.x, then just create a branch/tag
> with
> > a descriptive name (e.g. spark_1.x_stable).
> >
> > If you actually want a release, there is a need to follow the Apache
> > Release vote process (e.g. see
> > https://www.mail-archive.com/dev%40spark.apache.org/msg14223.html for
> Spark
> > preview release vote)
> >
> > Thanks
> >
> >
> >> On Wed, Aug 17, 2016 at 1:21 PM,  wrote:
> >>
> >> Preparing SystemML development version 0.11.0-incubating-SNAPSHOT.
> >>
> >>
> >> Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
> >> Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/
> >> commit/b6bde0d4
> >> Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/
> >> tree/b6bde0d4
> >> Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/
> >> diff/b6bde0d4
> >>
> >> Branch: refs/heads/master
> >> Commit: b6bde0d4599d551cf1dc903c72662888abc22787
> >> Parents: 05b6da0
> >> Author: Mike Dusenberry 
> >> Authored: Wed Aug 17 10:17:52 2016 -0700
> >> Committer: Mike Dusenberry 
> >> Committed: Wed Aug 17 10:17:52 2016 -0700
> >>
> >> --
> >> pom.xml | 4 ++--
> >> 1 file changed, 2 insertions(+), 2 deletions(-)
> >> --
> >>
> >>
> >> http://git-wip-us.apache.org/repos/asf/incubator-systemml/
> >> blob/b6bde0d4/pom.xml
> >> --
> >> diff --git a/pom.xml b/pom.xml
> >> index aba8808..a4c66a1 100644
> >> --- a/pom.xml
> >> +++ b/pom.xml
> >> @@ -25,7 +25,7 @@
> >>18
> >>
> >>org.apache.systemml
> >> -   0.11.0-incubating-preview
> >> +   0.11.0-incubating-SNAPSHOT
> >>systemml
> >>jar
> >>SystemML
> >> @@ -41,7 +41,7 @@
> >>scm:git:g...@github.com:apache/incubator-
> >> systemml
> >>scm:git:h
> >> ttps://git-wip-us.apache.org/repos/asf/incubator-systemml >> developerConnection>
> >>https://git-wip-us.apache.org/repos/asf?p=
> >> incubator-systemml.git
> >> -   0.11.0-incubating-preview
> >> +   HEAD
> >>
> >>
> >>JIRA
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
>



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Preview tag, was Re: [2/2] incubator-systemml git commit: Preparing SystemML development version 0.11.0-incubating-SNAPSHOT.

2016-08-17 Thread dusenberrymw 
Thanks, Luciano for pointing this out. As you mentioned, the intent was 
definitely just to tag a commit that was known to be stable on the Spark 1.x 
line. I've deleted the existing tag, and created a new "spark-1.x-stable" tag 
simply pointing to a previous commit that was tested on Spark 1.x. 

Thanks!

-Mike

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Aug 17, 2016, at 11:18 AM, Luciano Resende  wrote:
> 
> -1
> 
> Sorry Folks, this isn't a voted release and thus creating a tag without
> SNAPSHOT is not valid. Please delete this tag.
> 
> If what is wanted is to have a stable point in the codebase where folks can
> go back if a release is needed for 1.x, then just create a branch/tag with
> a descriptive name (e.g. spark_1.x_stable).
> 
> If you actually want a release, there is a need to follow the Apache
> Release vote process (e.g. see
> https://www.mail-archive.com/dev%40spark.apache.org/msg14223.html for Spark
> preview release vote)
> 
> Thanks
> 
> 
>> On Wed, Aug 17, 2016 at 1:21 PM,  wrote:
>> 
>> Preparing SystemML development version 0.11.0-incubating-SNAPSHOT.
>> 
>> 
>> Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
>> Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/
>> commit/b6bde0d4
>> Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/
>> tree/b6bde0d4
>> Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/
>> diff/b6bde0d4
>> 
>> Branch: refs/heads/master
>> Commit: b6bde0d4599d551cf1dc903c72662888abc22787
>> Parents: 05b6da0
>> Author: Mike Dusenberry 
>> Authored: Wed Aug 17 10:17:52 2016 -0700
>> Committer: Mike Dusenberry 
>> Committed: Wed Aug 17 10:17:52 2016 -0700
>> 
>> --
>> pom.xml | 4 ++--
>> 1 file changed, 2 insertions(+), 2 deletions(-)
>> --
>> 
>> 
>> http://git-wip-us.apache.org/repos/asf/incubator-systemml/
>> blob/b6bde0d4/pom.xml
>> --
>> diff --git a/pom.xml b/pom.xml
>> index aba8808..a4c66a1 100644
>> --- a/pom.xml
>> +++ b/pom.xml
>> @@ -25,7 +25,7 @@
>>18
>>
>>org.apache.systemml
>> -   0.11.0-incubating-preview
>> +   0.11.0-incubating-SNAPSHOT
>>systemml
>>jar
>>SystemML
>> @@ -41,7 +41,7 @@
>>scm:git:g...@github.com:apache/incubator-
>> systemml
>>scm:git:h
>> ttps://git-wip-us.apache.org/repos/asf/incubator-systemml> developerConnection>
>>https://git-wip-us.apache.org/repos/asf?p=
>> incubator-systemml.git
>> -   0.11.0-incubating-preview
>> +   HEAD
>>
>>
>>JIRA
> 
> 
> -- 
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/

Preview tag, was Re: [2/2] incubator-systemml git commit: Preparing SystemML development version 0.11.0-incubating-SNAPSHOT.

2016-08-17 Thread Luciano Resende 
-1

Sorry Folks, this isn't a voted release and thus creating a tag without
SNAPSHOT is not valid. Please delete this tag.

If what is wanted is to have a stable point in the codebase where folks can
go back if a release is needed for 1.x, then just create a branch/tag with
a descriptive name (e.g. spark_1.x_stable).

If you actually want a release, there is a need to follow the Apache
Release vote process (e.g. see
https://www.mail-archive.com/dev%40spark.apache.org/msg14223.html for Spark
preview release vote)

Thanks


On Wed, Aug 17, 2016 at 1:21 PM,  wrote:

> Preparing SystemML development version 0.11.0-incubating-SNAPSHOT.
>
>
> Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
> Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/
> commit/b6bde0d4
> Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/
> tree/b6bde0d4
> Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/
> diff/b6bde0d4
>
> Branch: refs/heads/master
> Commit: b6bde0d4599d551cf1dc903c72662888abc22787
> Parents: 05b6da0
> Author: Mike Dusenberry 
> Authored: Wed Aug 17 10:17:52 2016 -0700
> Committer: Mike Dusenberry 
> Committed: Wed Aug 17 10:17:52 2016 -0700
>
> --
>  pom.xml | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> --
>
>
> http://git-wip-us.apache.org/repos/asf/incubator-systemml/
> blob/b6bde0d4/pom.xml
> --
> diff --git a/pom.xml b/pom.xml
> index aba8808..a4c66a1 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -25,7 +25,7 @@
> 18
> 
> org.apache.systemml
> -   0.11.0-incubating-preview
> +   0.11.0-incubating-SNAPSHOT
> systemml
> jar
> SystemML
> @@ -41,7 +41,7 @@
> scm:git:g...@github.com:apache/incubator-
> systemml
> scm:git:h
> ttps://git-wip-us.apache.org/repos/asf/incubator-systemml developerConnection>
> https://git-wip-us.apache.org/repos/asf?p=
> incubator-systemml.git
> -   0.11.0-incubating-preview
> +   HEAD
> 
> 
> JIRA
>
>


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [DISCUSS] Migration to Spark 2.0.0

2016-08-17 Thread dusenberrymw 
Yes, I think this approach sounds great.  To that end, I created a new tag 
"0.11.0-incubating-preview" that points to a specific commit that contains new 
features that will be in the 0.11 release with specific support for the Spark 
1.x line.


- Mike

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Aug 16, 2016, at 4:44 PM, Frederick R Reiss  wrote:
> 
> I think the approach Glenn proposes here is fine.
> 
> Fred
> 
> Deron Eriksson ---08/16/2016 02:41:51 PM---Hi Glenn, I am fine with this 
> approach. If this approach is taken, I would like to
> 
> From: Deron Eriksson 
> To: dev@systemml.incubator.apache.org
> Date: 08/16/2016 02:41 PM
> Subject: Re: [DISCUSS] Migration to Spark 2.0.0
> 
> 
> 
> 
> Hi Glenn,
> 
> I am fine with this approach. If this approach is taken, I would like to
> set the documentation version in _config.yml to 0.10.x before the project
> is tagged (I recently set it to 0.11).
> 
> Deron
> 
> 
> On Thu, Aug 11, 2016 at 3:40 PM, Glenn Weidner  wrote:
> 
> > I would like to propose an alternative to supporting Spark 2.0 and Spark
> > 1.x within single stream.
> >
> > 1) Capture snapshot and establish label of current Apache SystemML master
> > which includes new features added since 0.10.0 release.
> >
> > 2) After step 1 completed, enable master to move forward with support for
> > Spark 2.x only.
> >
> > This is similar to what Fred initially proposed except step 1 would not
> > involve a separate release. The 0.11 release of Apache SystemML would be
> > compatible for Spark 2.0 and Scala 2.11.
> >
> > Thanks,
> > Glenn
> >
> > [image: Inactive hide details for Glenn Weidner---08/08/2016 03:33:43
> > PM---As a preliminary experiment in attempt to compile against bo]Glenn
> > Weidner---08/08/2016 03:33:43 PM---As a preliminary experiment in attempt
> > to compile against both Spark 2.0.0 and Spark 1.6.2 from same
> >
> > From: Glenn Weidner/Silicon Valley/IBM@IBMUS
> > To: dev@systemml.incubator.apache.org
> > Date: 08/08/2016 03:33 PM
> > Subject: Re: [DISCUSS] Migration to Spark 2.0.0
> > --
> >
> >
> >
> > As a preliminary experiment in attempt to compile against both Spark 2.0.0
> > and Spark 1.6.2 from same code base, I made another set of changes for
> > comparison against previous proposed changes for [SYSTEMML-776].
> > This experimental set can be viewed here:
> >
> > *https://github.com/gweidner/incubator-systemml/commit/0611f0c197e4a0e816b3325093168bc5162d62c0*
> > 
> >
> > This compiles against Spark 2.0.0 and Spark 1.6.2 except for fit/transform
> > overrides in LogisticRegression.scala due to:
> > SPARK-14500 Accept Dataset[] instead of DataFrame in MLlib APIs
> >
> > Detailed code comments and suggestions to try out can be made in the
> > branch commit instead of this mail thread.
> >
> > Thanks,
> > Glenn
> >
> > Deron Eriksson ---08/05/2016 02:02:10 PM---I am open to the idea of
> > supporting Spark 2 and Spark<2 concurrently if someone shows that it can be
> >
> > From: Deron Eriksson 
> > To: dev@systemml.incubator.apache.org
> > Date: 08/05/2016 02:02 PM
> > Subject: Re: [DISCUSS] Migration to Spark 2.0.0
> > --
> >
> >
> >
> > I am open to the idea of supporting Spark 2 and Spark<2 concurrently if
> > someone shows that it can be accomplished with minimal inconvenience.
> >
> > However, I would lean towards Fred's approach (Spark 1.6 release followed
> > shortly by a Spark 2 release). If possible, I want to be able to focus most
> > of our efforts towards the future rather than the past.
> >
> > Deron
> >
> >
> > On Thu, Aug 4, 2016 at 10:59 AM, Luciano Resende 
> > wrote:
> >
> > > That was going to be my suggestion... In Zeppelin, we just introduced
> > > support for different versions of scala and added support for spark 2.0
> > > based on profiles and a bit of reflections...
> > >
> > > Do we have to do anything related to Scala versions as well ?
> > >
> > > On Thursday, August 4, 2016, Matthias Boehm  wrote:
> > >
> > > > I would recommend to start an investigation if we could support both
> > the
> > > > 1.x and 2.x lines with a single code base. It seems feasible to
> > refactor
> > > > the code a bit, compile against 2.0 (or with profiles), and run on
> > either
> > > > 1.6 or 2.0. For example, by creating a wrapper that implements both
> > > > Iterable and Iterator, we could overcome the Iterator API change as
> > shown
> > > > by our LazyIterableIterator which did not require any change in related
> > > > functions. Btw, we did the same for MRv1 and Yarn by ensuring that on
> > > MRv1,
> > > > we don't touch Yarn related APIs. Similarly on Spark, we already
> > support
> > > > both legacy and