well, after trying to run our perftest suite with Spark 2.3 and Spark 2.2 this seems to be more complicated. Although the version update from 4.5.3 to 4.7.1 solved the problem with Spark 2.3 (which uses 4.7.1), SystemML would no longer be backwards compatible with Spark 2.2 and 2.1 (which use 4.5.3) because ANTLR checks bidirectional mismatches.Unfortunately, removing ANTLR from our jar does not help because the above versions are binary incompatible.
We need to hold off the release until we decided whether (1) we directly release for Spark 2.3 and drop 2.2 and 2.1, or (2) we release for Spark 2.2 and 2.1 with a subsequent ANTLR change and release for 2.3. Regards, Matthias On Thu, Mar 8, 2018 at 5:38 PM, Niketan Pansare <npan...@us.ibm.com> wrote: > +1. That will preserve current behavior on older Spark versions too. > > > On Mar 8, 2018, at 5:24 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > +1 on upgrading > > -------- Original message --------From: Matthias Boehm < > mboe...@gmail.com> Date: 3/8/18 5:19 PM (GMT-08:00) To: > dev@systemml.apache.org Subject: Re: Release Planning > > related to Spark 2.3, we might want to update our ANTLR version because > > with Spark 2.3 every parsed DML script (i.e., multiple times with > imported > > DML files) produces the following warning: > > > > "ANTLR Tool version 4.5.3 used for code generation does not match the > > current runtime version 4.7" > > > > Regards, > > Matthias > > > >> On Thu, Mar 1, 2018 at 5:22 PM, Matthias Boehm <mboe...@gmail.com> > wrote: > >> > >> Hi all, > >> > >> I'm sure you've seen that Spark 2.3 just got released. This lines up > >> beautifully with our own SystemML 1.1 release. Accordingly, I would > >> recommend to use Spark 2.3 for our due diligence algorithm-level Q/A. > How > >> about we shoot for an RC1 by March 12? This should give enough time to > run > >> over reasonably large data and fix all related issues. > >> > >> Regards, > >> Matthias > >> > >>> On Tue, Feb 6, 2018 at 12:51 PM, Matthias Boehm <mboe...@gmail.com> > wrote: > >>> > >>> yes, absolutely. Here is a list of new features and improvements - > please > >>> feel free to extend as needed: > >>> > >>> 1) Extended Caffe2DML and Keras2DML APIs > >>> 2) Support for large-dense blocks >16GB in CP > >>> 3) New builtin functions ifelse, xor, as well as and/or/not over > matrices > >>> 4) Single-precision support for native conv2d and mm operations. > >>> 5) Performance features and correctness of ultra-sparse operations > >>> 6) Codegen: new plan cache, nary cbind > >>> 7) Parfor: result merge with accumulators +=, reduced initialization > >>> overhead > >>> 8) Compiler improvements: avoid unnecessary spark instructions, > corrected > >>> memory estimates > >>> > >>> Until the RC, I'd like to support all deep learning builtin functions > in > >>> codegen, add a couple of pending parfor improvements, and potentially > do a > >>> first cut of the compiler/runtime integration for parameter servers. > >>> > >>> Regards, > >>> Matthias > >>> > >>> > >>> > >>> On Tue, Feb 6, 2018 at 12:36 PM, Niketan Pansare <npan...@us.ibm.com> > >>> wrote: > >>> > >>>> +1. > >>>> > >>>> We should consider including single precision native BLAS in the > release > >>>> notes as well. If possible, we should add JNI wrappers for PowerPC, > Windows > >>>> and Mac too in this release. > >>>> > >>>>> On Feb 6, 2018, at 12:27 PM, Berthold Reinwald <reinw...@us.ibm.com> > >>>> wrote: > >>>>> > >>>>> sure. > >>>>> > >>>>> Makes sense. Codegen and Keras2DML made good progress, and many other > >>>>> fixes/improvements. > >>>>> > >>>>> What else do we want time/track/highlight for it? > >>>>> > >>>>> Regards, > >>>>> Berthold Reinwald > >>>>> IBM Almaden Research Center > >>>>> office: (408) 927 2208; T/L: 457 2208 > >>>>> e-mail: reinw...@us.ibm.com > >>>>> > >>>>> > >>>>> > >>>>> From: Matthias Boehm <mboe...@gmail.com> > >>>>> To: dev@systemml.apache.org > >>>>> Date: 02/05/2018 11:05 PM > >>>>> Subject: Release Planning > >>>>> > >>>>> > >>>>> > >>>>> Hi all, > >>>>> > >>>>> since our 1.0 release in Dec, we already got a number enhancements > and > >>>> new > >>>>> features in, so I think it would be good to discuss the timeline for > >>>> our > >>>>> next SystemML 1.1 release. How about, we target mid March for a first > >>>> RC? > >>>>> Also, Berthold would you be willing to serve again as the release > >>>> manager? > >>>>> > >>>>> Regards, > >>>>> Matthias > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >> > >
