just a quick update: the ANTLR issue and most other things that came up
during QA are fixed now, except for performance issues with stratified
stats for which I need some more time (SYSTEMML-2181 tracks the open
issues). So it's probably a good idea to postpone the RC1 for a few days to
avoid unnecessary release efforts.

Regards,
Matthias

On Sun, Mar 11, 2018 at 5:43 PM, Matthias Boehm <mboe...@gmail.com> wrote:

> well, after trying to run our perftest suite with Spark 2.3 and Spark 2.2
> this seems to be more complicated. Although the version update from 4.5.3
> to 4.7.1 solved the problem with Spark 2.3 (which uses 4.7.1), SystemML
> would no longer be backwards compatible with Spark 2.2 and 2.1 (which use
> 4.5.3) because ANTLR checks bidirectional mismatches.Unfortunately,
> removing ANTLR from our jar does not help because the above versions are
> binary incompatible.
>
> We need to hold off the release until we decided whether (1) we directly
> release for Spark 2.3 and drop 2.2 and 2.1, or (2) we release for Spark 2.2
> and 2.1 with a subsequent ANTLR change and release for 2.3.
>
> Regards,
> Matthias
>
> On Thu, Mar 8, 2018 at 5:38 PM, Niketan Pansare <npan...@us.ibm.com>
> wrote:
>
>> +1. That will preserve current behavior on older Spark versions too.
>>
>> > On Mar 8, 2018, at 5:24 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> >
>> > +1 on upgrading
>> > -------- Original message --------From: Matthias Boehm <
>> mboe...@gmail.com> Date: 3/8/18  5:19 PM  (GMT-08:00) To:
>> dev@systemml.apache.org Subject: Re: Release Planning
>> > related to Spark 2.3, we might want to update our ANTLR version because
>> > with Spark 2.3 every parsed DML script (i.e., multiple times with
>> imported
>> > DML files) produces the following warning:
>> >
>> > "ANTLR Tool version 4.5.3 used for code generation does not match the
>> > current runtime version 4.7"
>> >
>> > Regards,
>> > Matthias
>> >
>> >> On Thu, Mar 1, 2018 at 5:22 PM, Matthias Boehm <mboe...@gmail.com>
>> wrote:
>> >>
>> >> Hi all,
>> >>
>> >> I'm sure you've seen that Spark 2.3 just got released. This lines up
>> >> beautifully with our own SystemML 1.1 release. Accordingly, I would
>> >> recommend to use Spark 2.3 for our due diligence algorithm-level Q/A.
>> How
>> >> about we shoot for an RC1 by March 12? This should give enough time to
>> run
>> >> over reasonably large data and fix all related issues.
>> >>
>> >> Regards,
>> >> Matthias
>> >>
>> >>> On Tue, Feb 6, 2018 at 12:51 PM, Matthias Boehm <mboe...@gmail.com>
>> wrote:
>> >>>
>> >>> yes, absolutely. Here is a list of new features and improvements -
>> please
>> >>> feel free to extend as needed:
>> >>>
>> >>> 1) Extended Caffe2DML and Keras2DML APIs
>> >>> 2) Support for large-dense blocks >16GB in CP
>> >>> 3) New builtin functions ifelse, xor, as well as and/or/not over
>> matrices
>> >>> 4) Single-precision support for native conv2d and mm operations.
>> >>> 5) Performance features and correctness of ultra-sparse operations
>> >>> 6) Codegen: new plan cache, nary cbind
>> >>> 7) Parfor: result merge with accumulators +=, reduced initialization
>> >>> overhead
>> >>> 8) Compiler improvements: avoid unnecessary spark instructions,
>> corrected
>> >>> memory estimates
>> >>>
>> >>> Until the RC, I'd like to support all deep learning builtin functions
>> in
>> >>> codegen, add a couple of pending parfor improvements, and potentially
>> do a
>> >>> first cut of the compiler/runtime integration for parameter servers.
>> >>>
>> >>> Regards,
>> >>> Matthias
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Feb 6, 2018 at 12:36 PM, Niketan Pansare <npan...@us.ibm.com>
>> >>> wrote:
>> >>>
>> >>>> +1.
>> >>>>
>> >>>> We should consider including single precision native BLAS in the
>> release
>> >>>> notes as well. If possible, we should add JNI wrappers for PowerPC,
>> Windows
>> >>>> and Mac too in this release.
>> >>>>
>> >>>>> On Feb 6, 2018, at 12:27 PM, Berthold Reinwald <reinw...@us.ibm.com
>> >
>> >>>> wrote:
>> >>>>>
>> >>>>> sure.
>> >>>>>
>> >>>>> Makes sense. Codegen and Keras2DML made good progress, and many
>> other
>> >>>>> fixes/improvements.
>> >>>>>
>> >>>>> What else do we want time/track/highlight for it?
>> >>>>>
>> >>>>> Regards,
>> >>>>> Berthold Reinwald
>> >>>>> IBM Almaden Research Center
>> >>>>> office: (408) 927 2208; T/L: 457 2208
>> >>>>> e-mail: reinw...@us.ibm.com
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> From:   Matthias Boehm <mboe...@gmail.com>
>> >>>>> To:     dev@systemml.apache.org
>> >>>>> Date:   02/05/2018 11:05 PM
>> >>>>> Subject:        Release Planning
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Hi all,
>> >>>>>
>> >>>>> since our 1.0 release in Dec, we already got a number enhancements
>> and
>> >>>> new
>> >>>>> features in, so I think it would be good to discuss the timeline for
>> >>>> our
>> >>>>> next SystemML 1.1 release. How about, we target mid March for a
>> first
>> >>>> RC?
>> >>>>> Also, Berthold would you be willing to serve again as the release
>> >>>> manager?
>> >>>>>
>> >>>>> Regards,
>> >>>>> Matthias
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>
>

Reply via email to