Re: Draft Release Notes 1.1.0

[Matthias Boehm]

thanks for the initial draft and extensions - I would remove internals
#2/#3 because they are still open, move the other internals to performance,
and include (or extend) the following:

* codegen extensions (operation support, extended optimizer, see
SYSTEMML-2065)
* new accumulator operator += (not just in parfor)
* matrix-matrix multiplication over compressed matrices
* zero row/column matrices and updated operations such as removeEmpty
* logical operator support over matrices AND/OR/NOT/XOR (besides the
bitwise ops)
* new second-order eval builtin function
* extended UDF framework
* performance: sparse left indexing, sparse reshape, ultra-sparse
transpose, ultra-sparse rand, binary in-place operations, sparse relu
backward, maxpooling, sparse im2col, ultra-sparse conv2d, read of
short-wide sparse matrices, avoid unnecessary evictions, lock-free
statistics maintenance, spark cpmm, spark aggregates, spark reshape, spark
binary ops, use common thread pool for multi-threaded cp operations, etc
* improved nnz maintenance, runtime propogation and memory management
* robustness for matrices with larger than int dimensions

This also remind me that we should probably make a pass over all new
builtin functions, operators and generalized/extended operations and update
our documentation accordingly.

Regards,
Matthias


On Wed, Mar 28, 2018 at 10:02 PM, Niketan Pansare 
wrote:

> Thanks Berthold. Overall, the draft looks good to me. Few minor additions:
>
> New Builtin Functions
> - Deep learning builtin functions: avg_pool and avg_pool_backward
> - assert
>
> New Layers in the NN library:
> - Average pooling
> - Upsampling
> - Low-rank fully connected
>
> Thanks,
>
> Niketan Pansare
> IBM Almaden Research Center
> E-mail: npansar At us.ibm.com
> http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
>
> [image: Inactive hide details for "Berthold Reinwald" ---03/28/2018
> 04:02:46 PM---Please add/update below release note draft prior to p]"Berthold
> Reinwald" ---03/28/2018 04:02:46 PM---Please add/update below release note
> draft prior to putting it on our website. Please also go to th
>
> From: "Berthold Reinwald" 
> To: dev@systemml.apache.org
> Date: 03/28/2018 04:02 PM
> Subject: Draft Release Notes 1.1.0
> --
>
>
>
> Please add/update below release note draft prior to putting it on our
> website. Please also go to the JIRA release notes link at the end, and
> update version/status if necessary. Thanks.
>
>
> * Release Notes - SystemML - Version SystemML 1.1
>
> New Capabilities/Features
> - Dense matrix blocks >16GB, and operations
> - Support bitwise operators not, and, or, xor, & LShift, Rshift
> - Additional ParFor result aggregation operations
> - UDFs callable in expressions
> - zero rws/columns matrices
> - Extended Caffe2DML and Keras2DML APIs
>
> New Builtin Functions
> - ifelse()
>
> Performance Improvements
> - Ultra-sparse operations
>
> Internals
> - Single-precision support for native conv2d and mm operations.
> - Consolidate replicated compilation chain
> - Generalize Binary Operations to (vector, matrix) Operands
> - Use common thread pool
>
> Bug Fixes
> - in APIs, performance bugs, optimizer, runtime, GPU backend, Spark
> backend.
>
> Experimental
> - Codegen
>
> Deprecate
> - Support for Spark 2.1/2.2 (make switch to newer ANTLR version)
>
> Detailed JIRA release notes are here:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_secure_ReleaseNote.jspa-3Fversion-
> 3D12342282-26styleName-3DText-26projectId-3D12319522-
> 26Create-3DCreate-26atl-5Ftoken-3DA5KQ-2D2QAV-2DT4JA-2DFDED-
> 257C2e97f76f75a466564dba2a7d0bec5ca06bffd66e-257Clout=
> DwIFAg=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_
> Kao_6a6SaOkZrfiSrYZVby0=fku28ex5cFV8MRqEKRuh-_I3UTefK_Qx_YXQD9UfoWU=
> K6GTOjWQiV6jki5_tkBF1tXLL2SM2Jta2ZXFdjBYbQw=
>
>
> Regards,
> Berthold Reinwald
> IBM Almaden Research Center
> office: (408) 927 2208 <%28408%29%20927-2208>; T/L: 457 2208
> e-mail: reinw...@us.ibm.com
>
>
>
>
>

Re: Draft Release Notes 1.1.0

[Niketan Pansare]

Thanks Berthold. Overall, the draft looks good to me. Few minor additions:

New Builtin Functions
- Deep learning builtin functions: avg_pool and avg_pool_backward
- assert

New Layers in the NN library:
- Average pooling
- Upsampling
- Low-rank fully connected

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   "Berthold Reinwald" 
To: dev@systemml.apache.org
Date:   03/28/2018 04:02 PM
Subject:Draft Release Notes 1.1.0



Please add/update below release note draft prior to putting it on our
website. Please also go to the JIRA release notes link at the end, and
update version/status if necessary. Thanks.


* Release Notes - SystemML - Version SystemML 1.1

New Capabilities/Features
- Dense matrix blocks >16GB, and operations
- Support bitwise operators not, and, or, xor, & LShift, Rshift
- Additional ParFor result aggregation operations
- UDFs callable in expressions
- zero rws/columns matrices
- Extended Caffe2DML and Keras2DML APIs

New Builtin Functions
- ifelse()

Performance Improvements
- Ultra-sparse operations

Internals
- Single-precision support for native conv2d and mm operations.
- Consolidate replicated compilation chain
- Generalize Binary Operations to (vector, matrix) Operands
- Use common thread pool

Bug Fixes
- in APIs, performance bugs, optimizer, runtime, GPU backend, Spark
backend.

Experimental
- Codegen

Deprecate
- Support for Spark 2.1/2.2 (make switch to newer ANTLR version)

Detailed JIRA release notes are here:
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_ReleaseNote.jspa-3Fversion-3D12342282-26styleName-3DText-26projectId-3D12319522-26Create-3DCreate-26atl-5Ftoken-3DA5KQ-2D2QAV-2DT4JA-2DFDED-257C2e97f76f75a466564dba2a7d0bec5ca06bffd66e-257Clout=DwIFAg=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=fku28ex5cFV8MRqEKRuh-_I3UTefK_Qx_YXQD9UfoWU=K6GTOjWQiV6jki5_tkBF1tXLL2SM2Jta2ZXFdjBYbQw=



Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinw...@us.ibm.com

Fwd: Contribution to SystemML

[Matthias Boehm]

-- Forwarded message --
From: Matthias Boehm 
Date: Wed, Mar 28, 2018 at 9:34 PM
Subject: Re: Contribution to SystemML
To: Govinda Malavipathirana 


well, first of all sorry that you wasted one of your proposals because it
would have been better to combine them into a single proposal (as commented
on the earlier draft). Unfortunately, I didn't see these duplicates before
the deadline.

Regarding your interest in getting familiar with the internals of SystemML,
that's great to hear. In addition to the existing JIRAs (you're welcome to
leave a comment if you start working on one), I'll create some additional
beginner tasks related to your interests in the next days.

Regards,
Matthias

On Wed, Mar 28, 2018 at 12:14 AM, Govinda Malavipathirana <
mp.govi...@gmail.com> wrote:

> Hi'
> I have submitted 2 proposals for Language .and Parameter Server project.
> In this proposal review period I would like to contribute to systemml for
> further understanding the systemml internals. I'm looking for some guidance
> to select issue that relates/more like to [SYSTEMML-2084] Language and
> Compiler Extension and [SYSTEMML-2089] Extended Keras2DML and Caffe2DML
> Script Generators. Thanks in advanced.
> Best regards,
> Govinda
>

Draft Release Notes 1.1.0

[Berthold Reinwald]

Please add/update below release note draft prior to putting it on our 
website. Please also go to the JIRA release notes link at the end, and 
update version/status if necessary. Thanks.


* Release Notes - SystemML - Version SystemML 1.1

New Capabilities/Features
- Dense matrix blocks >16GB, and operations
- Support bitwise operators not, and, or, xor, & LShift, Rshift
- Additional ParFor result aggregation operations
- UDFs callable in expressions
- zero rws/columns matrices
- Extended Caffe2DML and Keras2DML APIs

New Builtin Functions
- ifelse()

Performance Improvements
- Ultra-sparse operations

Internals
- Single-precision support for native conv2d and mm operations.
- Consolidate replicated compilation chain
- Generalize Binary Operations to (vector, matrix) Operands
- Use common thread pool

Bug Fixes 
- in APIs, performance bugs, optimizer, runtime, GPU backend, Spark 
backend.

Experimental
- Codegen

Deprecate
- Support for Spark 2.1/2.2 (make switch to newer ANTLR version)

Detailed JIRA release notes are here: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342282=Text=12319522=Create_token=A5KQ-2QAV-T4JA-FDED%7C2e97f76f75a466564dba2a7d0bec5ca06bffd66e%7Clout


Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinw...@us.ibm.com

Jenkins build is back to stable : SystemML-DailyTest #1560

[jenkins]

See 

[RESULT] [VOTE] Apache SystemML 1.1.0 (RC2)

[Berthold Reinwald]

Voting has passed with +1 binding votes from 4 PMC members, +1 vote from 1 
committer, and +1 vote from 1 contributor. Thanks everyone!

PMC MEMBERS:
   +1  Matthias Boehm, Glenn Weidner, Berthold Reinwald, Niketan Pansare

Committer
   +1  Krishna Kalyan 

Contributor
   +1  Ted Yu


Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinw...@us.ibm.com



From:   Berthold Reinwald/Almaden/IBM
To: dev@systemml.apache.org
Date:   03/23/2018 05:24 PM
Subject:[VOTE] Apache SystemML 1.1.0 (RC2)


Please vote on releasing the following candidate as Apache SystemML 
version 1.1.0 

The vote is open for at least 72 hours and passes if a majority of at 
least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache SystemML 1.1.0
[ ] -1 Do not release this package because ...

To learn more about Apache SystemML, please see 
http://systemml.apache.org/

The tag to be voted on is v1.1.0-rc2 (
deddaee1fd3f8e87ff3a4403edf06cdb022ba949):
https://github.com/apache/systemml/commit/deddaee1fd3f8e87ff3a4403edf06cdb022ba949

The release artifacts can be found at:
https://dist.apache.org/repos/dist/dev/systemml/1.1.0-rc2/

The maven release artifacts, including signatures, digests, etc. can be 
found at: 
https://repository.apache.org/content/repositories/orgapachesystemml-1029/org/apache/systemml/systemml/1.1.0/

Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinw...@us.ibm.com

GPU hardware support for SystemML bleeding edge.

[Janardhan Pulivarthi]

Greetings,

What are the hardware(hw) configuration that SystemML (an opensource
project) needs to support?

i.e., do we have support for all of these?
 - Kepler
 - Maxwell
 - Pascal
 - Volta

Lets say,
1. we have tuned a kernel only for a specific hw configuration(eg. Maxwell)
with `asm()` code, can we integrate it into our project
2. And can we ignore this hw specific optimized kernel for other
incompatible hardware and simply use the present kernels.

@Niketan - If the answer for the 1 & 2 is yes, then we first tune for
maxwell and then we support others, case by case as they (openai) have done.

Thanks,
Janardhan

Jenkins build became unstable: SystemML-DailyTest #1559

[jenkins]

See 

Contribution to SystemML

[Govinda Malavipathirana]

Hi'
I have submitted 2 proposals for Language .and Parameter Server project. In
this proposal review period I would like to contribute to systemml for
further understanding the systemml internals. I'm looking for some guidance
to select issue that relates/more like to [SYSTEMML-2084] Language and
Compiler Extension and [SYSTEMML-2089] Extended Keras2DML and Caffe2DML
Script Generators. Thanks in advanced.
Best regards,
Govinda