Re: [VOTE] Apache SystemML 1.0.0 (RC2)

[Matthias Boehm]

+1

I ran the perftest suite with the artifact on Spark 2.2 up to 80GB without
any failures or performance issues. On earlier versions, I also ran the
perftest suite with Spark 2.1 and 2.2, w/ and w/o codegen, and w/ auto
compression up to 800GB without remaining issues.

As a minor nitpick (to be resolved in any additional RCs or the next
release), the root directory of the artifact includes a SystemML-config.xml
file that only has a small subset of configuration parameters. We might
want to either delete this or pull in a copy of
SystemML-config.xml.template from the conf directory.

Regards,
Matthias

On Sat, Dec 9, 2017 at 1:53 AM, Luciano Resende 
wrote:

> On Fri, Dec 8, 2017 at 12:15 PM, Berthold Reinwald 
> wrote:
>
> > Please vote on releasing the following candidate as Apache SystemML
> > version 1.0.0
> >
> > The vote is open for at least 72 hours and passes if a majority of at
> > least 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache SystemML 1.0.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache SystemML, please see
> > http://systemml.apache.org/
> >
> > The tag to be voted on is v1.0.0-rc2 (
> > 7b44380e49425ac20912c2d42de8603a06fa55a1):
> > https://github.com/apache/systemml/commit/7b44380e49425ac20912c2d42de860
> > 3a06fa55a1
> >
> >
> Looks like the tag hash should be
> 6519f3383f8fd43af93ecba8742ae4b2b28d4b35
>
> And the link
> https://github.com/apache/systemml/tree/6519f3383f8fd43af93ecba8742ae4
> b2b28d4b35
>
>
> >
> > The release artifacts can be found at:
> > https://dist.apache.org/repos/dist/dev/systemml/1.0.0-rc2/
> >
> >
> > The maven release artifacts, including signatures, digests, etc. can be
> > found at:
> > https://repository.apache.org/content/repositories/
> > orgapachesystemml-1026/org/apache/systemml/systemml/1.0.0/
> >
> >
> >
> > ===
> > == Apache Release policy ==
> > ===
> > http://www.apache.org/legal/release-policy.html
> >
> > ===
> > == How can I help test this release? ==
> > ===
> > If you are a SystemML user, you can help us test this release by taking
> an
> > existing Algorithm or workload and running on this release candidate,
> then
> > reporting any regressions.
> >
> > 
> > == What justifies a -1 vote for this release? ==
> > 
> > -1 votes should only occur for significant stop-ship bugs or legal
> related
> > issues (e.g. wrong license, missing header files, etc). Minor bugs or
> > regressions should not block this release.
> >
> >
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Jenkins build is back to stable : SystemML-DailyTest #1408

[jenkins]

See 

Re: [DISCUSS] Roadmap SystemML 1.1 and beyond

[Janardhan Pulivarthi]

Hi all, my 0.02$ I am working on one by one.

Please add to the above list..

0. Algorithms
* Factorization machines, with regression & classification capabities with
the help of nn layers.[ 1437]
* A test suite for the nn optimization, with well known optimization test
functions. [1974]

1. Deep Learning
* I am working on model selection + hyperparameter optimization, a basic
implementation
will be possible by January. [SYSTEMML-1973] - some components of it are in
testing phase, now.
* I think distributed DL is a great idea, & it may be necessary now.

2. GPU backends
* Support for sparse operations - [SYSTEMML-2041] Implementation of block
sparse kernel enables us to model LSTM
with 10,000 hidden units, instead current state-of-the-art 1000 hidden units

6. Misc. compiler
* support for single-output UDFs in expressions.
* SPOOF compiler improvement
* Rewrites

8. Builtin functions
* Well known distribution functions - weibull, gamma etc.
* Generalization of operations, such as xor, and, other operations.

9. Documentation improvement.

Thanks,
Janardhan

On Sat, Dec 9, 2017 at 8:11 AM, Matthias Boehm  wrote:

> Hi all,
>
> with our SystemML 1.0 release around the corner, I think we should start
> the discussion on the roadmap for SystemML 1.1 and beyond. Below is an
> initial list as a starting point, but please help to add relevant items,
> especially for algorithms and APIs, which are barely covered so far.
>
> 1) Deep Learning
>  * Full compiler integration GPU backend
>  * Extended sparse operations on CPU/GPU
>  * Extended single-precision support CPU
>  * Distributed DL operations?
>
> 2) GPU Backend
>  * Full support for sparse operations
>  * Automatic decisions on CPU vs GPU operations
>  * Graduate GPU backends (enable by default)
>
> 3) Code generation
>  * Graduate code generation (enable by default)
>  * Support for deep learning operations
>  * Code generation for the heterogeneous HW, incl GPUs
>
> 4) Compressed Linear Algebra
>  * Support for matrix-matrix multiplications
>  * Support for deep learning operations
>  * Improvements for ultra-sparse datasets
>
> 5) Misc Runtime
>  * Large dense matrix blocks > 16GB
>  * NUMA-awareness (thread pools, matrix partitioning)
>  * Unified memory management (ops, bufferpool, RDDs/broadcasts)
>  * Support feather format for matrices and frames
>  * Parfor support for broadcasts
>  * Extended support for multi-threaded operations
>  * Boolean matrices
>
> 6) Misc Compiler
>  * Support single-output UDFs in expressions
>  * Consolidate replicated compilation chain (e.g., diff APIs)
>  * Holistic sum-product optimization and operator fusion
>  * Extended sparsity estimators
>  * Rewrites and compiler improvements for mini-batching
>  * Parfor optimizer support for shared reads
>
> 7) APIs
>  * Python Binding for JMLC API
>  * Consistency Python/Java APIs
>
>
> Regards,
> Matthias
>

Jenkins build became unstable: SystemML-DailyTest #1407

[jenkins]

See 

Re: [VOTE] Apache SystemML 1.0.0 (RC2)

[Luciano Resende]

On Fri, Dec 8, 2017 at 12:15 PM, Berthold Reinwald 
wrote:

> Please vote on releasing the following candidate as Apache SystemML
> version 1.0.0
>
> The vote is open for at least 72 hours and passes if a majority of at
> least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache SystemML 1.0.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache SystemML, please see
> http://systemml.apache.org/
>
> The tag to be voted on is v1.0.0-rc2 (
> 7b44380e49425ac20912c2d42de8603a06fa55a1):
> https://github.com/apache/systemml/commit/7b44380e49425ac20912c2d42de860
> 3a06fa55a1
>
>
Looks like the tag hash should be
6519f3383f8fd43af93ecba8742ae4b2b28d4b35

And the link
https://github.com/apache/systemml/tree/6519f3383f8fd43af93ecba8742ae4b2b28d4b35


>
> The release artifacts can be found at:
> https://dist.apache.org/repos/dist/dev/systemml/1.0.0-rc2/
>
>
> The maven release artifacts, including signatures, digests, etc. can be
> found at:
> https://repository.apache.org/content/repositories/
> orgapachesystemml-1026/org/apache/systemml/systemml/1.0.0/
>
>
>
> ===
> == Apache Release policy ==
> ===
> http://www.apache.org/legal/release-policy.html
>
> ===
> == How can I help test this release? ==
> ===
> If you are a SystemML user, you can help us test this release by taking an
> existing Algorithm or workload and running on this release candidate, then
> reporting any regressions.
>
> 
> == What justifies a -1 vote for this release? ==
> 
> -1 votes should only occur for significant stop-ship bugs or legal related
> issues (e.g. wrong license, missing header files, etc). Minor bugs or
> regressions should not block this release.
>
>


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/