Re: [DISCUSS] Adding tensorboard-like functionality to SystemML

2016-10-28 Thread Jeremy Anderson 
>
> Visualization is a good topic to bring up for the project. I would like to
> add another possible option of using TensorBoard directly. I have not
> looked into the file format used for TensorBoard, but it may be possible to
> simple adopt that format, and simply write our stats to that type of file.
> That would allow us to reuse that project without having to write our own.


Mike, I think this is a great place to start. I'd love to collaborate from
a design perspective, with anyone  that wants to technical side.

...

Jeremy Anderson
Github: https://github.com/objectadjective
Twitter: https://twitter.com/ObjectAdjective
LinkedIN: http://www.linkedin.com/in/objectadjective

On 29 October 2016 at 02:46,  wrote:

> Visualization is a good topic to bring up for the project. I would like to
> add another possible option of using TensorBoard directly. I have not
> looked into the file format used for TensorBoard, but it may be possible to
> simple adopt that format, and simply write our stats to that type of file.
> That would allow us to reuse that project without having to write our own.
>
> --
>
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> Sent from my iPhone.
>
>
> > On Oct 28, 2016, at 8:13 AM, Niketan Pansare  wrote:
> >
> > Hi Matthias,
> >
> > Thanks for your feedback.
> >
> > There is a tradeoff between keeping a feature in-house until it is
> stable, v/s continually getting community feedback as the work is getting
> done via PR and discussions. I am for the latter as it encourages community
> feedback as well as participation.
> >
> > I agree that our goal should be to complete the features you mentioned
> asap and yes, we are working hard towards making the GPU backend, the deep
> learning built-in functions and the algorithm wrappers (ones that are
> already added) to be 'non-experimental' in the 1.0 release :) ... Also,
> like you hinted, it is important to explicitly mark the experimental
> features in the documentation to avoid the 'bad impression'. The Python DSL
> will remain experimental until there is more interest from the community. I
> am fine with deleting the debugger since it is rarely used, if at all.
> >
> > Keeping inline with the Apache guidelines, this discussion is to allow
> community to decide on whether SystemML community should consider adding
> new visualization functionality (since this feature is user facing). If
> there is no interest, we can either postpone or discard this discussion :)
> >
> > Thanks,
> >
> > Niketan.
> >
> >> On Oct 28, 2016, at 1:24 AM, Matthias Boehm 
> wrote:
> >>
> >> Thanks for putting this together Niketan. However, could we please
> >> postpone this discussion after our 1.0 release? Right now, I'm concerned
> >> to see that we're adding many experimental features without really
> >> getting them done. This includes for example, the GPU backend, the new
> >> MLContext API, the Python DSL, the deep learning builtin functions, the
> >> Scala algorithm wrappers, the old Spark debugger interface, and
> >> compressed linear algebra. I think we should finish these features first
> >> before moving on. If we're not careful about that, it would quickly
> >> create a very bad impression for new users.
> >>
> >> Regards,
> >> Matthias
> >>
> >>> On 10/28/2016 1:20 AM, Niketan Pansare wrote:
> >>>
> >>>
> >>> Hi all,
> >>>
> >>> To give every context, I am working on a new deep learning API for
> SystemML
> >>> that is backed by the NN library (
> >>> https://github.com/apache/incubator-systemml/tree/
> master/scripts/staging/SystemML-NN/nn
> >>> ). This API allows the users to express their model using Caffe
> >>> specification and perform fit/predict similar to scikit-learn APIs. I
> have
> >>> created a sample notebook explaining the usage of the API:
> >>> https://github.com/niketanpansare/incubator-systemml/blob/
> 1b655ebeec6cdffd66b282eadc4810ecfd39e4f2/samples/jupyter-
> notebooks/Barista-API-Demo.ipynb
> >>> . This API also allows the user to load and store pre-trained models.
> See
> >>> https://github.com/niketanpansare/model_zoo/tree/
> master/caffe/vision/vgg/ilsvrc12
> >>>
> >>> As part of this API, I added a mini-tensorboard like functionality (see
> >>> step 6 and 7) using matplotlib. If there is enough interest, we can
> extend
> >>> and standardize the visualization functionality across all over
> algorithms.
> >>> Here are some initial discussion points:
> >>> 1. Primary visualization mechanism (Jupyter or a standalone app or
> both =>
> >>> former is useful for cloud offering such as DSX and latter provides the
> >>> design team more creative control)
> >>> 2. What to plot for each algorithm (data scientists and algorithms
> >>> developers will help us here).
> >>> 3. Standardize UI (if we decide to go with Jupyter, we need to extend
> the
> >>> code in _visualize method:
> >>>

Re: [DISCUSS] Adding tensorboard-like functionality to SystemML

2016-10-28 Thread dusenberrymw 
Visualization is a good topic to bring up for the project. I would like to add 
another possible option of using TensorBoard directly. I have not looked into 
the file format used for TensorBoard, but it may be possible to simple adopt 
that format, and simply write our stats to that type of file. That would allow 
us to reuse that project without having to write our own. 

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Oct 28, 2016, at 8:13 AM, Niketan Pansare  wrote:
> 
> Hi Matthias,
> 
> Thanks for your feedback.
> 
> There is a tradeoff between keeping a feature in-house until it is stable, 
> v/s continually getting community feedback as the work is getting done via PR 
> and discussions. I am for the latter as it encourages community feedback as 
> well as participation.
> 
> I agree that our goal should be to complete the features you mentioned asap 
> and yes, we are working hard towards making the GPU backend, the deep 
> learning built-in functions and the algorithm wrappers (ones that are already 
> added) to be 'non-experimental' in the 1.0 release :) ... Also, like you 
> hinted, it is important to explicitly mark the experimental features in the 
> documentation to avoid the 'bad impression'. The Python DSL will remain 
> experimental until there is more interest from the community. I am fine with 
> deleting the debugger since it is rarely used, if at all.
> 
> Keeping inline with the Apache guidelines, this discussion is to allow 
> community to decide on whether SystemML community should consider adding new 
> visualization functionality (since this feature is user facing). If there is 
> no interest, we can either postpone or discard this discussion :)
> 
> Thanks,
> 
> Niketan.
> 
>> On Oct 28, 2016, at 1:24 AM, Matthias Boehm  wrote:
>> 
>> Thanks for putting this together Niketan. However, could we please 
>> postpone this discussion after our 1.0 release? Right now, I'm concerned 
>> to see that we're adding many experimental features without really 
>> getting them done. This includes for example, the GPU backend, the new 
>> MLContext API, the Python DSL, the deep learning builtin functions, the 
>> Scala algorithm wrappers, the old Spark debugger interface, and 
>> compressed linear algebra. I think we should finish these features first 
>> before moving on. If we're not careful about that, it would quickly 
>> create a very bad impression for new users.
>> 
>> Regards,
>> Matthias
>> 
>>> On 10/28/2016 1:20 AM, Niketan Pansare wrote:
>>> 
>>> 
>>> Hi all,
>>> 
>>> To give every context, I am working on a new deep learning API for SystemML
>>> that is backed by the NN library (
>>> https://github.com/apache/incubator-systemml/tree/master/scripts/staging/SystemML-NN/nn
>>> ). This API allows the users to express their model using Caffe
>>> specification and perform fit/predict similar to scikit-learn APIs. I have
>>> created a sample notebook explaining the usage of the API:
>>> https://github.com/niketanpansare/incubator-systemml/blob/1b655ebeec6cdffd66b282eadc4810ecfd39e4f2/samples/jupyter-notebooks/Barista-API-Demo.ipynb
>>> . This API also allows the user to load and store pre-trained models. See
>>> https://github.com/niketanpansare/model_zoo/tree/master/caffe/vision/vgg/ilsvrc12
>>> 
>>> As part of this API, I added a mini-tensorboard like functionality (see
>>> step 6 and 7) using matplotlib. If there is enough interest, we can extend
>>> and standardize the visualization functionality across all over algorithms.
>>> Here are some initial discussion points:
>>> 1. Primary visualization mechanism (Jupyter or a standalone app or both =>
>>> former is useful for cloud offering such as DSX and latter provides the
>>> design team more creative control)
>>> 2. What to plot for each algorithm (data scientists and algorithms
>>> developers will help us here).
>>> 3. Standardize UI (if we decide to go with Jupyter, we need to extend the
>>> code in _visualize method:
>>> https://github.com/niketanpansare/incubator-systemml/blob/1b655ebeec6cdffd66b282eadc4810ecfd39e4f2/src/main/python/systemml/mllearn/estimators.py#L621
>>> )
>>> 4. Primary APIs to target (python, scala, command-line or all)
>>> 
>>> Thanks,
>>> 
>>> Niketan Pansare
>>> IBM Almaden Research Center
>>> E-mail: npansar At us.ibm.com
>>> http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
>>> 
>> 
>

Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)

2016-10-28 Thread dusenberrymw 
+1

I've been running large scale use-case tests and things ran well on rc3 and on 
this rc4. 

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Oct 28, 2016, at 4:08 PM, Luciano Resende  wrote:
> 
> Off course, my +1.
> 
> On Mon, Oct 24, 2016 at 4:11 PM, Luciano Resende 
> wrote:
> 
>> Please vote on releasing the following candidate as Apache SystemML
>> version 0.11.0-incubating !
>> 
>> The vote is open for at least 72 hours and passes if a majority of at
>> least 3 +1 PMC votes are cast.
>> 
>> [ ] +1 Release this package as Apache SystemML 0.11.0-incubating
>> [ ] -1 Do not release this package because ...
>> 
>> To learn more about Apache SystemML, please see
>> http://systemml.apache.org/
>> 
>> The tag to be voted on is v0.11.0-incubating-rc4 (
>> 6937683b01a13458990e698b0cf04f4f6ccecde3)
>> 
>> https://github.com/apache/incubator-systemml/tree/
>> 6937683b01a13458990e698b0cf04f4f6ccecde3
>> 
>> The release artifacts can be found at :
>> 
>> https://dist.apache.org/repos/dist/dev/incubator/systemml/0.
>> 11.0-incubating-rc4/
>> 
>> The maven release artifacts, including signatures, digests, etc. can be
>> found at:
>> 
>> https://repository.apache.org/content/repositories/orgapachesystemml-1010/
>> 
>> 
>> =
>> == Apache Incubator release policy ==
>> =
>> Please find below the guide to release management during incubation:
>> http://incubator.apache.org/guides/releasemanagement.html
>> 
>> ===
>> == How can I help test this release? ==
>> ===
>> If you are a SystemML user, you can help us test this release by taking an
>> existing Algorithm or workload and running on this release candidate, then
>> reporting any regressions.
>> 
>> 
>> == What justifies a -1 vote for this release? ==
>> 
>> -1 votes should only occur for significant stop-ship bugs or legal
>> related issues (e.g. wrong license, missing header files, etc). Minor bugs
>> or regressions should not block this release.
>> 
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>> 
> 
> 
> 
> -- 
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/

Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)

2016-10-28 Thread Luciano Resende 
Off course, my +1.

On Mon, Oct 24, 2016 at 4:11 PM, Luciano Resende 
wrote:

> Please vote on releasing the following candidate as Apache SystemML
> version 0.11.0-incubating !
>
> The vote is open for at least 72 hours and passes if a majority of at
> least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache SystemML 0.11.0-incubating
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache SystemML, please see
> http://systemml.apache.org/
>
> The tag to be voted on is v0.11.0-incubating-rc4 (
> 6937683b01a13458990e698b0cf04f4f6ccecde3)
>
> https://github.com/apache/incubator-systemml/tree/
> 6937683b01a13458990e698b0cf04f4f6ccecde3
>
> The release artifacts can be found at :
>
> https://dist.apache.org/repos/dist/dev/incubator/systemml/0.
> 11.0-incubating-rc4/
>
> The maven release artifacts, including signatures, digests, etc. can be
> found at:
>
> https://repository.apache.org/content/repositories/orgapachesystemml-1010/
>
>
> =
> == Apache Incubator release policy ==
> =
> Please find below the guide to release management during incubation:
> http://incubator.apache.org/guides/releasemanagement.html
>
> ===
> == How can I help test this release? ==
> ===
> If you are a SystemML user, you can help us test this release by taking an
> existing Algorithm or workload and running on this release candidate, then
> reporting any regressions.
>
> 
> == What justifies a -1 vote for this release? ==
> 
> -1 votes should only occur for significant stop-ship bugs or legal
> related issues (e.g. wrong license, missing header files, etc). Minor bugs
> or regressions should not block this release.
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)

2016-10-28 Thread Nakul Jindal 
+1

On Fri, Oct 28, 2016 at 12:28 PM, Glenn Weidner  wrote:

> Validated RC4 binaries on Windows environment:
> http://apache.github.io/incubator-systemml/release-
> process.html#all-binaries-execute
>
> --Glenn
>
> [image: Inactive hide details for Niketan Pansare---10/28/2016 11:49:14
> AM---All Python tests passed using Python 2.7 and Spark 1.6.1.]Niketan
> Pansare---10/28/2016 11:49:14 AM---All Python tests passed using Python 2.7
> and Spark 1.6.1. +1
>
> From: Niketan Pansare/Almaden/IBM@IBMUS
> To: dev@systemml.incubator.apache.org
> Date: 10/28/2016 11:49 AM
> Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)
> --
>
>
>
> All Python tests passed using Python 2.7 and Spark 1.6.1.
>
> +1
>
> Thanks,
>
> Niketan Pansare
> IBM Almaden Research Center
> E-mail: npansar At us.ibm.com
> *http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar*
> 
>
> Acs S ---10/27/2016 11:53:11 PM---+1 -Arvind From: Glenn Weidner <
> gweid...@us.ibm.com>
>
> From: Acs S 
> To: "dev@systemml.incubator.apache.org"  >
> Date: 10/27/2016 11:53 PM
> Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)
> --
>
>
>
> +1
> -Arvind  From: Glenn Weidner 
> To: dev@systemml.incubator.apache.org
> Sent: Thursday, October 27, 2016 11:07 PM
> Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)
>
> Completed performance test suite for:
>
> XS scenarios (80MB)
> S scenarios (800MB)
> M scenarios (8GB)
> L scenarios (80GB)
> XL subset (800GB)
>
> +1
>
> --Glenn
>
> Luciano Resende ---10/24/2016 04:11:22 PM---Please vote on releasing the
> following candidate as Apache SystemML version 0.11.0-incubating !
>
> From: Luciano Resende 
> To: dev@systemml.incubator.apache.org
> Date: 10/24/2016 04:11 PM
> Subject: [VOTE] Apache SystemML 0.11.0-incubating (RC4)
>
>
>
> Please vote on releasing the following candidate as Apache SystemML version
> 0.11.0-incubating !
>
> The vote is open for at least 72 hours and passes if a majority of at least
> 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache SystemML 0.11.0-incubating
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache SystemML, please see
> *http://systemml.apache.org/* 
>
> The tag to be voted on is v0.11.0-incubating-rc4
> (6937683b01a13458990e698b0cf04f4f6ccecde3)
>
> *https://github.com/apache/incu* 
> bator-systemml/tree/6937683b01a13458990e698b0cf04f4f6ccecde3
>
> The release artifacts can be found at :
>
> *https://dist.apache.org/repos/dist/dev/incubator/systemml/0*
> .
> 11.0-incubating-rc4/
>
> The maven release artifacts, including signatures, digests, etc. can be
> found at:
>
>
> *https://repository.apache.org/content/repositories/orgapachesystemml-1010/*
> 
>
>
> =
> == Apache Incubator release policy ==
> =
> Please find below the guide to release management during incubation:
> *http://incubator.apache.org/guides/releasemanagement.html*
> 
>
> ===
> == How can I help test this release? ==
> ===
> If you are a SystemML user, you can help us test this release by taking an
> existing Algorithm or workload and running on this release candidate, then
> reporting any regressions.
>
> 
> == What justifies a -1 vote for this release? ==
> 
> -1 votes should only occur for significant stop-ship bugs or legal related
> issues (e.g. wrong license, missing header files, etc). Minor bugs or
> regressions should not block this release.
>
> --
> Luciano Resende
> *http://twitter.com/lresende1975* 
> *http://lresende.blogspot.com/* 
>
>
>
>
>
>
>
>
>
>
>

Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)

2016-10-28 Thread Glenn Weidner 

Validated RC4 binaries on Windows environment:
http://apache.github.io/incubator-systemml/release-process.html#all-binaries-execute


--Glenn



From:   Niketan Pansare/Almaden/IBM@IBMUS
To: dev@systemml.incubator.apache.org
Date:   10/28/2016 11:49 AM
Subject:Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)



All Python tests passed using Python 2.7 and Spark 1.6.1.

+1

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar

Acs S ---10/27/2016 11:53:11 PM---+1 -Arvind From: Glenn Weidner


From: Acs S 
To: "dev@systemml.incubator.apache.org" 
Date: 10/27/2016 11:53 PM
Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)



+1
-Arvind  From: Glenn Weidner 
To: dev@systemml.incubator.apache.org
Sent: Thursday, October 27, 2016 11:07 PM
Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)

Completed performance test suite for:

XS scenarios (80MB)
S scenarios (800MB)
M scenarios (8GB)
L scenarios (80GB)
XL subset (800GB)

+1

--Glenn

Luciano Resende ---10/24/2016 04:11:22 PM---Please vote on releasing the
following candidate as Apache SystemML version 0.11.0-incubating !

From: Luciano Resende 
To: dev@systemml.incubator.apache.org
Date: 10/24/2016 04:11 PM
Subject: [VOTE] Apache SystemML 0.11.0-incubating (RC4)



Please vote on releasing the following candidate as Apache SystemML version
0.11.0-incubating !

The vote is open for at least 72 hours and passes if a majority of at least
3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache SystemML 0.11.0-incubating
[ ] -1 Do not release this package because ...

To learn more about Apache SystemML, please see http://systemml.apache.org/

The tag to be voted on is v0.11.0-incubating-rc4
(6937683b01a13458990e698b0cf04f4f6ccecde3)

https://github.com/apache/incu
bator-systemml/tree/6937683b01a13458990e698b0cf04f4f6ccecde3

The release artifacts can be found at :

https://dist.apache.org/repos/dist/dev/incubator/systemml/0.
11.0-incubating-rc4/

The maven release artifacts, including signatures, digests, etc. can be
found at:

https://repository.apache.org/content/repositories/orgapachesystemml-1010/


=
== Apache Incubator release policy ==
=
Please find below the guide to release management during incubation:
http://incubator.apache.org/guides/releasemanagement.html

===
== How can I help test this release? ==
===
If you are a SystemML user, you can help us test this release by taking an
existing Algorithm or workload and running on this release candidate, then
reporting any regressions.


== What justifies a -1 vote for this release? ==

-1 votes should only occur for significant stop-ship bugs or legal related
issues (e.g. wrong license, missing header files, etc). Minor bugs or
regressions should not block this release.

--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)

2016-10-28 Thread Niketan Pansare 

All Python tests passed using Python 2.7 and Spark 1.6.1.

+1

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   Acs S 
To: "dev@systemml.incubator.apache.org"

Date:   10/27/2016 11:53 PM
Subject:Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)



+1
-Arvind  From: Glenn Weidner 
 To: dev@systemml.incubator.apache.org
 Sent: Thursday, October 27, 2016 11:07 PM
 Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)

Completed performance test suite for:

XS scenarios (80MB)
S scenarios (800MB)
M scenarios (8GB)
L scenarios (80GB)
XL subset (800GB)

+1

--Glenn

Luciano Resende ---10/24/2016 04:11:22 PM---Please vote on releasing the
following candidate as Apache SystemML version 0.11.0-incubating !

From: Luciano Resende 
To: dev@systemml.incubator.apache.org
Date: 10/24/2016 04:11 PM
Subject: [VOTE] Apache SystemML 0.11.0-incubating (RC4)



Please vote on releasing the following candidate as Apache SystemML version
0.11.0-incubating !

The vote is open for at least 72 hours and passes if a majority of at least
3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache SystemML 0.11.0-incubating
[ ] -1 Do not release this package because ...

To learn more about Apache SystemML, please see http://systemml.apache.org/

The tag to be voted on is v0.11.0-incubating-rc4
(6937683b01a13458990e698b0cf04f4f6ccecde3)

https://github.com/apache/incu
bator-systemml/tree/6937683b01a13458990e698b0cf04f4f6ccecde3

The release artifacts can be found at :

https://dist.apache.org/repos/dist/dev/incubator/systemml/0.
11.0-incubating-rc4/

The maven release artifacts, including signatures, digests, etc. can be
found at:

https://repository.apache.org/content/repositories/orgapachesystemml-1010/


=
== Apache Incubator release policy ==
=
Please find below the guide to release management during incubation:
http://incubator.apache.org/guides/releasemanagement.html

===
== How can I help test this release? ==
===
If you are a SystemML user, you can help us test this release by taking an
existing Algorithm or workload and running on this release candidate, then
reporting any regressions.


== What justifies a -1 vote for this release? ==

-1 votes should only occur for significant stop-ship bugs or legal related
issues (e.g. wrong license, missing header files, etc). Minor bugs or
regressions should not block this release.

--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [DISCUSS] Adding tensorboard-like functionality to SystemML

2016-10-28 Thread Niketan Pansare 
Hi Matthias,

Thanks for your feedback.

There is a tradeoff between keeping a feature in-house until it is stable, v/s 
continually getting community feedback as the work is getting done via PR and 
discussions. I am for the latter as it encourages community feedback as well as 
participation.

I agree that our goal should be to complete the features you mentioned asap and 
yes, we are working hard towards making the GPU backend, the deep learning 
built-in functions and the algorithm wrappers (ones that are already added) to 
be 'non-experimental' in the 1.0 release :) ... Also, like you hinted, it is 
important to explicitly mark the experimental features in the documentation to 
avoid the 'bad impression'. The Python DSL will remain experimental until there 
is more interest from the community. I am fine with deleting the debugger since 
it is rarely used, if at all.

Keeping inline with the Apache guidelines, this discussion is to allow 
community to decide on whether SystemML community should consider adding new 
visualization functionality (since this feature is user facing). If there is no 
interest, we can either postpone or discard this discussion :)

Thanks,

Niketan.

> On Oct 28, 2016, at 1:24 AM, Matthias Boehm  wrote:
> 
> Thanks for putting this together Niketan. However, could we please 
> postpone this discussion after our 1.0 release? Right now, I'm concerned 
> to see that we're adding many experimental features without really 
> getting them done. This includes for example, the GPU backend, the new 
> MLContext API, the Python DSL, the deep learning builtin functions, the 
> Scala algorithm wrappers, the old Spark debugger interface, and 
> compressed linear algebra. I think we should finish these features first 
> before moving on. If we're not careful about that, it would quickly 
> create a very bad impression for new users.
> 
> Regards,
> Matthias
> 
>> On 10/28/2016 1:20 AM, Niketan Pansare wrote:
>> 
>> 
>> Hi all,
>> 
>> To give every context, I am working on a new deep learning API for SystemML
>> that is backed by the NN library (
>> https://github.com/apache/incubator-systemml/tree/master/scripts/staging/SystemML-NN/nn
>> ). This API allows the users to express their model using Caffe
>> specification and perform fit/predict similar to scikit-learn APIs. I have
>> created a sample notebook explaining the usage of the API:
>> https://github.com/niketanpansare/incubator-systemml/blob/1b655ebeec6cdffd66b282eadc4810ecfd39e4f2/samples/jupyter-notebooks/Barista-API-Demo.ipynb
>> . This API also allows the user to load and store pre-trained models. See
>> https://github.com/niketanpansare/model_zoo/tree/master/caffe/vision/vgg/ilsvrc12
>> 
>> As part of this API, I added a mini-tensorboard like functionality (see
>> step 6 and 7) using matplotlib. If there is enough interest, we can extend
>> and standardize the visualization functionality across all over algorithms.
>> Here are some initial discussion points:
>> 1. Primary visualization mechanism (Jupyter or a standalone app or both =>
>> former is useful for cloud offering such as DSX and latter provides the
>> design team more creative control)
>> 2. What to plot for each algorithm (data scientists and algorithms
>> developers will help us here).
>> 3. Standardize UI (if we decide to go with Jupyter, we need to extend the
>> code in _visualize method:
>> https://github.com/niketanpansare/incubator-systemml/blob/1b655ebeec6cdffd66b282eadc4810ecfd39e4f2/src/main/python/systemml/mllearn/estimators.py#L621
>> )
>> 4. Primary APIs to target (python, scala, command-line or all)
>> 
>> Thanks,
>> 
>> Niketan Pansare
>> IBM Almaden Research Center
>> E-mail: npansar At us.ibm.com
>> http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
>> 
>

Re: [DISCUSS] Adding tensorboard-like functionality to SystemML

2016-10-28 Thread Matthias Boehm 
Thanks for putting this together Niketan. However, could we please 
postpone this discussion after our 1.0 release? Right now, I'm concerned 
to see that we're adding many experimental features without really 
getting them done. This includes for example, the GPU backend, the new 
MLContext API, the Python DSL, the deep learning builtin functions, the 
Scala algorithm wrappers, the old Spark debugger interface, and 
compressed linear algebra. I think we should finish these features first 
before moving on. If we're not careful about that, it would quickly 
create a very bad impression for new users.


Regards,
Matthias

On 10/28/2016 1:20 AM, Niketan Pansare wrote:



Hi all,

To give every context, I am working on a new deep learning API for SystemML
that is backed by the NN library (
https://github.com/apache/incubator-systemml/tree/master/scripts/staging/SystemML-NN/nn
). This API allows the users to express their model using Caffe
specification and perform fit/predict similar to scikit-learn APIs. I have
created a sample notebook explaining the usage of the API:
https://github.com/niketanpansare/incubator-systemml/blob/1b655ebeec6cdffd66b282eadc4810ecfd39e4f2/samples/jupyter-notebooks/Barista-API-Demo.ipynb
. This API also allows the user to load and store pre-trained models. See
https://github.com/niketanpansare/model_zoo/tree/master/caffe/vision/vgg/ilsvrc12

As part of this API, I added a mini-tensorboard like functionality (see
step 6 and 7) using matplotlib. If there is enough interest, we can extend
and standardize the visualization functionality across all over algorithms.
Here are some initial discussion points:
1. Primary visualization mechanism (Jupyter or a standalone app or both =>
former is useful for cloud offering such as DSX and latter provides the
design team more creative control)
2. What to plot for each algorithm (data scientists and algorithms
developers will help us here).
3. Standardize UI (if we decide to go with Jupyter, we need to extend the
code in _visualize method:
https://github.com/niketanpansare/incubator-systemml/blob/1b655ebeec6cdffd66b282eadc4810ecfd39e4f2/src/main/python/systemml/mllearn/estimators.py#L621
)
4. Primary APIs to target (python, scala, command-line or all)

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar

Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)

2016-10-28 Thread Acs S 
+1
-Arvind  From: Glenn Weidner 
 To: dev@systemml.incubator.apache.org 
 Sent: Thursday, October 27, 2016 11:07 PM
 Subject: Re: [VOTE] Apache SystemML 0.11.0-incubating (RC4)
   
Completed performance test suite for:

XS scenarios (80MB)
S scenarios (800MB)
M scenarios (8GB)
L scenarios (80GB)
XL subset (800GB)

+1

--Glenn

Luciano Resende ---10/24/2016 04:11:22 PM---Please vote on releasing the 
following candidate as Apache SystemML version 0.11.0-incubating !

From: Luciano Resende 
To: dev@systemml.incubator.apache.org
Date: 10/24/2016 04:11 PM
Subject: [VOTE] Apache SystemML 0.11.0-incubating (RC4)



Please vote on releasing the following candidate as Apache SystemML version
0.11.0-incubating !

The vote is open for at least 72 hours and passes if a majority of at least
3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache SystemML 0.11.0-incubating
[ ] -1 Do not release this package because ...

To learn more about Apache SystemML, please see http://systemml.apache.org/

The tag to be voted on is v0.11.0-incubating-rc4
(6937683b01a13458990e698b0cf04f4f6ccecde3)

https://github.com/apache/incu
bator-systemml/tree/6937683b01a13458990e698b0cf04f4f6ccecde3

The release artifacts can be found at :

https://dist.apache.org/repos/dist/dev/incubator/systemml/0.
11.0-incubating-rc4/

The maven release artifacts, including signatures, digests, etc. can be
found at:

https://repository.apache.org/content/repositories/orgapachesystemml-1010/


=
== Apache Incubator release policy ==
=
Please find below the guide to release management during incubation:
http://incubator.apache.org/guides/releasemanagement.html

===
== How can I help test this release? ==
===
If you are a SystemML user, you can help us test this release by taking an
existing Algorithm or workload and running on this release candidate, then
reporting any regressions.


== What justifies a -1 vote for this release? ==

-1 votes should only occur for significant stop-ship bugs or legal related
issues (e.g. wrong license, missing header files, etc). Minor bugs or
regressions should not block this release.

-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/