Re: Documentation search utility.

[Niketan Pansare]

Alternatively, we can move the search box from navigation bar to the
overview page. This will still keep our website mobile enabled. I
understand this is less intuitive but we want our website to be
functioning.

Thanks,

Niketan

> On Mar 13, 2019, at 7:37 AM, Janardhan 
wrote:
>
> Hi Niketan,
>
> thanks for testing out. I have developed it on
> Windows + chrome/mozilla dev edtion  with full screen=works fine
> Windows + chrome/mozilla dev edtion  on minimize = dropdown is not
> functioning [This is a known issue
>
<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SYSTEMML-2D1176=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=smYICQjQTUvunbpOVDWJXz030REXBpTzTQHs3ZZ09d4=eo71291yZws4Kxp66vMbVfN_r22QgRlvqYrQvQgGzQE=>]

>
> So, I would like to fix the navigation bar to accomodate based on some
> standard practice.
>
> Thanks,
> Janardhan
>
>> On Wed, Mar 13, 2019 at 7:26 PM Niketan Pansare 
wrote:
>>
>>
>> Hi Janardhan,
>>
>> Thanks for adding a search bar. It is a very useful feature. I tested it
on
>> Mac and iPhone.
>>
>> Mac + Firefox/Safari/Chrome = Worked as expected.
>> iPhone + Safari = Broke the previously working menu/navigation bar.
>>
>> I am okay if we don’t support search bar on iPhone but we still want the
>> website to work on it.
>>
>> Thanks,
>>
>> Niketan
>>
>>> On Mar 12, 2019, at 7:42 PM, Janardhan 
>> wrote:
>>>
>>> Hi,
>>>
>>> Please find the live link for the search bar enabled docs for 1.2.0
>>>
>>
>>
https://urldefense.proofpoint.com/v2/url?u=https-3A__j143-2Dbot.github.io_systemml_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=MV9iP1yUXyF_xpiBqNJ_-TW6mqfYg96RZjEBqcIgmlw=7tRZ30RMg_vgNp5oOx-XZLZYAD3y2yenJYJmtmAs51U=

>>
>>>
>>> 1. For now the crawler searches in 1.2.0 only,
>>> the functionality will be extended in a few days.
>>> 2. on minimize the dropdown is not working properly this will be
>> addressed
>>> in a different PR.
>>>
>>> Thank you,
>>> Janardhan
>>>
>>>> On Wed, Mar 6, 2019 at 8:59 AM Janardhan  wrote:
>>>>
>>>> Hi all,
>>>>
>>>> Since, the documentation have grown a lot and google search is not
doing
>> a
>>>> good job finding what is needed for users.
>>>>
>>>> Adding algolia search to our website?. Please provide your advice.
>>>>
>>>> 1. Here is the snapshot:
>>>>
>>>>
>>
>>
https://urldefense.proofpoint.com/v2/url?u=https-3A__user-2Dimages.githubusercontent.com_46682169_53852562-2D789c9100-2D3fe8-2D11e9-2D8d37-2D1100c5ab0d27.png=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=MV9iP1yUXyF_xpiBqNJ_-TW6mqfYg96RZjEBqcIgmlw=WJYTDJbDM6eoYjYDDJp_LizSkyNTHuSoEplR0RYoXoU=

>>
>>>>
>>>> 2. PR #855 :)
>>>>
>>>>
>>>> Thanks a lot,
>>>> Janardhan
>>>>
>>

Re: Documentation search utility.

[Niketan Pansare]

Hi Janardhan,

Thanks for adding a search bar. It is a very useful feature. I tested it on
Mac and iPhone.

Mac + Firefox/Safari/Chrome = Worked as expected.
iPhone + Safari = Broke the previously working menu/navigation bar.

I am okay if we don’t support search bar on iPhone but we still want the
website to work on it.

Thanks,

Niketan

> On Mar 12, 2019, at 7:42 PM, Janardhan 
wrote:
>
> Hi,
>
> Please find the live link for the search bar enabled docs for 1.2.0
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__j143-2Dbot.github.io_systemml_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=MV9iP1yUXyF_xpiBqNJ_-TW6mqfYg96RZjEBqcIgmlw=7tRZ30RMg_vgNp5oOx-XZLZYAD3y2yenJYJmtmAs51U=

>
> 1. For now the crawler searches in 1.2.0 only,
> the functionality will be extended in a few days.
> 2. on minimize the dropdown is not working properly this will be
addressed
> in a different PR.
>
> Thank you,
> Janardhan
>
>> On Wed, Mar 6, 2019 at 8:59 AM Janardhan  wrote:
>>
>> Hi all,
>>
>> Since, the documentation have grown a lot and google search is not doing
a
>> good job finding what is needed for users.
>>
>> Adding algolia search to our website?. Please provide your advice.
>>
>> 1. Here is the snapshot:
>>
>>
https://urldefense.proofpoint.com/v2/url?u=https-3A__user-2Dimages.githubusercontent.com_46682169_53852562-2D789c9100-2D3fe8-2D11e9-2D8d37-2D1100c5ab0d27.png=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=MV9iP1yUXyF_xpiBqNJ_-TW6mqfYg96RZjEBqcIgmlw=WJYTDJbDM6eoYjYDDJp_LizSkyNTHuSoEplR0RYoXoU=

>>
>> 2. PR #855 :)
>>
>>
>> Thanks a lot,
>> Janardhan
>>

Fwd: [NOTICE] Mandatory migration of git repositories to gitbox.apache.org

[Niketan Pansare]

Hi all,
 
As per the notice from Apache Infrastructure folks, we must migrate our 
git repositories from git-wip-us.apache.org to gitbox.apache.org. To 
coordinate the move with the Apache Infrastructure team, I have created 
https://issues.apache.org/jira/browse/INFRA-17553  and requested them to 
perform the migration on Monday, Jan 7th 2019. If you have concerns 
regarding the migration, please feel free to raise them on the above JIRA.
 
Thanks,
 
Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



 Forwarded Message 
Subject: [NOTICE] Mandatory migration of git repositories to 
gitbox.apache.org
Date: Thu, 03 Jan 2019 13:19:24 -
From: Apache Infrastructure Team 
Reply-To: dev@systemml.apache.org, us...@infra.apache.org
To: dev@systemml.apache.org

Hello, systemml folks.
As stated earlier in 2018, all git repositories must be migrated from
the git-wip-us.apache.org URL to gitbox.apache.org, as the old service
is being decommissioned. Your project is receiving this email because
you still have repositories on git-wip-us that needs to be migrated.

The following repositories on git-wip-us belong to your project:
  - systemml.git
  - systemml-website.git


We are now entering the mandated (coordinated) move stage of the roadmap,
and you are asked to please coordinate migration with the Apache
Infrastructure Team before February 7th. All repositories not migrated
on February 7th will be mass migrated without warning, and we'd appreciate
it if we could work together to avoid a big mess that day :-).

Moving to gitbox means you will get full write access on GitHub as well,
and be able to close/merge pull requests and much more.

To have your repositories moved, please follow these steps:

- Ensure consensus on the move (a link to a lists.apache.org thread will
   suffice for us as evidence).
- Create a JIRA ticket at 

Your migration should only take a few minutes. If you wish to migrate
at a specific time of day or date, please do let us know in the ticket.

As always, we appreciate your understanding and patience as we move
things around and work to provide better services and features for
the Apache Family.

Should you wish to contact us with feedback or questions, please do so
at: us...@infra.apache.org.


With regards,
Apache Infrastructure

Re: New committer: Guobao Li

[Niketan Pansare]

Welcome, Guobao.

> On Sep 4, 2018, at 4:09 PM, dusenberr...@gmail.com wrote:
> 
> Welcome, Guobao

Re: [VOTE] Apache SystemML 1.2.0 (RC1)

[Niketan Pansare]

+1. Ran GPU and integration tests.
 
Thanks,
 
Niketan.

-Deron Eriksson  wrote: -

>To: dev@systemml.apache.org
>From: Deron Eriksson 
>Date: 08/21/2018 02:18PM
>Subject: Re: [VOTE] Apache SystemML 1.2.0 (RC1)
>
>+1
>
>Ran basic tests to verify bin artifacts run and that artifacts can be
>built
>and run using source artifact.
>
>License files in artifacts appeared to be correct to me.
>
>Note that I believe the lite jar should be removed from
>INVALID URI REMOVED
>repos_dist_dev_systemml_1.2.0-2Drc1_=DwIBaQ=jf_iaSHvJObTbx-siA1ZO
>g=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=Uqz4fylsOCFVH3PSKrk
>wgncGqidpgrr7TfqU8ejtt_8=1Z9SWlyHbLIlpCzNvqbU5rJg9ejYxQfQ75d5kdYBDg
>k=
>
>Deron
>
>
>
>On Tue, Aug 21, 2018 at 11:51 AM Anthony Thomas
>
>wrote:
>
>> +1
>>
>> I ran the Python test suite on Red Hat Linux under Spark 2.2.0
>(Python
>> 2.7.5) and encountered no errors.
>>
>> Regards,
>> Anthony
>>
>> On Tue, Aug 21, 2018 at 7:49 AM Guobao Li 
>wrote:
>>
>> > +1
>> >
>> > As an initiator and user of paramserv func, I just launched
>several tests
>> > on local pc with a script using paramserv func without mkl. And
>no bug is
>> > observed.
>> >
>> > Regards,
>> > Guobao
>> >
>> > On Sun, Aug 19, 2018 at 8:09 PM Matthias Boehm
>
>> wrote:
>> >
>> > > +1
>> > >
>> > > I ran the perftest suite multiple times up to 80GB with and
>without
>> > > codegen. After fixing all the issues and regressions, the
>entire suite
>> > > ran successfully against Spark 2.2 and 2.3 and all use cases
>showed
>> > > equal or better performance compared to SystemML 1.1.
>> > >
>> > > Regards,
>> > > Matthias
>> > >
>> > > On Fri, Aug 17, 2018 at 8:41 AM, Berthold Reinwald <
>> reinw...@us.ibm.com>
>> > > wrote:
>> > > > Please vote on releasing the following candidate as Apache
>SystemML
>> > > > version 1.2.0
>> > > >
>> > > > The vote is open for at least 72 hours and passes if a
>majority of at
>> > > > least 3 +1 PMC votes are cast.
>> > > >
>> > > > [ ] +1 Release this package as Apache SystemML 1.2.0
>> > > > [ ] -1 Do not release this package because ...
>> > > >
>> > > > To learn more about Apache SystemML, please see
>> > > >
>INVALID URI REMOVED
>rg_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6Sa
>OkZrfiSrYZVby0=Uqz4fylsOCFVH3PSKrkwgncGqidpgrr7TfqU8ejtt_8=057nyK
>fCe4JiHKKjqQ3xTNjpdfcOOj1wrdKzhd8U860=
>> > > >
>> > > >
>> > > > The tag to be voted on is v1.2.0-rc1 (
>> > > > a1a05e29f6ee78f3c33fea355f62c78ce21766ee):
>> > > >
>INVALID URI REMOVED
>e_systemml_tree_v1.2.0-2Drc1=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=HzVC
>6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=Uqz4fylsOCFVH3PSKrkwgncGqid
>pgrr7TfqU8ejtt_8=jUXES2WEsh-ZQ2dHQ95veWxLknf5vVgZQDB28fsX-4o=
>> > > >
>> > > >
>> > > > The release artifacts can be found at:
>> > > >
>INVALID URI REMOVED
>repos_dist_dev_systemml_1.2.0-2Drc1_=DwIBaQ=jf_iaSHvJObTbx-siA1ZO
>g=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=Uqz4fylsOCFVH3PSKrk
>wgncGqidpgrr7TfqU8ejtt_8=1Z9SWlyHbLIlpCzNvqbU5rJg9ejYxQfQ75d5kdYBDg
>k=
>> > > >
>> > > >
>> > > > The maven release artifacts, including signatures, digests,
>etc. can
>> be
>> > > > found at:
>> > > >
>> > >
>> >
>>
>INVALID URI REMOVED
>e.org_content_repositories_orgapachesystemml-2D1030_org_apache_system
>ml_systemml_1.2.0_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQr
>pc383_Kao_6a6SaOkZrfiSrYZVby0=Uqz4fylsOCFVH3PSKrkwgncGqidpgrr7TfqU8
>ejtt_8=pNN1jhtVSsI6fmBJKHHGlbKh9lAoWyTo6CXAZ2Mmkuw=
>> > > >
>> > > >
>> > > >
>> > > > ===
>> > > > == Apache Release policy ==
>> > > > ===
>> > > >
>INVALID URI REMOVED
>gal_release-2Dpolicy.html=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=HzVC6v7
>9boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=Uqz4fylsOCFVH3PSKrkwgncGqidpgr
>r7TfqU8ejtt_8=TSCGVXvU49JsYjd2rP6VheHZmIuflH057IqWCvb-0HY=
>> > > >
>> > > >
>> > > > ===
>> > > > == How can I help test this release? ==
>> > > > ===
>> > > > If you are a SystemML user, you can help us test this release
>by
>> taking
>> > > an
>> > > >
>> > > >
>> > > > existing Algorithm or workload and running on this release
>candidate,
>> > > then
>> > > >
>> > > >
>> > > > reporting any regressions.
>> > > >
>> > > > 
>> > > > == What justifies a -1 vote for this release? ==
>> > > > 
>> > > > -1 votes should only occur for significant stop-ship bugs or
>legal
>> > > related
>> > > >
>> > > >
>> > > > issues (e.g. wrong license, missing header files, etc). Minor
>bugs or
>> > > > regressions should not block this release.
>> > > >
>> > > >
>> > > >
>> > > > Regards,
>> > > > Berthold Reinwald
>> > > > IBM Almaden Research Center
>> > > > office: (408) 927 2208; T/L: 457 2208
>> > > > e-mail: reinw...@us.ibm.com
>> > > >
>> > >
>> >
>>
>

Re: GSoC Project Presentation Guobao (Parameter Server)

[Niketan Pansare]

Congratulations Guobao!

> On Aug 9, 2018, at 11:27 PM, Krishna Kalyan  wrote:
> 
> Congratulations Guobao

Re: `conv2d_backward_data()` received "Incorrect dimensions" error. Thanks

[Niketan Pansare]

Hi Janardhan,

The error is due to incorrect usage of the builtin function. The first
parameter should be the filter matrix and second parameter should be the
error matrix from the next layer. Note: the error matrix will be same size
as the input matrix only in case of same padding. From
https://stackoverflow.com/questions/37674306/what-is-the-difference-between-same-and-valid-padding-in-tf-nn-max-pool-of-t
:

   For the SAME padding, the output height and width are computed as:
out_height = ceil(float(in_height) / float(strides[1]))
out_width = ceil(float(in_width) / float(strides[2]))
And
   For the VALID padding, the output height and width are computed as:
out_height = ceil(float(in_height - filter_height + 1) / float(strides[1]))
out_width = ceil(float(in_width - filter_width + 1) / float(strides[2]))

I would also recommend checking out our nn library (
https://github.com/apache/systemml/blob/master/scripts/nn/layers/conv2d_builtin.dml#L127
) and DML documentation (
http://apache.github.io/systemml/dml-language-reference.html#deep-learning-built-in-functions
).

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   Janardhan 
To: dev@systemml.apache.org
Date:   07/18/2018 07:16 AM
Subject:`conv2d_backward_data()` received "Incorrect dimensions" error.
Thanks



Hi, I used the following code same as found in reference. I am receiving
error as ` Incorrect dimensions in DnnOp: 25 != 9`

C = 1; H = 5; W = 5
input = matrix(seq(1, C*H*W), rows=1, cols=C*H*W)
filter = matrix(seq(1, C*3*3), rows=1, cols=C*3*3)
output = conv2d_backward_data(input, filter, stride=[1,1],
padding=[1,1], input_shape=[1,C,H,W], filter_shape=[1,C,3,3])
print(toString(matrix(output, rows=5, cols=5)))

I do not fully know about this function works, please accept sorry if the
report is incorrect.

Thank you,
Janardhan

Re: which() Function in SystemML

[Niketan Pansare]

Hi Kuno,

Thanks for trying out SystemML. One way to implement your logic in SystemML
is as follows:
X = matrix("1 4 -99 2 5 -99", rows=6, cols=1)
Y = removeEmpty(target=(X > 2)*seq(1, nrow(X)), margin="rows")
print("\nIndexes:\n" + toString(Y))

For handling NAs, you can replace NAs by certain value (in this case -99 or
one of the builtin constants, i.e. Inf or NaN ... see
https://github.com/apache/systemml/blob/master/src/test/scripts/org/apache/sysml/api/mlcontext/builtin-constants-test.dml
) using the "na.strings" parameter of the transform builtin function (
http://apache.github.io/systemml/dml-language-reference.html#transforming-frames
).

Hope this helps.

Regards,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   "Baeriswyl Kuno (IT-SWE-CC1-JV6)" 
To: "dev@systemml.apache.org" 
Date:   07/05/2018 05:29 AM
Subject:which() Function in SystemML



Hello,

I'm a developer want to use SystemMl for running R-Code from our business
people on a Spark cluster.

I've studied
http://apache.github.io/systemml/dml-language-reference
 , however, haven't found a implementation of the R function "which" or any
alternative functionality. Has anyone an idea how I could

# Given
v = c(1,4,NA,2, 5, NA)

# Expect indexes where value meets condition =  int[] 2 5
v2 = which(v>2)

# Expect indexes where is.na returns TRUE =   int[] 3 6
v3 = which(is.na(v))

I've already considered the functions replace() and removeEmpty(), but they
don't exactly meets my needs.

Thanks a lot in advance
Kuno

Re: [DISCUSS] Blocksparse kernels

[Niketan Pansare]

Hi Janardhan,

>> 1. Can you help me, estimate how much would it take to implement
blocksparse kernels practically.
This is a difficult question to answer as it depends on how comfortable you
are with writing and optimizing sparse kernels. To implement block-sparse
kernel as per your document, one needs to know (each step is progressively
more difficult than the previous step):
1. How to implement and compile a simple CUDA kernel.
2. How to implement a non-block sparse kernel such that results match with
the CuSPARSE code.
3. How to optimize a non-block sparse kernel such that performance match
with the CuSPARSE code.
4. How to optimize a non-block sparse kernel for a given hardware such that
performance match with the CuSPARSE code on that hardware. This requires
working knowledge of different Nvidia devices and how to tweak sass code.
5. How to implement a block sparse kernel such that results match with the
CuSPARSE code.
6. How to optimize a block sparse kernel such that performance match with
the CuSPARSE code.
7. How to optimize a block sparse kernel for a given hardware such that
performance match with the CuSPARSE code on that hardware.

I would recommend before attempting a block-sparse kernel:
- Picking up JIRAs that will help you through steps (1)-(3). May be, you
want to try SYSTEMML-937 and SYSTEMML-2312/SYSTEMML-2313 first.
- You can skip (4) and (7) as it involves maintenance overhead.

For (2)-(3), you can either use SystemML or the CUDA code I sent in the
earlier thread as baseline.

>> 2. Would like to spare some time to review PRs ( ~2 PRs per week).
Sure. You may want to batch similar PRs as it reduces turn-around time.

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   Janardhan <janard...@apache.org>
To: dev@systemml.apache.org, Niketan Pansare <npan...@us.ibm.com>,
reinw...@us.ibm.com, Nakul Jindal <naku...@gmail.com>
Date:   05/10/2018 10:34 AM
Subject:[DISCUSS] Blocksparse kernels



Hi Niketan, Nakul, and Berthold,

1. Can you help me, estimate how much would it take to implement
blocksparse kernels practically.

2. Would like to spare some time to review PRs ( ~2 PRs per week).

a. Relevant Jira: https://issues.apache.org/jira/browse/SYSTEMML-2041

b. My proposal:

https://docs.google.com/document/d/1cgPdyhhG3kQZxeP1VYOnQoZuTVA2216CC_aWdurmTNw/edit?usp=sharing

c. Research paper:
https://s3-us-west-2.amazonaws.com/openai-assets/blocksparse/blocksparsepaper.pdf

Thank you,
Janardhan

Re: GSoC 2018 Student Guobao Li

[Niketan Pansare]

Welcome, Guobao Li.

> On May 1, 2018, at 11:54 PM, Berthold Reinwald  wrote:
> 
> Welcome, Guobao

Re: Fwd: Fw: Request for a beginner JIRA

[Niketan Pansare]

Hi Daiki/Matthias,

I have updated the relevant JIRA for Python testsuite with my initial
thoughts: https://issues.apache.org/jira/browse/SYSTEMML-1092.  Let's use
that for further discussion :)

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   Matthias Boehm <mboe...@gmail.com>
To: dev@systemml.apache.org
Date:   04/03/2018 10:33 PM
Subject:Fwd: Fw: Request for a beginner JIRA



Thanks for your interest Daiki. I created two JIRAs SYSTEMML-2233 and
SYSTEMML-2232 that might me a good starting point. I would recommend
to begin with 2233 as a basic cleanup task, which is meant to get you
comfortable. The other task is then a bit more involved but would
improve our function namespace handling (e.g., for better statistics
outputs).

If you are mostly interested in working on the Python APIs, an awesome
project would be to include the Python MLContext API into our
testsuite (or a separate Python testsuite). @Niketan: Do you have more
thoughts on that? Our dev environment anyway requires Python for the
distribution build.

Regarding the development environment, you would typically clone the
repository, setup your preferred IDE and work on local branches for
new features. Once a change is ready for review, you submit a PR which
automatically triggers a run of our testsuite on our jenkins
infrastructure. You can also run these tests locally through maven or
directly through junit, which is especially useful for debugging. The
same applies for Python as well but as mentioned above the Python API
is not yet part of the automated testsuite, so yes changes there
involve semi-manual testing.

The issue you've encountered with Java 9 might be related to [1]. I
just gave it a try and core SystemML compiles and runs fine with Java
9 when build directly through the IDE. If it's related to Scala, we
should find and discuss a resolution for our next 1.2 release.

[1]
https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_49016074_systemml-2Dbuild-2Dfailed-2Don-2Dmacos=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=gIVm8wmVG2A-WnsDIh6tyZW7wGqqs-hcSJT37K0omFY=FZmIus_BmseG0Yr7V7_pAE45pDUVtGeKQCMOClKbADk=


Regards,
Matthias

From: Daiki Matsunaga/Japan/IBM
To: dev@systemml.apache.org, Matthias Boehm1/US/IBM@IBM
Date: 04/03/2018 12:46 AM
Subject: Request for a beginner JIRA


HI everyone,

My name is Daiki from IBM Japan and I'm just starting out with
building/going through some examples so I can get used to the project
before contributing.

Request for JIRA

I would like some guidance on choosing which JIRA issue to choose
from. I don't have any strong preference for which part of SystemML to
contribute,but

Questions

I apologize in advance if this is too simple but what are the basic
steps involved during the iteration of development, build, test,
especially for Python. For example, do you write out code in Java or
Python on the source (e.g. a feature branched out from master), build,
and pip install and then check on Jupyter notebook? Or do you directly
edit the Python code that is generated after pip install?

Some things I noticed during install/build

Problem: It says Java 8+ is required but Java 9 does not work. With
Java 9 installed on my machine(Mac OS Sierra 10.12.5. ), there was a
build failure with "mvn clean package" as well as in the pyspark shell
with "ml = MLContext(sc)". This would be an issue with the current
documentation since it automatically downloaded Java 9 with "brew
install Caskroom/cask/java".
Solution: Since I had both versions of Java, I set the bash profile
Java environment variable to the following and it worked:
export JAVA_HOME="$(/usr/libexec/java_home -v 1.8)"






Daiki Matsunaga
Cognitive AI
GBS, IBM Japan
Phone: 050-3150-7460 / 080-5915-7460
E-mail: e36...@jp.ibm.com

Re: Draft Release Notes 1.1.0

[Niketan Pansare]

Thanks Berthold. Overall, the draft looks good to me. Few minor additions:

New Builtin Functions
- Deep learning builtin functions: avg_pool and avg_pool_backward
- assert

New Layers in the NN library:
- Average pooling
- Upsampling
- Low-rank fully connected

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   "Berthold Reinwald" <reinw...@us.ibm.com>
To: dev@systemml.apache.org
Date:   03/28/2018 04:02 PM
Subject:Draft Release Notes 1.1.0



Please add/update below release note draft prior to putting it on our
website. Please also go to the JIRA release notes link at the end, and
update version/status if necessary. Thanks.


* Release Notes - SystemML - Version SystemML 1.1

New Capabilities/Features
- Dense matrix blocks >16GB, and operations
- Support bitwise operators not, and, or, xor, & LShift, Rshift
- Additional ParFor result aggregation operations
- UDFs callable in expressions
- zero rws/columns matrices
- Extended Caffe2DML and Keras2DML APIs

New Builtin Functions
- ifelse()

Performance Improvements
- Ultra-sparse operations

Internals
- Single-precision support for native conv2d and mm operations.
- Consolidate replicated compilation chain
- Generalize Binary Operations to (vector, matrix) Operands
- Use common thread pool

Bug Fixes
- in APIs, performance bugs, optimizer, runtime, GPU backend, Spark
backend.

Experimental
- Codegen

Deprecate
- Support for Spark 2.1/2.2 (make switch to newer ANTLR version)

Detailed JIRA release notes are here:
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_ReleaseNote.jspa-3Fversion-3D12342282-26styleName-3DText-26projectId-3D12319522-26Create-3DCreate-26atl-5Ftoken-3DA5KQ-2D2QAV-2DT4JA-2DFDED-257C2e97f76f75a466564dba2a7d0bec5ca06bffd66e-257Clout=DwIFAg=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=fku28ex5cFV8MRqEKRuh-_I3UTefK_Qx_YXQD9UfoWU=K6GTOjWQiV6jki5_tkBF1tXLL2SM2Jta2ZXFdjBYbQw=



Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinw...@us.ibm.com

Re: Release Planning

[Niketan Pansare]

+1.

We should consider including single precision native BLAS in the release notes 
as well. If possible, we should add JNI wrappers for PowerPC, Windows and Mac 
too in this release. 

> On Feb 6, 2018, at 12:27 PM, Berthold Reinwald  wrote:
> 
> sure.
> 
> Makes sense. Codegen and Keras2DML made good progress, and many other 
> fixes/improvements.
> 
> What else do we want time/track/highlight for it?
> 
> Regards,
> Berthold Reinwald
> IBM Almaden Research Center
> office: (408) 927 2208; T/L: 457 2208
> e-mail: reinw...@us.ibm.com
> 
> 
> 
> From:   Matthias Boehm 
> To: dev@systemml.apache.org
> Date:   02/05/2018 11:05 PM
> Subject:Release Planning
> 
> 
> 
> Hi all,
> 
> since our 1.0 release in Dec, we already got a number enhancements and new
> features in, so I think it would be good to discuss the timeline for our
> next SystemML 1.1 release. How about, we target mid March for a first RC?
> Also, Berthold would you be willing to serve again as the release manager?
> 
> Regards,
> Matthias
> 
> 
> 
> 

Re: dev environment on windows

[Niketan Pansare]

Hi Janardhan,

The artifact in your error message
http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22jcuda-natives%22 is
available on maven central. However, it is supported only for 64-bit JVM
and hence org.jcuda:jcuda-natives:jar:${jcuda.os}-${jcuda.arch}:0.8.0 fails
. Can you please set JAVA_HOME to something like "C:\Program Files\Java
\jdk1.8.0_151"  (in case you have 64-bit JDK) ?

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   Janardhan Pulivarthi <janardhan.pulivar...@gmail.com>
To: dev@systemml.apache.org
Date:   12/06/2017 08:17 AM
Subject:Re: dev environment on windows



Hi, - Thanks all for the help.

BTW, my centOS 7 server is working fine on google cloud.

1. I resolved JDK issue by presenting `JAVA_HOME` instead of placing direct
folder path(C:\Program Files (x86)\Java\jdk1.8.0_151) in `PATH`.

2.  Before running, I have deleted `~/.m2` folder.
     (a) If I run `mvn clean package` through windows cmd, or
     (b) If I run mvn through life cycle option in IntelliJ (as per Dylan's
advice), the error thrown is...

[ERROR] Failed to execute goal on project systemml: Could not resolve
dependencies for project org.apache.systemml:systemml:jar:1.0.0-SNAPSHOT:
The following artifacts could not be resolved:
org.jcuda:jcuda-natives:jar:${jcuda.os}-${jcuda.arch}:0.8.0,
org.jcuda:jcublas-natives:jar:${jcuda.os}-${jcuda.arch}:0.8.0,
org.jcuda:jcusparse-natives:jar:${jcuda.os}-${jcuda.arch}:0.8.0,
org.jcuda:jcusolver-natives:jar:${jcuda.os}-${jcuda.arch}:0.8.0,
org.jcuda:jcudnn-natives:jar:${jcuda.os}-${jcuda.arch}:0.8.0: Could not
find artifact org.jcuda:jcuda-natives:jar:${jcuda.os}-${jcuda.arch}:0.8.0
in central (https://repo1.maven.org/maven2) -> [Help 1]


Where as I could `org/cuda` folder in mvn repo and my local repo. (as shown
below).


​


​
​

Thanks a lot,
Janardhan


On Tue, Dec 5, 2017 at 2:43 AM, Niketan Pansare <npan...@us.ibm.com> wrote:
  >> How to build systemml on windows, as there is a backslash problem.

  Simplest option: Compile SystemML from command-line (assuming you have
  installed maven: https://maven.apache.org/install.html)
  cd 
  mvn package
  OR
  mvn package -P distribution

  This will help separate compilation and IDE issues. Can you please
  elaborate on "the backslash problem" ?

  >> When build systemml on intellij idea, it has the compilation error
  with caffe2dml class.

  Also, please note that Caffe2DML is a scala class. You may want to check
  following two issues:
  1. I have personally not tested the IntelliJ's scala integration, but as
  per Dylan's email, it works fine for him. As per our pom, we recommend
  Scala 2.11.
  2. Java version issue pointed in the Berthold's email. As an FYI, I have
  not reproduced this issue on my Scala IDE setup (http://scala-ide.org/).

  If we are missing any step, we may want to update our IDE guide:
  
http://apache.github.io/systemml/developer-tools-systemml#import-systemml-project-to-intellij


  Thanks,

  Niketan Pansare
  IBM Almaden Research Center
  E-mail: npansar At us.ibm.com
  http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar

  "Berthold Reinwald" ---12/04/2017 01:00:12 PM---Please make sure that you
  specify a JRE/JDK of 1.8 when you use a maven run configuration.

  From: "Berthold Reinwald" <reinw...@us.ibm.com>
  To: dev@systemml.apache.org
  Date: 12/04/2017 01:00 PM
  Subject: Re: dev environment on windows



  Please make sure that you specify a JRE/JDK of 1.8 when you use a maven
  run configuration.

  If I switch the runtime JRE in my maven configuration back to jdk1.7,
  then
  I observe the same caffe2dml compilation error. caffe2dml randomly seems
  to be the 1st class being compiled to throw the error.

  Regards,
  Berthold Reinwald
  IBM Almaden Research Center
  office: (408) 927 2208; T/L: 457 2208
  e-mail: reinw...@us.ibm.com



  From:   Dylan Hutchison <dhutc...@cs.washington.edu>
  To:     dev@systemml.apache.org
  Date:   12/02/2017 01:19 PM
  Subject:        Re: dev environment on windows



  I have used Intellij on windows and had no problems.  Did you open a new
  project from SystemML's pom.xml?  If so, it should automatically
  configure
  itself to match the pom.  Then you can use the "Maven projects" tab on
  the
  right side of Intellij to run "mvn package".

  On Sat, Dec 2, 2017 at 1:13 PM, Matthias Boehm <mboe...@gmail.com> wrote:

  > I'm using eclipse on win and simply defined two maven run
  configurations
  > for
  >
  > (1) the default build (base=${workspace_loc:/systemml}, goals=package),
  > and
  > (2) the distribution build ((base=${workspace_loc:/systemml},
  > goals=package, profiles=distribution).
  >
  > I'm sure intellij provides a similar builtin maven support, but maybe
  an

Re: dev environment on windows

[Niketan Pansare]

>> How to build systemml on windows, as there is a backslash problem.

Simplest option: Compile SystemML from command-line (assuming you have
installed maven: https://maven.apache.org/install.html)
cd 
mvn package
OR
mvn package -P distribution

This will help separate compilation and IDE issues. Can you please
elaborate on "the backslash problem" ?

>> When build systemml on intellij idea, it has the compilation error with
caffe2dml class.

Also, please note that Caffe2DML is a scala class. You may want to check
following two issues:
1. I have personally not tested the IntelliJ's scala integration, but as
per Dylan's email, it works fine for him. As per our pom, we recommend
Scala 2.11.
2. Java version issue pointed in the Berthold's email. As an FYI, I have
not reproduced this issue on my Scala IDE setup (http://scala-ide.org/).

If we are missing any step, we may want to update our IDE guide:
http://apache.github.io/systemml/developer-tools-systemml#import-systemml-project-to-intellij

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   "Berthold Reinwald" <reinw...@us.ibm.com>
To: dev@systemml.apache.org
Date:   12/04/2017 01:00 PM
Subject:Re: dev environment on windows



Please make sure that you specify a JRE/JDK of 1.8 when you use a maven
run configuration.

If I switch the runtime JRE in my maven configuration back to jdk1.7, then
I observe the same caffe2dml compilation error. caffe2dml randomly seems
to be the 1st class being compiled to throw the error.

Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinw...@us.ibm.com



From:   Dylan Hutchison <dhutc...@cs.washington.edu>
To: dev@systemml.apache.org
Date:   12/02/2017 01:19 PM
Subject:Re: dev environment on windows



I have used Intellij on windows and had no problems.  Did you open a new
project from SystemML's pom.xml?  If so, it should automatically configure
itself to match the pom.  Then you can use the "Maven projects" tab on the
right side of Intellij to run "mvn package".

On Sat, Dec 2, 2017 at 1:13 PM, Matthias Boehm <mboe...@gmail.com> wrote:

> I'm using eclipse on win and simply defined two maven run configurations
> for
>
> (1) the default build (base=${workspace_loc:/systemml}, goals=package),
> and
> (2) the distribution build ((base=${workspace_loc:/systemml},
> goals=package, profiles=distribution).
>
> I'm sure intellij provides a similar builtin maven support, but maybe an
> intellij user can comment on that as well.
>
> Regards,
> Matthias
>
>
> On Sat, Dec 2, 2017 at 10:08 AM, Janardhan Pulivarthi <
> janardhan.pulivar...@gmail.com> wrote:
>
> > Hi all,
> >
> > 1. How to build systemml on windows, as there is a backslash problem.
> >
> > 2. When build systemml on intellij idea, it has the compilation error
> with
> > caffe2dml class.
> >
> >
> > Thanks.
> >
>

Re: SystemML 1.0 release timeline

[Niketan Pansare]

+1.

Thanks,

Niketan.

> On Nov 7, 2017, at 10:50 PM, Matthias Boehm 
wrote:
>
> Hi all,
>
> we made some good progress regarding deep learning support, code
> generation, and low-latency scoring - so, I'm looking forward to our
> upcoming 1.0 release. Since it's our first stable release, I think it
would
> be a good idea to allocate some extra time for QA. How about we shoot for
a
> release candidate Dec 1?
>
> Regards,
> Matthias

[DISCUSS] Support for lower precision in SystemML

[Niketan Pansare]

Hi all, 

We are in process of adding support for lower precision and wanted to give 
everyone heads up. By lower precision, I mean support storing matrices in float 
array (or half precision array) and performing operations using float kernels. 
Initial experiments suggest that we can get up to 2x improvements in terms of 
performance for Deep Learning algorithms. Also, this reduces  the memory 
requirements by 2x. 

Please provide any concerns or suggestions. 

The high-level plan is as follows:
1. Support lower precision on GPU. Please see 
https://github.com/apache/systemml/pull/688
2. Support lower precision with native BLAS.
3. Support lower precision on CP/Spark. This includes writing float matrix in 
binary format and updating memory estimation in hops.
4. Extend Python APIs to support lower precision.

The first two steps requires the conversion of double array to float/half 
precision array.

Thanks 

Niketan.

Re: [PROPOSAL] R4ML Integration with SystemML

[Niketan Pansare]

>> a) Does it mean you are proposing spliting R4ML into two R-wrapper and
R4ML?

I was only suggesting how you ought to stage the PRs into SystemML once the
vote passes :)

>> So I was thinking is it absolutely must have to sync between api?

Soft-yes, we should try our best to do so.

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   alok singh <singh_a...@hotmail.com>
To: "dev@systemml.apache.org" <dev@systemml.apache.org>
Date:   09/22/2017 02:40 PM
Subject:Re: [PROPOSAL] R4ML Integration with SystemML




see comments Alok:



From: Niketan Pansare <npan...@us.ibm.com>
Sent: Friday, September 22, 2017 2:11 PM
To: dev@systemml.apache.org
Subject: Re: [PROPOSAL] R4ML Integration with SystemML

>>> As pointed out earlier, R4ML is not just R interface so it is based on
the earlier product of IBM on R and it has many product feature.
Also note that the pure ML Ctx and the cmd options for dml is not ideally
allow all the things user want to do in his ML code.
The solution could be to create wrapper to make user happy  . but we have
created those wrapper but those are in R and from user point for view it
feels that are just writing the R code
If the ultimate goal is to have just MLCtx based R interface than I think
it undermines and R4ML value proposition.
(We can definitely just expose MLCtx api. However calling Logistic
Regression example just for the purpose of MLCtx won't be best) R4ML.mlogit
has better apis

May be we are not on same page.
(a) MLContext is not the only API, but an important one that needs to be
supported.
(b) Like R4ML, our mllearn wrappers aim to simplify the usage for the
Python users. These wrappers were designed so that if someone wrote a
python script that uses scikit-learn or mllib. Then, a simple change from
`from sklearn import LogisticRegression`  to `from systemml.mllearn import
LogisticRegression` should in principle allow SystemML to be incorporated
in their workflow.

Alok:
a) Does it mean you are proposing spliting R4ML into two R-wrapper and
R4ML? I think that could be idea one can potentially look into it. I second
it. That way one can have pure R wrapper and like mllearn kind of R4ML

b) Currently we can sure expose the MLContext from R as public api but to
use all the code involves many convulations to make life easier for R user.
For example see code func *execute* *output* *getDF* in
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_aloknsingh_r4ml_blob_0d79b3c7975be55989466869fe99ccfd47dd6dc3_R4ML_R_sysml.bridge.R=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=SivKuY8Zn0LQAmM2UmppEwy4L-lROLYUzT9iYnS4Njg=VZv9IEtLnaXzZ3mp1bICD4zRv3SL2VO7b68H0wHTCis=


>> 1) I think it will require a lot of work for scala and python api to be
in sync with r4ml api.
Also I feel that if the goal is too have just python, scala than we have to
do the coding at R4ML. but I think goals was to merge this project.

I guess the goal is to make SystemML better and more user-friendly. To do
that, we have to try our best to keep our APIs across language consistent.
I understand it might require lot of work for Scala and Python APIs to be
in sync with R4ML API,  but it has to be done.


Since R4ML was designed in isolation with the SystemML project, I am
recommending to do a gradual merge of (1) the additional features and (2)
features that diverge from SystemML APIs so as to be R friendly; thus,
allowing the SystemML community  to comment on them before merging. This
also allows the R4ML features that match one-to-one with the Python and
Scala APIs to be merged quickly and not be in the PR until we agree to
every (1) and (2) features :)

Alok: See the previous comments I like we should explore the idea of
splitting the way you splitted mllearn. Still more discussion needed as I
see it.  At this stage those changes will require complete change at R4ML
to have those.

Another way to think would be that R4ML can be independent package, which
eventually be pushed to CRAN.
note that in the spark dev repo. Spark core is there and SparkR is there as
seperate dir and python is there as seperate dir

Initially, SparkScala, SparkR and pyspark tried to be in sync but I think
now many features are been added which is not causing sync between sparkR
and pyspark and similar between SparkScala and SpakR and  PySpark.

So I was thinking is it absolutely must have to sync between api? Since all
these will cater to different user.

These are ideas.




Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar

 http://researcher.watson.ibm.com/researcher/photos/3531.jpg

Niketan Pansare - IBM
researcher.watson.ibm.com
Niketan Pansare is a Senior Software Engineer at IBM Research Almaden,
wh

Re: [PROPOSAL] R4ML Integration with SystemML

[Niketan Pansare]

>>> As pointed out earlier, R4ML is not just R interface so it is based on
the earlier product of IBM on R and it has many product feature.
Also note that the pure ML Ctx and the cmd options for dml is not ideally
allow all the things user want to do in his ML code.
The solution could be to create wrapper to make user happy  . but we have
created those wrapper but those are in R and from user point for view it
feels that are just writing the R code
 If the ultimate goal is to have just MLCtx based R interface than I think
it undermines and R4ML value proposition.
(We can definitely just expose MLCtx api. However calling Logistic
Regression example just for the purpose of MLCtx won't be best) R4ML.mlogit
has better apis

May be we are not on same page.
(a) MLContext is not the only API, but an important one that needs to be
supported.
(b) Like R4ML, our mllearn wrappers aim to simplify the usage for the
Python users. These wrappers were designed so that if someone wrote a
python script that uses scikit-learn or mllib. Then, a simple change from
`from sklearn import LogisticRegression` to `from systemml.mllearn import
LogisticRegression` should in principle allow SystemML to be incorporated
in their workflow.

>> 1) I think it will require a lot of work for scala and python api to be
in sync with r4ml api.
Also I feel that if the goal is too have just python, scala than we have to
do the coding at R4ML. but I think goals was to merge this project.

I guess the goal is to make SystemML better and more user-friendly. To do
that, we have to try our best to keep our APIs across language consistent.
I understand it might require lot of work for Scala and Python APIs to be
in sync with R4ML API, but it has to be done.

Since R4ML was designed in isolation with the SystemML project, I am
recommending to do a gradual merge of (1) the additional features and (2)
features that diverge from SystemML APIs so as to be R friendly; thus,
allowing the SystemML community to comment on them before merging. This
also allows the R4ML features that match one-to-one with the Python and
Scala APIs to be merged quickly and not be in the PR until we agree to
every (1) and (2) features :)

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   alok singh <singh_a...@hotmail.com>
To: "dev@systemml.apache.org" <dev@systemml.apache.org>,
"de...@apache.org" <de...@apache.org>
Date:   09/22/2017 12:30 PM
Subject:Re: [PROPOSAL] R4ML Integration with SystemML





Here are Niketan's question

Thanks for taking time to answer our questions and also for considering to
help SystemML community. I have couple more questions:

Niketan:1.
 In case there is inconsistency, do you (as R4ML developers) feel
comfortable changing R4ML interface to be compatible with our other APIs ?
May be you can go over the below two links and imagine adding a
corresponding R tab:
- MLContext Programming guide:
https://urldefense.proofpoint.com/v2/url?u=http-3A__apache.github.io_systemml_spark-2Dmlcontext-2Dprogramming-2Dguide=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=d7aHl15rr92bxoHo26sphduc7Q_4C0GizrRv_AR5pEM=xyErlMsfwKjn_qfkXHpjLG8E1B70N5zVX-OWl5LU-yU=


apache.github.io<
https://urldefense.proofpoint.com/v2/url?u=http-3A__apache.github.io_systemml_spark-2Dmlcontext-2Dprogramming-2Dguide=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=d7aHl15rr92bxoHo26sphduc7Q_4C0GizrRv_AR5pEM=xyErlMsfwKjn_qfkXHpjLG8E1B70N5zVX-OWl5LU-yU=
 >
apache.github.io
Spark MLContext Programming Guide. Overview; Spark Shell Example. Start
Spark Shell with SystemML; Create MLContext; Hello World; LeNet on MNIST
Example; DataFrame ...



- Algorithm wrappers:
https://urldefense.proofpoint.com/v2/url?u=http-3A__apache.github.io_systemml_algorithms-2Dclassification.html-23multinomial-2Dlogistic-2Dregression=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=d7aHl15rr92bxoHo26sphduc7Q_4C0GizrRv_AR5pEM=TpQy-5v3cbhFJfGbEodsNvhrU8gDWexYBwN9x2eXzlc=


ALOK: Hi Niketan

 As pointed out earlier, R4ML is not just R interface
so it is based on the earlier product of IBM on R and it has many product
feature.

Also note that the pure ML Ctx and the cmd options for dml is not ideally
allow all the things user want to do in his ML code.
The solution could be to create wrapper to make user happy  . but we have
created those wrapper but those are in R and from user point for view it
feels that are just writing the R code

see some of the examples at

https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_SparkTC_r4ml_tree_master_R4ML_inst_examples=DwIFAw=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=d7aHl15rr92bxoHo26sphduc7Q_4C0GizrRv_AR5pEM=r4-fcsboHpxlbVf6KyY7C6ptdLcjmyT2g1h

Re: Consistency SystemML configuration properties

[Niketan Pansare]

+1 for consistency especially before 1.0 release.

Thanks 

Niketan 

> On Sep 16, 2017, at 4:50 PM, Glenn Weidner  wrote:
> 
> +1 for 'sysml'.
> 
> Thanks,
> Glenn
> 
> Matthias Boehm ---09/16/2017 04:34:46 PM---Currently, our SystemML 
> configuration properties use an inconsistent prefix scheme. For example, som
> 
> From: Matthias Boehm 
> To: dev@systemml.apache.org
> Date: 09/16/2017 04:34 PM
> Subject: Consistency SystemML configuration properties
> 
> 
> 
> 
> Currently, our SystemML configuration properties use an inconsistent prefix
> scheme. For example, some properties use the prefix 'dml' (e.g.,
> dml.yarn.appmaster), others 'systemml' (e.g., systemml.stats.finegrained),
> and yet others no prefix at all (e.g., localtmpdir).
> 
> We discussed this before but we never actually made the change. I would
> prefer the prefix 'sysml' for consistency with our package names. Are there
> any other preferences? If not, I'll make this change end of next week.
> 
> Regards,
> Matthias
> 
> 
> 

[DISCUSS] Package jcuda of other platforms in the extra jar

[Niketan Pansare]

Hi all,

Currently, we only package jcuda's linux shared libraries in our python pip
package (as most SystemML developers use linux for testing the GPU
backend). This makes it difficult to use the GPU backend on other platform
via our Python APIs. To improve the usability, I would suggest that we
package the jcuda jars of other platforms as well. The drawback of this
approach is that size of the extra jar will increase.

Please let me know your thoughts.

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar

Re: Install - Configure Jupyter Notebook

[Niketan Pansare]

Hi Gustavo,

You can paste that code into the commandline:
$ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS="notebook"
pyspark --master local[*] --conf "spark.driver.memory=12g" --conf
spark.driver.maxResultSize=0 --conf spark.akka.frameSize=128 --conf
spark.default.parallelism=100

The above command tells "pyspark" that the python driver is jupyter. For
more details, please see
https://github.com/apache/spark/blob/master/bin/pyspark#L27

Alternatively, you can follow Arijit's suggestion.

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar



From:   arijit chakraborty <ak...@hotmail.com>
To: "dev@systemml.apache.org" <dev@systemml.apache.org>
Date:   07/02/2017 04:22 AM
Subject:Re: Install - Configure Jupyter Notebook



Hi Gustavo,


You can put that pyspark details in the jupyter console itself.


import os
import sys
import pandas as pd
import numpy as np

spark_path = "C:\spark"
os.environ['SPARK_HOME'] = spark_path
os.environ['HADOOP_HOME'] = spark_path

sys.path.append(spark_path + "/bin")
sys.path.append(spark_path + "/python")
sys.path.append(spark_path + "/python/pyspark/")
sys.path.append(spark_path + "/python/lib")
sys.path.append(spark_path + "/python/lib/pyspark.zip")
sys.path.append(spark_path + "/python/lib/py4j-0.10.4-src.zip")

from pyspark import SparkContext
from pyspark import SparkConf

sc = SparkContext("local[*]", "test")


# SystemML Specifications:


from pyspark.sql import SQLContext
import systemml as sml
sqlCtx = SQLContext(sc)
ml = sml.MLContext(sc)


But this is not a very good way of doing it. I did it as I'm using windows
and it's easier to do it like that.


Regards,

Arijit


From: Gustavo Frederico <gustavo.freder...@thinkwrap.com>
Sent: Sunday, July 2, 2017 10:16:03 AM
To: dev@systemml.apache.org
Subject: Install - Configure Jupyter Notebook


A basic question: step 3 in
https://systemml.apache.org/install-systemml.html <
https://systemml.apache.org/install-systemml.html>  for “Configure Jupyter
Notebook” has
# Start Jupyter Notebook Server
PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS="notebook" pyspark
--master local[*] --conf "spark.driver.memory=12g" --conf
spark.driver.maxResultSize=0 --conf spark.akka.frameSize=128 --conf
spark.default.parallelism=100
Where does that go? There are no details in this step…

Thanks

Gustavo