That's a good suggestion. One thing I want to point out that Builtins.java
[1] contains all the dml-bodied and java-bodied built-ins. For the dml
built-ins, it is easier to parse the corresponding dml script (if
documentation is available there), but for other built-ins, I am not aware
of any particular source today where they are documented -- a few of them
are documented as a part of commit message but not all.
Though it is certainly possible to slice the dml builtins from [1] out, but
it will be awesome to generate docs for all of those.
[1]
https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
Regards,
Arnab..
On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian <
baunsga...@tugraz.at>
wrote:
Hi dev team!
Update / suggestion.
First of all, i think it is great that we have people working on our
documentation. It is really important. But I can't help to note that the
improvements, are going in an unmaintainable direction where we have to
maintain the documentation at multiple locations.
For instance our new toOneHot function already had some basic
documentation inside its build-in script already [1], it would be much
nicer if we modified that and then had some automation to generate the
docs
made in commit [2]. Now we have to maintain two locations, resulting in
high risk of either being outdated.
We need to ask ourselves where do we want to make the edit of
documentation. I would argue that it should be located as close to the
internal definition of a function, because then when making or modifying
that function the documentation is also reviewed.
To achieve this i suggest that we instead relocate the documentation
source and generation to an appropriately named file inside [3], that
simply loops through all function definitions from [4], and generate
markdown files for our /docs/ folder.
Shifting to such an implementation enables us to systematically cover all
functions defined in the system. It also is a full list of the systems
current build-in functions thereby giving us the exact file paths to
extract the source documentation from the comments and generate the docs
for those files.
In the future it could be extended to verify and detecting which input
and
output types each function definition has, such that we have even less
manual documentation needs.
Best regards
Sebastian
[1]
https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml
[2]
https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8
[3]
https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common
[4]
https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
________________________________
From: arnab phani <phaniar...@gmail.com>
Sent: Sunday, May 31, 2020 3:15:44 PM
To: dev@systemml.apache.org
Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Hi,
Another advantage of markdown syntax is that it gives more freedom to
add a
description as needed. Java methods and builtins might not fall in the
same
bucket and different builtins might need different ways to describe it.
For example, glm needs more details than other built-ins. Too much
standardization can be a bad idea.
Regards,
Arnab.
On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian <
baunsga...@tugraz.at>
wrote:
Hi Janardhan,
That would be one option, lets call it option 1, I have another
suggestion:
1. Markdown syntax with out commented individual lines.
2. Jdoc like syntax with annotations
Pros and cons
* 1 Pros
* Easy Syntax most ppl know it.
* Easy to implement.
* 1 Cons
* Enables custom docs (everyone make their own distinct format
again)
* No standard way of verifying correctness of the docs (correct
parameter names etc.)
* No standard way of marking Input, return parameters.
* Requires modified Syntax with # at each line. where # normally
means header in markdown
* 2 Pros
* Easy Syntax Java ppl know it
* Standard way of making input
* Do not have to change any syntax at all from Jdoc
* Already supported syntax in DML
* 2 Cons
* Harder to implement since one would have to look into java doc
extraction and what is needed to support that, maybe we have to make
such a
thing ourselves?
I personally like option 2 with Jdoc more and then extending into
automatically parsing and making the either markdown files or HTML
files
for the docs that you either way have to do in option 1.
Best regards
Sebastian
________________________________
From: Janardhan <janardhan.pulivar...@gmail.com>
Sent: Sunday, May 31, 2020 9:25:43 AM
To: dev@systemml.apache.org
Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Hi Sebastian, :)
Since the builtin DML files do not have full documentation yet, Can we
write
markdown syntax with the following heading in each dml file itself!
1. Function description
2. Usage
3. Arguments
4. Returns
5. Usage
After that, as you've mentioned can be parsed with only removing # at
the
start of each line, and write
all this data to one `builtins-reference.md` file.
Thank you,
Janardhan
On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian <
baunsga...@tugraz.at>
wrote:
If this is meant as a question then, yes from me (this probably calls
for
a vote?), if:
1. someone is able to change the settings for the repository, to
use
the docs folder as the GitHub docs folder.
2. someone wants to make a PR containing the content of the
gh-pages
branch.
Best regards
Sebastian
________________________________
From: Janardhan <janardhan.pulivar...@gmail.com>
Sent: Tuesday, May 26, 2020 1:22:11 PM
To: dev@systemml.apache.org
Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`.
Thanks.
Hi Sebastian,
Can we move the `docs` folder in the existing `master` branch
and merge the `gh-pages` branch to `docs` folder in the top-level
directory. :)
- Janardhan
On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian <
baunsga...@tugraz.at>
wrote:
Hi Janardhan,
Reworking the documentation seems like a really good idea!
A first step in my opinion is to start in the main repository.
I would suggest the following:
- Merge the current gh-pages branch into master docs folder.
- Change the documentations page :
http://apache.github.io/systemml/
SystemDS Documentation - SystemDS<http://apache.github.io/systemml/>
apache.github.io
SystemDS Documentation
to
use the docs folder for documentation. Should be doable inside the
settings
on the repository.
- Delete the gh-pages branch
I want this change because then new PRs can change the
documentation
in
one PR rather than two (which to be honest is not going to happen
most
of
the time).
The downside (we have to mention this) is that when you clone the
repository, you have slightly larger downloads, but i think the
trade-off
is fair. If you see any other issues please mention these in this
thread.
Next task I would like to suggest to include:
- Java docs generated and
- Python docs generated to also be included as a sub-folder in
docs,
so
that the current webpage contains the docs for that to.
How to include these in a sensible manner is currently a good
question,
and has to be seriously well considered. Including these will also
enable
us to move documentation closer to the individual code parts,
making
it
again more likely to get the documentation done.
Another more programmatic task is to make a parser for our DML
files,
such
that we can generate webpage documentation based on the
documentation
syntax used (much like the python and java docs generator). This
would
be
great since it will remove the duplication of documentation in the
individual DML files, and the documentation we have already in
files
such
as: docs/dml-language-reference.md.
This also gives us an excuse to clean up that documentation/scripts
which
has very different implementations in our scripts folder:
examples:
- <https://github.com/apache/systemml/tree/master/scripts>
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
https://github.com/apache/systemml/tree/master/scripts>
systemml/scripts at master · apache/systemml · GitHub<
https://github.com/apache/systemml/tree/master/scripts>
github.com
Mirror of Apache SystemML. Contribute to apache/systemml development by
creating an account on GitHub.
https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
systemml/batch_norm1d.dml at master · apache/systemml · GitHub<
https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
github.com
Mirror of Apache SystemML. Contribute to apache/systemml development by
creating an account on GitHub.
<
https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
[https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub<
https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
github.com
Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
by creating an account on GitHub.
- <
https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
[https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
systemml/glm.dml at master · Baunsgaard/systemml · GitHub<
https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
github.com
Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
by creating an account on GitHub.
<https://github.com/apache/systemml/tree/master/scripts>
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
https://github.com/apache/systemml/tree/master/scripts>
systemml/scripts at master · apache/systemml · GitHub<
https://github.com/apache/systemml/tree/master/scripts>
github.com
Mirror of Apache SystemML. Contribute to apache/systemml development by
creating an account on GitHub.
https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
systemml/Cox.dml at master · apache/systemml · GitHub<
https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
github.com
Mirror of Apache SystemML. Contribute to apache/systemml development by
creating an account on GitHub.
Furthermore, I would suggest to postpone changing the main website:
https://github.com/apache/systemml-website
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
https://github.com/apache/systemml-website>
GitHub - apache/systemml-website: Mirror of Apache SystemML site<
https://github.com/apache/systemml-website>
github.com
Mirror of Apache SystemML site. Contribute to apache/systemml-website
development by creating an account on GitHub.
But that task has to be started when the next release is scheduled,
because then we need to find out how to copy the current
documentation
to a
archive list of static webpages located:
https://systemml.apache.org/documentation
Documentation<https://systemml.apache.org/documentation>
systemml.apache.org
Project Documentation Page
best regards
Sebastian
________________________________
From: Janardhan <janard...@apache.org>
Sent: Sunday, May 24, 2020 11:02:53 AM
To: dev@systemml.apache.org
Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Hi Sebastian,
In our discussion[1][2] a while ago, there was a topic about
changing
website
stack.
Let us start a discussion about this in this thread.
We are planning to work on this, after a feasibility study and tech
stack
vetting.
Anybody would like to give suggestions would be great.
[1] https://github.com/apache/systemml/pull/877
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
https://github.com/apache/systemml/pull/877>
[MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by
Baunsgaard · Pull Request #877 · apache/systemml · GitHub<
https://github.com/apache/systemml/pull/877>
github.com
Brute change of documentation page to reflect name change. To fully
enforce this two more steps are needed:
https://github.com/apache/systemml-website Access to SystemML website
server on https://systemml.apache.org/ Furthermore, we might want to
consider starting fresh in the documentation.
[2] https://github.com/apache/systemml-website/pull/66
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
https://github.com/apache/systemml-website/pull/66>
[SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request
#66 · apache/systemml-website · GitHub<
https://github.com/apache/systemml-website/pull/66>
github.com
Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used,
and exported as tasks. Documentation for gulpfile.js. Stick to Susy2,
since
mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art.
Commit package-lock.json file to version control.
Thank you,
Janardhan
